Skip to content
Get started

SDK: Footage

footage() is a layer that shows a video in a part: footage() alone is the part's own take in sync with its speech, footage(file, opts) another file, footage(videoItem, opts) another video item of the project. It stacks with the part's other layers in the order they are listed (video.md). A take, its cut list and its captions are in speech.md.

import { footage, part, take, video } from "@allyhub/sdk";
import broll from "./broll/inbox.mp4";
export default video({
format: "9x16",
parts: () => [
part({
key: "intro",
speech: take("./host.mp4").range("So here is", "every time"),
layers: [
footage({
reframe: "face",
punchIn: ["free", "insane@1.15"],
pullback: true,
}),
footage(broll, {
at: "inbox",
dur: 1.5,
box: [0.55, 0.1, 0.4, 0.3],
radius: 24,
}),
footage("./screen.mp4", {
from: 3,
crop: [0, 0, 0.5, 1],
speed: 2,
fit: "contain",
audio: true,
gainDb: -6,
}),
],
}),
part({
key: "inset",
speech: take("./host.mp4").range("Now look"),
layers: [
footage({
box: {
portrait: [0.1, 0.6, 0.8, 0.3],
default: [
{ t: 0, box: "full" },
{ t: "look", box: [0.6, 0.6, 0.35, 0.35], ease: "in-out" },
],
},
}),
],
}),
],
});
  • A file: an import (import clip from "./clip.mp4") or a path. A path is relative to the folder of the module that wrote the footage() call, also once its part is spread into another piece (project.md).
  • No file: the part’s take, in sync with its speech, following its cut list. A part with no take is AK-A001; a take that is not a video is AK-A003. A take played at another speed (take(…).speed(1.2)) cannot show its own picture: part() throws.
  • A video item: an imported video() of the project (below).
  • A still (PNG, JPEG, WebP, AVIF, GIF or SVG) is drawn as picture() draws it (pictures.md). Any other file that is not a video (mp4, webm or mov) is AK-A003; a file that is not there AK-A001.
  • When the layer lasts longer than the file, its last frame holds; before a take’s file starts (speech in the lead), its first frame holds. A video with alpha (ProRes 4444, VP9 with alpha) shows what is under its transparent pixels.

Besides at, until and dur (word references or seconds; default the whole part):

Option Type Default What it is
from seconds 0 Seconds into the file at the layer’s start. A take’s own footage follows its speech instead
speed over 0, up to 16 1 Playback speed; its sound follows, pitch too
audio boolean false Mix the file’s own sound in. A take’s own footage sounds as the part’s speech
gainDb dB 0 Its sound’s gain
box Rect, "full", BoxKey[], or per frame shape "full" Where it shows, [x, y, w, h] as fractions of the frame, inside it
fit "cover", "contain" "cover" How the picture fills its box
crop Rect, or per frame shape none A part of the source (fractions of it), then fitted to the box
shape "rect", "circle" "rect" "circle": the ellipse inscribed in the box
radius px at 1080p 0 Rounds the box’s corners, at most half its shorter side; what is outside the corners shows what is under the layer. Not with "circle"
reframe "face", { face: true, faceY? }, "center", { x, y } none Crops the source to the box’s shape round the speaker’s face (a smoothed path; the face’s centre at faceY of the crop’s height, default 0.4), its centre, or a point of the source
speaker a face id (whole number from 0) the track’s default speaker The face the layer follows: for a face reframe, punch-ins, the pullback and ctx.face
punchIn (WordRef | PunchIn)[] none Zooms in on words (below)
pullback true or Pullback none Opens zoomed in and eases back to the whole picture
sharpen 0 to 1 none Sharpens after scaling, for footage drawn larger than its pixels (the footage-scale fact). Renders and stills only
background FootageBackground none The picture behind the person treated apart from them (below)
  • Lengths in px at 1080p (radius, a background’s blur, feather, shrink) scale with the format’s shorter side, and radius with a keyframed box.
  • A reframe with no face (no face track yet, or none seen then) is centred on { x: 0.5, y: 0.4 } of the source.
  • Per frame shape: box and crop take a PerOrient, { landscape?, portrait?, square?, default? }. Each piece takes the entry for its ctx.dims.orient, else default, else none (the full frame, no crop). So pieces of one film in two formats can share their parts.

box as BoxKey[]: [{ t, box, ease? }], t a word reference or seconds, box a rectangle or "full", ease the move into that key ("linear", "in", "out", "in-out"; default "linear").

  • The picture keeps its shape in every box, fitted as fit says: a box that changes shape shows more or less of it, never a stretched one.
  • Every key lies inside the frame; footage() throws for one that reaches past it. A push-in larger than the frame is a punch-in.
  • A keyframed box takes no reframe, punchIn or pullback (footage() throws).
  • A punch-in is a word reference, "free" (scale 1.12), "free@1.15" (a decimal after @ is the scale; "free@2" is still the second “free”), or { at, until?, scale?, in?, out?, ease?, center? }. It eases in over in seconds (default 0.2) centred on at, holds for the word (or until until), and eases out over out (default 0.25) with ease (default "in-out"). scale is over 1, up to 3; in and out are over 0, up to 30.
  • A pullback starts at scale (default 1.12, up to 3) and eases (ease, default "out") back to the whole picture over dur seconds (default 0.6, up to 10).
  • Both zoom about center, a point { x, y } as fractions of the box; default the speaker’s face, else { x: 0.5, y: 0.42 }.

A face track and a person matte are what AllyHub measures once from a video and keeps in the project’s media index (.allyhub/cache/index/<sha256 of the file>/).

  • The face track is read by reframe: "face", the punch-in and pullback centres, speaker, and in a layer by ctx.face(t, id?) (one face’s box [x, y, w, h] in the frame’s pixels at part time t, by default the layer’s speaker; undefined when none) and ctx.faces(t) ([{ id, box }]), from the topmost footage showing at t that has one. Every face has a stable id; allyhub frames --overlay draws them with their ids.
  • The person matte is read by graphic layers drawn z: "behind-person" and by a footage layer’s background.
  • Every command that draws a piece (check, render, dev, frames, …) makes what the media index lacks first; the first time downloads the runtime and models into ~/.allyhub/. ALLYHUB_NO_INDEX=1 skips it.
  • Until one is made, a face reframe is centred, layers behind the person are drawn in front, and a background is left as it is; a speaker that is not a face of the track is centred too. Each is a fallback fact, which allyhub inspect prints.

A part’s layers are listed back to front: each is drawn over those before it, footage and graphics alike. A graphic layer listed before a footage layer is under it (the graphics-under-footage fact lists such layers).

A graphic layer with z: "behind-person" is drawn with the part’s own footage (footage() without a file) wherever that is listed: right over its picture and under its speaker, who is cut out with the take’s person matte. Without the part’s own footage it is drawn in front (a fallback fact). z: "overlay" is over the finished picture (video.md).

background treats the picture behind the person apart from the person, through the file’s person matte.

import { footage } from "@allyhub/sdk";
export const backgrounds = [
footage({ background: { blur: 24 } }),
footage({ background: { dim: 0.4, desaturate: 1 } }),
footage({ background: { color: "var(--ak-color-bg)" } }),
footage({ background: { image: "./studio.jpg", blur: 6 } }),
footage({ background: { color: "transparent" } }),
];
Field Range What it is
blur 0 to 200 The background’s Gaussian blur, its standard deviation as CSS blur(), px at 1080p
dim 0 to 1 Darker, to black
desaturate 0 to 1 Greyer, to gray
color a CSS colour A flat colour instead; "transparent" draws the person alone over what is under the layer. Not with blur, dim, desaturate or image
image PNG, JPEG or WebP An image instead (an import or a path from the module), filling the box (cover); blur, dim and desaturate then treat it
feather 0 to 40, default 2 The softness of the person’s edge, px at 1080p (MATTE_FEATHER)
shrink −40 to 40, default 4 How far the person’s edge is pulled in, px at 1080p; negative lets it out (MATTE_SHRINK)
  • The order is: the treated background, graphic layers drawn z: "behind-person", the person, the layers in front.
  • The matte is made firm, then softened by feather and moved by shrink.
  • An image that is not a picture is AK-A003. Until the file has a matte the footage shows as it is (a fallback fact).

The render and stills draw footage with ffmpeg, the exact frame of the file for every frame. The live preview plays each footage layer as a muted <video> kept to the clock, placed by the same plan. Where they differ: the preview plays sharpen unsharpened, and approximates a background (its blur is of the whole picture, its matte edge coarser). allyhub frames draws as the render does.

A video item of the project, imported, is drawn by the stage at its own time, never rendered first.

import { custom, part, video } from "@allyhub/sdk";
export default video({
format: "16x9",
deliverable: false,
parts: () => [
part({
key: "open",
dur: 3,
anchors: { turn: 1.5 },
layers: [custom({ render() {} })],
}),
],
});
import { footage, part, video } from "@allyhub/sdk";
import demo from "../demo/demo.ts";
export default video({
format: "9x16",
parts: () => [
part({
key: "inset",
dur: 6,
layers: [footage(demo, { box: [0.55, 0.1, 0.4, 0.3], from: "@turn" })],
}),
],
});
  • footage(videoItem, opts) takes at, until, dur, from (its own moment at the layer’s start: seconds, an anchor of one of its parts such as "@turn", or a part’s key), speed (over 0, up to 16), box and fit, audio and gainDb (its speech, room and sound effects; at speed 1 only), camera and cameras (the camera its scenery layers are drawn from, on its clock: one for all, or by layer name), and quality: { scale } (how finely its scenery layers are drawn here, over 0 up to 4, as scenery()’s; scenery.md). crop, reframe, speaker, punchIn, pullback, sharpen, background, shape and radius throw.
  • It draws its graphic layers only: a video item with footage of its own cannot be shown inside another yet (the load fails and says so).
  • What it reads counts as the outer video’s, so the part cache and the check follow it. A video that reaches itself through the videos it shows is AK-M010.
  • deliverable: false marks a video only shown inside others: it is not rendered as a film of its own unless named.
  • take(videoItem).range(from, to) makes a part that plays a stretch of it, hearing its sound and saying its words; footage() without a file in that part draws its picture in sync (speech.md).
  • ctx.video(videoItem, el) draws it in an element of a layer and resolves to an EmbeddedVideo ({ id, duration, width, height, seek(t, layout?), at(ref) }); seek it from render through ctx.wait:
import { custom, part, type EmbeddedVideo } from "@allyhub/sdk";
import demo from "../demo/demo.ts";
const shown = new WeakMap<HTMLElement, EmbeddedVideo>();
export const screen = part({
key: "screen",
dur: 6,
layers: [
custom({
mount(el, ctx) {
if (!ctx.video) return;
ctx.wait(ctx.video(demo, el).then((v) => shown.set(el, v)));
},
render(el, t, ctx) {
const v = shown.get(el);
if (!v) return;
const at = Math.min(Math.max(t, 0), v.duration - 1e-3);
ctx.wait(v.seek(at, ctx.layoutOnly === true));
},
}),
],
});

A separate microphone and a camera start at different moments. allyhub align <reference> <other> measures the offset between them from their sound (other = reference + offset), how sharply they match, and how it drifts. Writing it in is the piece’s: the microphone is the part’s speech, the camera its picture.

import { footage, part, take, video } from "@allyhub/sdk";
const OFFSET = 1.5; // from allyhub align mic.wav cam.mp4
export default video({
format: "16x9",
parts: () => [
part({
key: "intro",
speech: take("./mic.wav").range("So here is", "every time"),
layers: [footage("./cam.mp4", { from: 12.84 + OFFSET })],
}),
],
});
  • align also prints the drift as a line, offset(t) = at0 + slope × t, and the speed (1 + slope) that keeps the camera in step; and, in its JSON, pictureOffset: the offset for the camera’s picture when its file’s sound starts later on its own clock.
  • Only a take’s own footage follows its cut list: to cut both, make the two one file first.
  • allyhub frames cuts draws a review sheet per cut (a cut take’s jumps, part joins) to look at the result.
Was this page helpful?