# SDK: Footage

> footage() is a layer that shows a video in a part: footage() alone is the part's own take in sync with its speech, footage(file, opts) another file, footage(videoItem, opts) another video item of the project. It stacks with the part's other layers in the order they are listed (video.md). A take, its cut list and its captions are in speech.md.

Source: https://allyhub.com/docs/reference/sdk/footage · AllyHub 0.10.0
Index of all docs: https://allyhub.com/llms.txt

```ts file=talk/short/short.ts


  format: "9x16",
  parts: () => [
    part({
      key: "intro",
      speech: take("./host.mp4").range("So here is", "every time"),
      layers: [
        footage({
          reframe: "face",
          punchIn: ["free", "insane@1.15"],
          pullback: true,
        }),
        footage(broll, {
          at: "inbox",
          dur: 1.5,
          box: [0.55, 0.1, 0.4, 0.3],
          radius: 24,
        }),
        footage("./screen.mp4", {
          from: 3,
          crop: [0, 0, 0.5, 1],
          speed: 2,
          fit: "contain",
          audio: true,
          gainDb: -6,
        }),
      ],
    }),
    part({
      key: "inset",
      speech: take("./host.mp4").range("Now look"),
      layers: [
        footage({
          box: {
            portrait: [0.1, 0.6, 0.8, 0.3],
            default: [
              { t: 0, box: "full" },
              { t: "look", box: [0.6, 0.6, 0.35, 0.35], ease: "in-out" },
            ],
          },
        }),
      ],
    }),
  ],
});
```

## The file

- **A file**: an import (`import clip from "./clip.mp4"`) or a path. A path is relative to the folder of the module that wrote the `footage()` call, also once its part is spread into another piece ([project.md](https://allyhub.com/docs/reference/sdk/project.md)).
- **No file**: the part's take, in sync with its speech, following its cut list. A part with no take is AK-A001; a take that is not a video is AK-A003. A take played at another speed (`take(…).speed(1.2)`) cannot show its own picture: `part()` throws.
- **A video item**: an imported `video()` of the project ([below](#a-video-item-as-footage)).
- A still (PNG, JPEG, WebP, AVIF, GIF or SVG) is drawn as `picture()` draws it ([pictures.md](https://allyhub.com/docs/reference/sdk/pictures.md#footage-of-a-still)). Any other file that is not a video (mp4, webm or mov) is AK-A003; a file that is not there AK-A001.
- When the layer lasts longer than the file, its last frame holds; before a take's file starts (speech in the lead), its first frame holds. A video with alpha (ProRes 4444, VP9 with alpha) shows what is under its transparent pixels.

## Options

Besides `at`, `until` and `dur` (word references or seconds; default the whole part):

| Option | Type | Default | What it is |
| --- | --- | --- | --- |
| `from` | seconds | 0 | Seconds into the file at the layer's start. A take's own footage follows its speech instead |
| `speed` | over 0, up to 16 | 1 | Playback speed; its sound follows, pitch too |
| `audio` | boolean | false | Mix the file's own sound in. A take's own footage sounds as the part's speech |
| `gainDb` | dB | 0 | Its sound's gain |
| `box` | `Rect`, `"full"`, `BoxKey[]`, or per frame shape | `"full"` | Where it shows, `[x, y, w, h]` as fractions of the frame, inside it |
| `fit` | `"cover"`, `"contain"` | `"cover"` | How the picture fills its box |
| `crop` | `Rect`, or per frame shape | none | A part of the source (fractions of it), then fitted to the box |
| `shape` | `"rect"`, `"circle"` | `"rect"` | `"circle"`: the ellipse inscribed in the box |
| `radius` | px at 1080p | 0 | Rounds the box's corners, at most half its shorter side; what is outside the corners shows what is under the layer. Not with `"circle"` |
| `reframe` | `"face"`, `{ face: true, faceY? }`, `"center"`, `{ x, y }` | none | Crops the source to the box's shape round the speaker's face (a smoothed path; the face's centre at `faceY` of the crop's height, default 0.4), its centre, or a point of the source |
| `speaker` | a face id (whole number from 0) | the track's default speaker | The face the layer follows: for a face reframe, punch-ins, the pullback and `ctx.face` |
| `punchIn` | `(WordRef \| PunchIn)[]` | none | Zooms in on words ([below](#punch-ins-and-the-pullback)) |
| `pullback` | `true` or `Pullback` | none | Opens zoomed in and eases back to the whole picture |
| `sharpen` | 0 to 1 | none | Sharpens after scaling, for footage drawn larger than its pixels (the `footage-scale` fact). Renders and stills only |
| `background` | `FootageBackground` | none | The picture behind the person treated apart from them ([below](#the-background-behind-the-person)) |

- **Lengths** in px at 1080p (`radius`, a background's `blur`, `feather`, `shrink`) scale with the format's shorter side, and `radius` with a keyframed box.
- **A reframe** with no face (no face track yet, or none seen then) is centred on `{ x: 0.5, y: 0.4 }` of the source.
- **Per frame shape**: `box` and `crop` take a `PerOrient`, `{ landscape?, portrait?, square?, default? }`. Each piece takes the entry for its `ctx.dims.orient`, else `default`, else none (the full frame, no crop). So pieces of one film in two formats can share their parts.

### Box keyframes

`box` as `BoxKey[]`: `[{ t, box, ease? }]`, `t` a word reference or seconds, `box` a rectangle or `"full"`, `ease` the move into that key (`"linear"`, `"in"`, `"out"`, `"in-out"`; default `"linear"`).

- The picture keeps its shape in every box, fitted as `fit` says: a box that changes shape shows more or less of it, never a stretched one.
- Every key lies inside the frame; `footage()` throws for one that reaches past it. A push-in larger than the frame is a punch-in.
- A keyframed box takes no `reframe`, `punchIn` or `pullback` (`footage()` throws).

### Punch-ins and the pullback

- **A punch-in** is a word reference, `"free"` (scale 1.12), `"free@1.15"` (a decimal after `@` is the scale; `"free@2"` is still the second "free"), or `{ at, until?, scale?, in?, out?, ease?, center? }`. It eases in over `in` seconds (default 0.2) centred on `at`, holds for the word (or until `until`), and eases out over `out` (default 0.25) with `ease` (default `"in-out"`). `scale` is over 1, up to 3; `in` and `out` are over 0, up to 30.
- **A pullback** starts at `scale` (default 1.12, up to 3) and eases (`ease`, default `"out"`) back to the whole picture over `dur` seconds (default 0.6, up to 10).
- Both zoom about `center`, a point `{ x, y }` as fractions of the box; default the speaker's face, else `{ x: 0.5, y: 0.42 }`.

## Faces and mattes

A face track and a person matte are what AllyHub measures once from a video and keeps in the project's media index (`.allyhub/cache/index/<sha256 of the file>/`).

- The **face track** is read by `reframe: "face"`, the punch-in and pullback centres, `speaker`, and in a layer by `ctx.face(t, id?)` (one face's box `[x, y, w, h]` in the frame's pixels at part time `t`, by default the layer's speaker; `undefined` when none) and `ctx.faces(t)` (`[{ id, box }]`), from the topmost footage showing at `t` that has one. Every face has a stable id; `allyhub frames --overlay` draws them with their ids.
- The **person matte** is read by graphic layers drawn `z: "behind-person"` and by a footage layer's `background`.
- Every command that draws a piece (`check`, `render`, `dev`, `frames`, …) makes what the media index lacks first; the first time downloads the runtime and models into `~/.allyhub/`. `ALLYHUB_NO_INDEX=1` skips it.
- Until one is made, a face reframe is centred, layers behind the person are drawn in front, and a background is left as it is; a `speaker` that is not a face of the track is centred too. Each is a `fallback` fact, which `allyhub inspect` prints.

## Stacking with graphic layers

A part's `layers` are listed back to front: each is drawn over those before it, footage and graphics alike. A graphic layer listed before a footage layer is under it (the `graphics-under-footage` fact lists such layers).

A graphic layer with `z: "behind-person"` is drawn with the part's own footage (`footage()` without a file) wherever that is listed: right over its picture and under its speaker, who is cut out with the take's person matte. Without the part's own footage it is drawn in front (a `fallback` fact). `z: "overlay"` is over the finished picture ([video.md](https://allyhub.com/docs/reference/sdk/video.md)).

## The background behind the person

`background` treats the picture behind the person apart from the person, through the file's person matte.

```ts

  footage({ background: { blur: 24 } }),
  footage({ background: { dim: 0.4, desaturate: 1 } }),
  footage({ background: { color: "var(--ak-color-bg)" } }),
  footage({ background: { image: "./studio.jpg", blur: 6 } }),
  footage({ background: { color: "transparent" } }),
];
```

| Field | Range | What it is |
| --- | --- | --- |
| `blur` | 0 to 200 | The background's Gaussian blur, its standard deviation as CSS `blur()`, px at 1080p |
| `dim` | 0 to 1 | Darker, to black |
| `desaturate` | 0 to 1 | Greyer, to gray |
| `color` | a CSS colour | A flat colour instead; `"transparent"` draws the person alone over what is under the layer. Not with `blur`, `dim`, `desaturate` or `image` |
| `image` | PNG, JPEG or WebP | An image instead (an import or a path from the module), filling the box (cover); `blur`, `dim` and `desaturate` then treat it |
| `feather` | 0 to 40, default 2 | The softness of the person's edge, px at 1080p (`MATTE_FEATHER`) |
| `shrink` | −40 to 40, default 4 | How far the person's edge is pulled in, px at 1080p; negative lets it out (`MATTE_SHRINK`) |

- The order is: the treated background, graphic layers drawn `z: "behind-person"`, the person, the layers in front.
- The matte is made firm, then softened by `feather` and moved by `shrink`.
- An image that is not a picture is AK-A003. Until the file has a matte the footage shows as it is (a `fallback` fact).

## The preview and the render

The render and stills draw footage with ffmpeg, the exact frame of the file for every frame. The live preview plays each footage layer as a muted `<video>` kept to the clock, placed by the same plan. Where they differ: the preview plays `sharpen` unsharpened, and approximates a `background` (its blur is of the whole picture, its matte edge coarser). `allyhub frames` draws as the render does.

## A video item as footage

A video item of the project, imported, is drawn by the stage at its own time, never rendered first.

```ts file=demo/demo.ts

  format: "16x9",
  deliverable: false,
  parts: () => [
    part({
      key: "open",
      dur: 3,
      anchors: { turn: 1.5 },
      layers: [custom({ render() {} })],
    }),
  ],
});
```

```ts file=film/film.ts


  format: "9x16",
  parts: () => [
    part({
      key: "inset",
      dur: 6,
      layers: [footage(demo, { box: [0.55, 0.1, 0.4, 0.3], from: "@turn" })],
    }),
  ],
});
```

- `footage(videoItem, opts)` takes `at`, `until`, `dur`, `from` (its own moment at the layer's start: seconds, an anchor of one of its parts such as `"@turn"`, or a part's key), `speed` (over 0, up to 16), `box` and `fit`, `audio` and `gainDb` (its speech, room and sound effects; at speed 1 only), `camera` and `cameras` (the camera its scenery layers are drawn from, on its clock: one for all, or by layer name), and `quality: { scale }` (how finely its scenery layers are drawn here, over 0 up to 4, as `scenery()`'s; [scenery.md](https://allyhub.com/docs/reference/sdk/scenery.md)). `crop`, `reframe`, `speaker`, `punchIn`, `pullback`, `sharpen`, `background`, `shape` and `radius` throw.
- It draws its graphic layers only: a video item with footage of its own cannot be shown inside another yet (the load fails and says so).
- What it reads counts as the outer video's, so the part cache and the check follow it. A video that reaches itself through the videos it shows is AK-M010.
- `deliverable: false` marks a video only shown inside others: it is not rendered as a film of its own unless named.
- `take(videoItem).range(from, to)` makes a part that plays a stretch of it, hearing its sound and saying its words; `footage()` without a file in that part draws its picture in sync ([speech.md](https://allyhub.com/docs/reference/sdk/speech.md)).
- `ctx.video(videoItem, el)` draws it in an element of a layer and resolves to an `EmbeddedVideo` (`{ id, duration, width, height, seek(t, layout?), at(ref) }`); seek it from `render` through `ctx.wait`:

```ts file=film/screen.ts


const shown = new WeakMap<HTMLElement, EmbeddedVideo>();

  key: "screen",
  dur: 6,
  layers: [
    custom({
      mount(el, ctx) {
        if (!ctx.video) return;
        ctx.wait(ctx.video(demo, el).then((v) => shown.set(el, v)));
      },
      render(el, t, ctx) {
        const v = shown.get(el);
        if (!v) return;
        const at = Math.min(Math.max(t, 0), v.duration - 1e-3);
        ctx.wait(v.seek(at, ctx.layoutOnly === true));
      },
    }),
  ],
});
```

## Two recordings of one moment

A separate microphone and a camera start at different moments. `allyhub align <reference> <other>` measures the offset between them from their sound (other = reference + offset), how sharply they match, and how it drifts. Writing it in is the piece's: the microphone is the part's speech, the camera its picture.

```ts file=talk/two/two.ts

const OFFSET = 1.5; // from allyhub align mic.wav cam.mp4

  format: "16x9",
  parts: () => [
    part({
      key: "intro",
      speech: take("./mic.wav").range("So here is", "every time"),
      layers: [footage("./cam.mp4", { from: 12.84 + OFFSET })],
    }),
  ],
});
```

- `align` also prints the drift as a line, offset(t) = at0 + slope × t, and the speed (1 + slope) that keeps the camera in step; and, in its JSON, `pictureOffset`: the offset for the camera's picture when its file's sound starts later on its own clock.
- Only a take's own footage follows its cut list: to cut both, make the two one file first.
- `allyhub frames cuts` draws a review sheet per cut (a cut take's jumps, part joins) to look at the result.
