# SDK: Speech

> What a part says: a recording (take()), text to speech (say()) or a stretch of another video item (take(piece)). Its words time the part, place its layers and sounds, duck the music and make the captions. This file covers those, word references, anchors, shown-and-said text, captions and the words and edits files.

Source: https://allyhub.com/docs/reference/sdk/speech · AllyHub 0.10.0
Index of all docs: https://allyhub.com/llms.txt

```ts file=launch/film.ts


  format: "16x9",
  parts: ({ s }) => [
    part({ key: "hook", speech: take(vo).saying(s("launch.film.hook.say")) }),
    part({
      key: "how",
      speech: take("./vo/en.mp3").saying(s("launch.film.how.say")).speed(1.1),
    }),
    part({
      key: "host",
      speech: take("./host.mp4")
        .range("So here is", "every time")
        .edits("./edits/host.json")
        .denoise(12)
        .roomTone({ limitDb: 4, fade: 0.06 }),
    }),
    part({
      key: "cta",
      speech: say(s("launch.film.cta.say"), { voice: "rachel", speed: 1.05 }),
    }),
  ],
});
```

A path written as a string (`take("./host.mp4")`, `.edits("./edits/host.json")`) is relative to the module that wrote it, also once the part is spread into another piece ([project.md](https://allyhub.com/docs/reference/sdk/project.md)).

## A part's length

A part with speech lasts lead + speech + tail: `lead` before the first word (default 0.2 s) and `tail` after the last (default 0.3 s), less any `overlap` ([J and L cuts](#j-and-l-cuts)). A part without speech lasts `dur` (default 4 s). The defaults are `PART_DEFAULTS`, changed for the project by `defaults.part` in `allyhub.config.ts` and for one video by `video({ defaults: { part } })`.

## `take(file)`

A recording, audio or video (an import, or a path), with its words in `<name>.words.json` beside it ([Words files](#words-files)). `allyhub words <file>` hears it on this computer and writes that file. A take without one is AK-T001.

Alone, a take is the whole recording, from its first word to its last. Each method returns a new take:

- `.range(from, to?)`: the stretch from the first words `from` to the end of the first `to` after them (phrases; case and punctuation ignored); without `to`, to the end. A phrase not found is AK-T003.
- `.saying(text)`: the stretch that says this line, usually `s(key)`, found by its words, misheard words tolerated; the line's spelling is the caption. Parts that take the same file each search after the previous part's stretch. A line not found is AK-T003.
- `.edits(path)`: a cut list, a `.json` file ([Edits files](#edits-files)). One that is not there, or does not match the format, is AK-T005: the take would play uncut.
- `.denoise(db?)`: reduces its noise by `db` (over 0, up to 97; default 12) with ffmpeg's FFT denoiser, in the render and the preview.
- `.roomTone(tone)`: how the part's lead and tail are filled where the take has no silence of its own round the part's words ([Room tone](#room-tone)).
- `.speed(times)`: plays it `times` as fast, its pitch kept (0.5 to 2); its words, the part's length and everything timed by them follow. The file is not changed. A part whose `footage()` shows the take's own picture cannot change its speed (`part()` throws).

With `.edits()`, `range` and `saying` look only at the words the edits keep: a phrase or line said only in removed words is AK-T006. A take a voice tool generated keeps the text it was made from in its `.gen.json`: a part whose line is not in that text is AK-T007.

### Room tone

A part's lead and tail play the take's own room round its words. Where that runs out, a room tone fills them:

- The fill is the quietest steady stretch the build measured, or `{ from, to }` (seconds of the file, both given, at least 0.05 s apart), looped. `.roomTone(false)` leaves digital silence.
- Its level is matched to the floor of the take's own room beside the join (the mean power of the quietest quarter of the 10 ms frames within 0.3 s), at most `limitDb` either way (0 to 60; default 6). `match: false` plays it as recorded; `gainDb` (−60 to 24) sets its gain instead. Where the take keeps under 0.1 s of its own room at a join, or its levels are not known, it plays as recorded.
- It cross-fades with the take over `fade` seconds (0 to 1; default 0.04), equal-power.

The values are `ROOM_MATCH` ([defaults.md](https://allyhub.com/docs/reference/sdk/defaults.md)). `allyhub inspect timeline` prints each fill's gain and where it came from.

## `take(piece)`

A stretch of another video item of the project, imported, with a clock of its own:

```ts file=scene/scene.ts

  format: "16x9",
  deliverable: false,
  parts: () => [
    part({ key: "desk", dur: 6, anchors: { turn: 1.2, press: [3, 4.5] } }),
  ],
});
```

```ts file=short/short.ts


  format: "9x16",
  parts: () => [
    part({
      key: "cut",
      speech: take(scene).range("@turn", "@/press"),
      layers: [footage()],
    }),
  ],
});
```

- `.range(from, to?)` takes seconds of its clock, an anchor of one of its parts (`"@turn"`, `"@/turn"`, `"@turn+0.2"`) or a part's key; without `to`, to its end. A reference it does not have is AK-T003.
- Its ends go on the frame grid, never into a word they are not in (a stretch that splits a word is a `cut-in-word` fact). The part lasts the stretch exactly, with no lead or tail; it hears the item's speech, room tone and sound effects (not its music), says its words and has its anchors.
- `footage()` without a file draws its picture in sync, in the part's frame whatever the item's own format ([footage.md](https://allyhub.com/docs/reference/sdk/footage.md)). `deliverable: false` marks an item that is only shown inside others.

## `say(text, opts?)`

Text to speech. `allyhub voice` makes the missing files with the cloud voice tool, each in `assets/media/gen/say/<hash>` with its words and its record (each tool's description says what it costs). A `say` without its file is AK-T002; until it has one, the part lasts about as long as the line takes to say (`SPEECH_RATE_DEFAULTS`: 0.38 s a word, 0.22 s a Chinese, Japanese or Korean character; at least 1 s), and its anchors are placed in proportion along the line.

- `voice`: the voice tool's voice id (`allyhub ai-media find-voices` finds them); default the tool's own.
- `tool`: the voice tool; default `"speak"` (`"clone-voice"` speaks in the voice of a recording you give in `params`; `allyhub help ai-media <tool>` lists its input).
- `params`: more input for the tool, plain JSON: fields its schema has besides the text and voice (`text` and `voiceId` throw).
- `speed`: 0.5 to 2, default 1: plays the file faster or slower, pitch kept, as `.speed()` does. Not sent to the tool, so the same file.

The file is named by a hash of the text read aloud, the voice, the tool and the params: a changed line needs a new file. Text from `s(key)` is read aloud as its said side ([Shown and said](#shown-and-said)).

## Word references

A word reference names a moment of a part by what is said there. A layer's `at` and `until`, a part's `still`, a sound's `at`, a footage `punchIn` and every prop declared `time()` take one; a layer reads one with `ctx.at(ref)`, in seconds from the part's start.

| Reference | Moment |
| --- | --- |
| `"inbox"` | The start of the first time "inbox" is said in the part |
| `"inbox@2"` | The second time |
| `"inbox+0.2"`, `"inbox@2-0.35"` | Shifted by seconds |
| `"@name"`, `"@/name"`, `"@name+0.2"` | An [anchor](#anchors)'s start, a stretch's end, shifted |
| `1.8`, `"1.8"` | Seconds from the part's start |

A reference may be a phrase; case and punctuation are ignored. With `say` or `saying` a reference is found in the line first, so a misheard word still lands where it is said. A reference the part does not say, or any word in a part without speech, is AK-T004.

## Anchors

An anchor is a name written in the line a part says, in each language's script ([project.md](https://allyhub.com/docs/reference/sdk/project.md#scripts)). The code refers to the name, so a reworded or translated line keeps what hangs on it.

```yaml
parts:
  - id: how
    say: "@{step1}Paste your form link. We @{step2}match the answers you @{have}already have@{/have}."
```

| In the line | In code | Moment |
| --- | --- | --- |
| `@{name}` | `"@name"` | A point: the start of the word after it (the end of the last word, at the line's end) |
| `@{name}…@{/name}` | `"@name"`, `"@/name"` | A stretch: the start of its first word, the end of its last |
| | `"@name+0.2"`, `"@/name-0.1"` | Shifted by seconds |

```ts file=launch/how.ts

  part({
    key: "how",
    speech: say(s("launch.film.how.say")),
    sounds: [{ at: "@step1", sound: "tick" }],
    layers: [
      custom({
        at: "@step2",
        until: "@/have+0.3",
        mount(el, ctx) {
          el.dataset["step"] = String(ctx.at("@step1"));
          el.dataset["film"] = String(ctx.film?.at("@step1") ?? 0);
        },
        render() {},
      }),
    ],
  });
```

- An anchor is looked up in the line the part says, so the speech comes from that script line: `say(s("<prefix>.<part>.say"))` or `take(…).saying(s(…))`. A name the line does not have is AK-T004.
- A name is letters, digits, `_` and `-`, once in a line; stretches may overlap. A name written twice, an end without a start, a stretch with no words and an anchor in a plural are AK-M007. An anchor goes before a `<shown|said>` piece, not inside it.
- An anchor lands by the line's letters matched to the words of its sound.
- `ctx.at("@name")` is seconds from the part's start; `ctx.film.at("@name")` is film seconds and takes an anchor of any part. `allyhub inspect timeline` prints where each landed.

### A part's own anchors

A part names moments in seconds from its start, with no speech needed: `part({ anchors: { press: 7.1, map: [9.6, 13.4] } })`. A number is a point, `[from, to]` a stretch; `"@press"`, `"@map"`, `"@/map"` and `"@press+0.2"` work as above. A name is the line's or the part's own, not both: a name in both is AK-T004, and `"@name"` is the part's own.

## Shown and said

A piece of a script's `say` shown one way and said another is `<shown|said>`: `from <12|twelve> fields`, `<AI|A I>`. The line as shown is what `s()` returns, titles and captions show; the line as said is what `say` reads aloud and what `saying` looks for in a recording. Either side may be empty: `<™|>` is shown and not said, `<|[laughs]>` said and not shown. In `assets/text/<lang>.json`, the same is a `{ "display", "speak" }` value.

## Captions

Burned-in captions are a graphic layer in each part that has them: a starter component (`captionWord`, `captionLine`, `captionSubtitle`, added with `allyhub add`), a component of the project's, or `custom()`. In any layer:

- `ctx.words()` is the part's words on its clock, `{ text, start, end, confidence? }`; empty without speech.
- `ctx.captions(opts?)` is the cues heard over the part, on its clock:

```ts fragment
type CaptionTrackCue = {
  start: number;
  end: number;
  text: string;
  words: { text: string; start: number; end: number }[];
};
```

| Option | Default | What it is |
| --- | --- | --- |
| `maxChars` | 32 (16 for Chinese, Japanese and Korean) | Most characters in a cue |
| `maxWords` | none | Most words in a cue |
| `keep` | none | Phrases a cue never breaks inside, matched as said; the cue round one may run over the limits |
| `close` | 0.4 | A gap between two cues shorter than this many seconds closes |
| `edges` | false | `true`: the first cue from the part's start, the last to its end |

Cues come from the part's speech (a cut take's kept words, spelled as the script when there is one), break at sentence and clause ends, then by length, and never span two parts. In a J or L cut, the words heard under a neighbour's picture are there too. Where they go and which words they stress is the layer's code:

```ts file=launch/captions.ts

  mount(el, ctx) {
    for (const cue of ctx.captions({ maxChars: 20, edges: true })) {
      const line = h("div", { opacity: 0 });
      line.dataset["start"] = String(cue.start);
      line.dataset["end"] = String(cue.end);
      const stressed = emphasized(cue.words, ["free", "two minutes"]);
      const emoji = wordEmoji(cue.words, { free: "🎁" });
      cue.words.forEach((w, i) => {
        const color = stressed[i] ? "var(--ak-color-accent)" : "inherit";
        line.append(h("span", { color }, [`${w.text} `]));
        const e = emoji[i];
        if (e) line.append(emojiEl(e));
      });
      el.append(line);
    }
  },
  render(el, t) {
    for (const line of Array.from(el.children) as HTMLElement[]) {
      const on =
        t >= Number(line.dataset["start"]) && t < Number(line.dataset["end"]);
      line.style.opacity = on ? "1" : "0";
    }
  },
});
```

`emphasized` and `wordEmoji` match as said (case, accents and punctuation ignored, whole words; a phrase marks each of its words, its emoji on its last). `captionTop` and `freeAreas` measure a place for them ([layers.md](https://allyhub.com/docs/reference/sdk/layers.md)).

Subtitle files (`allyhub render --type srt,vtt`) come from each voiced part's line. A piece whose layers call `ctx.captions()` has its subtitles cut as the first such layer asked; otherwise by `CAPTION_CUE_DEFAULTS`: one line a cue of at most 42 characters (16 in Chinese, Japanese and Korean), 0.8 to 6 s, 2 frames before the next.

## J and L cuts

`part({ overlap: { before, after } })`, seconds from 0 to 10: `before` seconds of the part's speech play under the previous part's picture, `after` seconds under the next part's. The part's picture is shorter by them. Each is at most the part's lead (or tail) plus half its speech; the first part's `before` and the last part's `after` do nothing.

## Speech gain

`part({ speechGainDb })` (−40 to 24, default 0) turns the part's speech up or down after it is recorded ([sound.md](https://allyhub.com/docs/reference/sdk/sound.md#how-the-voice-was-recorded)), as an editor's clip volume: the recording's compressor does not even it out, and its room tone moves with it.

## Words files

`<name>.words.json` beside the recording (its extension replaced): the words said in it, in the order said, with times in seconds of the file. `allyhub words` writes it; an agent may correct it. A file that does not match the format is not read (AK-T001).

```json
{
  "version": 1,
  "language": "en",
  "duration": 14.2,
  "words": [
    { "text": "So", "start": 0.42, "end": 0.6, "confidence": 0.98 },
    { "text": "here", "start": 0.63, "end": 0.81 }
  ]
}
```

`version` is 1; `language` (a BCP 47 tag), `duration` (seconds) and `model` (what recognised it) are optional. Each word's `text` keeps its punctuation, `end` is at or after `start`, and `confidence` (0 to 1) is absent when recognition did not say. `$schema` may name a JSON Schema; any other field is an error.

## Edits files

A take's cut list, a JSON file named by `.edits("./edits/<name>.json")`. It removes exactly what it lists, without touching the recording; `{ "version": 1 }` removes nothing. Which words are fillers, retakes or false starts is the agent's to decide. A person cuts words and keeps them again in the video editor's script, which writes this file's `remove` alone, and only where the code names the file: a take without `.edits()` is read only there.

```json
{
  "version": 1,
  "take": "talk/host.mp4",
  "remove": [{ "from": 167, "to": 189, "text": "Now second, figure out" }],
  "sounds": [{ "from": 61.17, "to": 61.4 }],
  "inWord": [{ "from": 12.31, "to": 12.62 }],
  "maxPause": 0.35,
  "pad": { "before": 0.12, "after": 0.08 }
}
```

- `remove`: words, by their indices in the words file (both ends included), with their `text`. When the take is heard again, a range is found again by its text, nearest its old place.
- `sounds`: stretches of the file (seconds) between words; cut only between two kept words, never into one.
- `inWord`: stretches of the file cut even inside a kept word; each end moves only to the quietest 10 ms within 40 ms. One that reaches the first or last word of a part's stretch moves that end.
- `maxPause`: every pause between kept words longer than it (0.05 to 10 s) is shortened to it.
- `pad`: the silence a cut keeps, `before` it after the last kept word (default 0.12 s) and `after` it before the next kept word (default 0.08 s); never past half the silence.
- `take`: the recording it cuts, a note for people; the code names the file.
- `version` is 1; `$schema` may name a JSON Schema; any other field is an error.

Every cut lands in the silence, at the quietest 10 ms within 40 ms, on the frame grid, with 10 ms fades. The take's words, its footage and its sound follow the kept stretches. `allyhub words <take>` prints the take's words with their indices and the sounds between them; `allyhub words edits <take> <edits>` shows what an edits file keeps and removes and measures every join.
