SDK: Speech
What a part says: a recording (take()), text to speech (say()) or a stretch of another video item (take(piece)). Its words time the part, place its layers and sounds, duck the music and make the captions. This file covers those, word references, anchors, shown-and-said text, captions and the words and edits files.
import { part, say, take, video } from "@allyhub/sdk";
import vo from "./vo/en.mp3";
export default video({ format: "16x9", parts: ({ s }) => [ part({ key: "hook", speech: take(vo).saying(s("launch.film.hook.say")) }), part({ key: "how", speech: take("./vo/en.mp3").saying(s("launch.film.how.say")).speed(1.1), }), part({ key: "host", speech: take("./host.mp4") .range("So here is", "every time") .edits("./edits/host.json") .denoise(12) .roomTone({ limitDb: 4, fade: 0.06 }), }), part({ key: "cta", speech: say(s("launch.film.cta.say"), { voice: "rachel", speed: 1.05 }), }), ],});A path written as a string (take("./host.mp4"), .edits("./edits/host.json")) is relative to the module that wrote it, also once the part is spread into another piece (project.md).
A part’s length
Section titled “A part’s length”A part with speech lasts lead + speech + tail: lead before the first word (default 0.2 s) and tail after the last (default 0.3 s), less any overlap (J and L cuts). A part without speech lasts dur (default 4 s). The defaults are PART_DEFAULTS, changed for the project by defaults.part in allyhub.config.ts and for one video by video({ defaults: { part } }).
take(file)
Section titled “take(file)”A recording, audio or video (an import, or a path), with its words in <name>.words.json beside it (Words files). allyhub words <file> hears it on this computer and writes that file. A take without one is AK-T001.
Alone, a take is the whole recording, from its first word to its last. Each method returns a new take:
.range(from, to?): the stretch from the first wordsfromto the end of the firsttoafter them (phrases; case and punctuation ignored); withoutto, to the end. A phrase not found is AK-T003..saying(text): the stretch that says this line, usuallys(key), found by its words, misheard words tolerated; the line’s spelling is the caption. Parts that take the same file each search after the previous part’s stretch. A line not found is AK-T003..edits(path): a cut list, a.jsonfile (Edits files). One that is not there, or does not match the format, is AK-T005: the take would play uncut..denoise(db?): reduces its noise bydb(over 0, up to 97; default 12) with ffmpeg’s FFT denoiser, in the render and the preview..roomTone(tone): how the part’s lead and tail are filled where the take has no silence of its own round the part’s words (Room tone)..speed(times): plays ittimesas fast, its pitch kept (0.5 to 2); its words, the part’s length and everything timed by them follow. The file is not changed. A part whosefootage()shows the take’s own picture cannot change its speed (part()throws).
With .edits(), range and saying look only at the words the edits keep: a phrase or line said only in removed words is AK-T006. A take a voice tool generated keeps the text it was made from in its .gen.json: a part whose line is not in that text is AK-T007.
Room tone
Section titled “Room tone”A part’s lead and tail play the take’s own room round its words. Where that runs out, a room tone fills them:
- The fill is the quietest steady stretch the build measured, or
{ from, to }(seconds of the file, both given, at least 0.05 s apart), looped..roomTone(false)leaves digital silence. - Its level is matched to the floor of the take’s own room beside the join (the mean power of the quietest quarter of the 10 ms frames within 0.3 s), at most
limitDbeither way (0 to 60; default 6).match: falseplays it as recorded;gainDb(−60 to 24) sets its gain instead. Where the take keeps under 0.1 s of its own room at a join, or its levels are not known, it plays as recorded. - It cross-fades with the take over
fadeseconds (0 to 1; default 0.04), equal-power.
The values are ROOM_MATCH (defaults.md). allyhub inspect timeline prints each fill’s gain and where it came from.
take(piece)
Section titled “take(piece)”A stretch of another video item of the project, imported, with a clock of its own:
import { part, video } from "@allyhub/sdk";
export default video({ format: "16x9", deliverable: false, parts: () => [ part({ key: "desk", dur: 6, anchors: { turn: 1.2, press: [3, 4.5] } }), ],});import { footage, part, take, video } from "@allyhub/sdk";
import scene from "../scene/scene.ts";
export default video({ format: "9x16", parts: () => [ part({ key: "cut", speech: take(scene).range("@turn", "@/press"), layers: [footage()], }), ],});.range(from, to?)takes seconds of its clock, an anchor of one of its parts ("@turn","@/turn","@turn+0.2") or a part’s key; withoutto, to its end. A reference it does not have is AK-T003.- Its ends go on the frame grid, never into a word they are not in (a stretch that splits a word is a
cut-in-wordfact). The part lasts the stretch exactly, with no lead or tail; it hears the item’s speech, room tone and sound effects (not its music), says its words and has its anchors. footage()without a file draws its picture in sync, in the part’s frame whatever the item’s own format (footage.md).deliverable: falsemarks an item that is only shown inside others.
say(text, opts?)
Section titled “say(text, opts?)”Text to speech. allyhub voice makes the missing files with the cloud voice tool, each in assets/media/gen/say/<hash> with its words and its record (each tool’s description says what it costs). A say without its file is AK-T002; until it has one, the part lasts about as long as the line takes to say (SPEECH_RATE_DEFAULTS: 0.38 s a word, 0.22 s a Chinese, Japanese or Korean character; at least 1 s), and its anchors are placed in proportion along the line.
voice: the voice tool’s voice id (allyhub ai-media find-voicesfinds them); default the tool’s own.tool: the voice tool; default"speak"("clone-voice"speaks in the voice of a recording you give inparams;allyhub help ai-media <tool>lists its input).params: more input for the tool, plain JSON: fields its schema has besides the text and voice (textandvoiceIdthrow).speed: 0.5 to 2, default 1: plays the file faster or slower, pitch kept, as.speed()does. Not sent to the tool, so the same file.
The file is named by a hash of the text read aloud, the voice, the tool and the params: a changed line needs a new file. Text from s(key) is read aloud as its said side (Shown and said).
Word references
Section titled “Word references”A word reference names a moment of a part by what is said there. A layer’s at and until, a part’s still, a sound’s at, a footage punchIn and every prop declared time() take one; a layer reads one with ctx.at(ref), in seconds from the part’s start.
| Reference | Moment |
|---|---|
"inbox" |
The start of the first time “inbox” is said in the part |
"inbox@2" |
The second time |
"inbox+0.2", "inbox@2-0.35" |
Shifted by seconds |
"@name", "@/name", "@name+0.2" |
An anchor’s start, a stretch’s end, shifted |
1.8, "1.8" |
Seconds from the part’s start |
A reference may be a phrase; case and punctuation are ignored. With say or saying a reference is found in the line first, so a misheard word still lands where it is said. A reference the part does not say, or any word in a part without speech, is AK-T004.
Anchors
Section titled “Anchors”An anchor is a name written in the line a part says, in each language’s script (project.md). The code refers to the name, so a reworded or translated line keeps what hangs on it.
parts: - id: how say: "@{step1}Paste your form link. We @{step2}match the answers you @{have}already have@{/have}."| In the line | In code | Moment |
|---|---|---|
@{name} |
"@name" |
A point: the start of the word after it (the end of the last word, at the line’s end) |
@{name}…@{/name} |
"@name", "@/name" |
A stretch: the start of its first word, the end of its last |
"@name+0.2", "@/name-0.1" |
Shifted by seconds |
import { custom, part, say, type MessageReader } from "@allyhub/sdk";
export const how = (s: MessageReader) => part({ key: "how", speech: say(s("launch.film.how.say")), sounds: [{ at: "@step1", sound: "tick" }], layers: [ custom({ at: "@step2", until: "@/have+0.3", mount(el, ctx) { el.dataset["step"] = String(ctx.at("@step1")); el.dataset["film"] = String(ctx.film?.at("@step1") ?? 0); }, render() {}, }), ], });- An anchor is looked up in the line the part says, so the speech comes from that script line:
say(s("<prefix>.<part>.say"))ortake(…).saying(s(…)). A name the line does not have is AK-T004. - A name is letters, digits,
_and-, once in a line; stretches may overlap. A name written twice, an end without a start, a stretch with no words and an anchor in a plural are AK-M007. An anchor goes before a<shown|said>piece, not inside it. - An anchor lands by the line’s letters matched to the words of its sound.
ctx.at("@name")is seconds from the part’s start;ctx.film.at("@name")is film seconds and takes an anchor of any part.allyhub inspect timelineprints where each landed.
A part’s own anchors
Section titled “A part’s own anchors”A part names moments in seconds from its start, with no speech needed: part({ anchors: { press: 7.1, map: [9.6, 13.4] } }). A number is a point, [from, to] a stretch; "@press", "@map", "@/map" and "@press+0.2" work as above. A name is the line’s or the part’s own, not both: a name in both is AK-T004, and "@name" is the part’s own.
Shown and said
Section titled “Shown and said”A piece of a script’s say shown one way and said another is <shown|said>: from <12|twelve> fields, <AI|A I>. The line as shown is what s() returns, titles and captions show; the line as said is what say reads aloud and what saying looks for in a recording. Either side may be empty: <™|> is shown and not said, <|[laughs]> said and not shown. In assets/text/<lang>.json, the same is a { "display", "speak" } value.
Captions
Section titled “Captions”Burned-in captions are a graphic layer in each part that has them: a starter component (captionWord, captionLine, captionSubtitle, added with allyhub add), a component of the project’s, or custom(). In any layer:
ctx.words()is the part’s words on its clock,{ text, start, end, confidence? }; empty without speech.ctx.captions(opts?)is the cues heard over the part, on its clock:
type CaptionTrackCue = { start: number; end: number; text: string; words: { text: string; start: number; end: number }[];};| Option | Default | What it is |
|---|---|---|
maxChars |
32 (16 for Chinese, Japanese and Korean) | Most characters in a cue |
maxWords |
none | Most words in a cue |
keep |
none | Phrases a cue never breaks inside, matched as said; the cue round one may run over the limits |
close |
0.4 | A gap between two cues shorter than this many seconds closes |
edges |
false | true: the first cue from the part’s start, the last to its end |
Cues come from the part’s speech (a cut take’s kept words, spelled as the script when there is one), break at sentence and clause ends, then by length, and never span two parts. In a J or L cut, the words heard under a neighbour’s picture are there too. Where they go and which words they stress is the layer’s code:
import { custom } from "@allyhub/sdk";import { emojiEl, emphasized, h, wordEmoji } from "@allyhub/sdk/dom";
export const captions = custom({ mount(el, ctx) { for (const cue of ctx.captions({ maxChars: 20, edges: true })) { const line = h("div", { opacity: 0 }); line.dataset["start"] = String(cue.start); line.dataset["end"] = String(cue.end); const stressed = emphasized(cue.words, ["free", "two minutes"]); const emoji = wordEmoji(cue.words, { free: "🎁" }); cue.words.forEach((w, i) => { const color = stressed[i] ? "var(--ak-color-accent)" : "inherit"; line.append(h("span", { color }, [`${w.text} `])); const e = emoji[i]; if (e) line.append(emojiEl(e)); }); el.append(line); } }, render(el, t) { for (const line of Array.from(el.children) as HTMLElement[]) { const on = t >= Number(line.dataset["start"]) && t < Number(line.dataset["end"]); line.style.opacity = on ? "1" : "0"; } },});emphasized and wordEmoji match as said (case, accents and punctuation ignored, whole words; a phrase marks each of its words, its emoji on its last). captionTop and freeAreas measure a place for them (layers.md).
Subtitle files (allyhub render --type srt,vtt) come from each voiced part’s line. A piece whose layers call ctx.captions() has its subtitles cut as the first such layer asked; otherwise by CAPTION_CUE_DEFAULTS: one line a cue of at most 42 characters (16 in Chinese, Japanese and Korean), 0.8 to 6 s, 2 frames before the next.
J and L cuts
Section titled “J and L cuts”part({ overlap: { before, after } }), seconds from 0 to 10: before seconds of the part’s speech play under the previous part’s picture, after seconds under the next part’s. The part’s picture is shorter by them. Each is at most the part’s lead (or tail) plus half its speech; the first part’s before and the last part’s after do nothing.
Speech gain
Section titled “Speech gain”part({ speechGainDb }) (−40 to 24, default 0) turns the part’s speech up or down after it is recorded (sound.md), as an editor’s clip volume: the recording’s compressor does not even it out, and its room tone moves with it.
Words files
Section titled “Words files”<name>.words.json beside the recording (its extension replaced): the words said in it, in the order said, with times in seconds of the file. allyhub words writes it; an agent may correct it. A file that does not match the format is not read (AK-T001).
{ "version": 1, "language": "en", "duration": 14.2, "words": [ { "text": "So", "start": 0.42, "end": 0.6, "confidence": 0.98 }, { "text": "here", "start": 0.63, "end": 0.81 } ]}version is 1; language (a BCP 47 tag), duration (seconds) and model (what recognised it) are optional. Each word’s text keeps its punctuation, end is at or after start, and confidence (0 to 1) is absent when recognition did not say. $schema may name a JSON Schema; any other field is an error.
Edits files
Section titled “Edits files”A take’s cut list, a JSON file named by .edits("./edits/<name>.json"). It removes exactly what it lists, without touching the recording; { "version": 1 } removes nothing. Which words are fillers, retakes or false starts is the agent’s to decide. A person cuts words and keeps them again in the video editor’s script, which writes this file’s remove alone, and only where the code names the file: a take without .edits() is read only there.
{ "version": 1, "take": "talk/host.mp4", "remove": [{ "from": 167, "to": 189, "text": "Now second, figure out" }], "sounds": [{ "from": 61.17, "to": 61.4 }], "inWord": [{ "from": 12.31, "to": 12.62 }], "maxPause": 0.35, "pad": { "before": 0.12, "after": 0.08 }}remove: words, by their indices in the words file (both ends included), with theirtext. When the take is heard again, a range is found again by its text, nearest its old place.sounds: stretches of the file (seconds) between words; cut only between two kept words, never into one.inWord: stretches of the file cut even inside a kept word; each end moves only to the quietest 10 ms within 40 ms. One that reaches the first or last word of a part’s stretch moves that end.maxPause: every pause between kept words longer than it (0.05 to 10 s) is shortened to it.pad: the silence a cut keeps,beforeit after the last kept word (default 0.12 s) andafterit before the next kept word (default 0.08 s); never past half the silence.take: the recording it cuts, a note for people; the code names the file.versionis 1;$schemamay name a JSON Schema; any other field is an error.
Every cut lands in the silence, at the quietest 10 ms within 40 ms, on the frame grid, with 10 ms fades. The take’s words, its footage and its sound follow the kept stretches. allyhub words <take> prints the take’s words with their indices and the sounds between them; allyhub words edits <take> <edits> shows what an edits file keeps and removes and measures every join.