SDK: Sound
Nothing makes a sound unless the piece puts it there: each part's speech (speech.md), the music, footage with audio: true and the sound effects a part places. Each stem is shaped, levelled to a loudness, the music dips under speech, and the sum is mastered. The render's mix and the preview's are the same computation, sample by sample. This file covers music, sound effects, the mix, the master and how the voice was recorded.
import { part, take, video } from "@allyhub/sdk";
import chime from "./sfx/chime.wav";
export default video({ format: "16x9", music: { file: "./music/song.mp3", bpm: 113, downbeat: 0.42, play: [ { from: { bar: 0 }, to: { bar: 7 } }, { from: { bar: 3 }, to: { bar: 7 } }, ], end: "bar", snap: true, eq: [ { type: "highpass", hz: 60 }, { type: "lowpass", hz: 20000, keys: [ { at: 0, hz: 1100 }, { at: "@drop", hz: 1100 }, { at: "@drop+0.3", hz: 20000 }, ], }, ], compress: { threshold: -6, ratio: 1.5 }, gain: [ { at: 0, db: -4 }, { at: "@drop", db: -4 }, { at: "@drop+0.1", db: 0 }, ], }, mix: { music: -24, duck: { db: -6, hold: 1 }, musicFade: { out: 3 } }, master: { lufs: -16, truePeak: -1.5 }, mic: { setup: "phone", distance: 1.3, room: "bedroom" }, parts: ({ s }) => [ part({ key: "how", speech: take("./vo/en.mp3").saying(s("launch.film.how.say")), anchors: { drop: 2 }, sounds: [ { at: "paste", sound: "tick", what: "step 1" }, { at: 2.4, sound: "./sfx/ding.wav", gainDb: -3 }, { at: "free-0.05", sound: chime, shape: { eq: [{ type: "highpass", hz: 200 }] }, }, ], }), ],});A path written as a string (the music’s file, a sound’s) is relative to the module that wrote it (project.md).
video({ music }): an audio file (an import, or a path), or MusicOptions. Any audio file is music: a song, or a bed of a room’s sound.
-
file: the music, or one per language ({ en: a, "zh-cn": b }); a language without an entry has none. -
from: where in the file it starts, seconds; default 0. -
play: stretches of the file,[{ from, to }], back to back; each place in seconds,{ beat: n }or{ bar: n }, and notoruns to the file’s end. The film moves on by each stretch’s exact length, so stretches cut on the beat keep the beat. Default the file fromfromto its end. -
loop:trueplays the whole file again and again, each seam crossed over 0.25 s;{ from, to }plays that stretch again and again afterplay. Default false: it plays once. -
seam: each seam’s equal-power cross-fade, seconds, before the join, so a downbeat there is heard whole; default 0.03 (0.25 withloop: true). -
bpm: beats per minute, forsnap, beats and bars; default the tempo the build measured. -
downbeat: the file’s second of bar 0; bars and beats count from 0. Default the first beat measured, or 0 with a givenbpm. -
meter: beats a bar, for{ bar }places andend: "bar"; default 4. -
end: where it stops,"film"(the video’s end, the default),"bar"or"beat"(the last bar line or beat before it); its fade out ends there. -
snap: parts without speech end on the music’s beats as the stretches lay them; default false. -
gain: its level over the film,[{ at, db }], dB over its mix level, linear in dB between points and held outside them, added to its dip under speech. -
filter: an ffmpeg audio filter chain run on the file as it is decoded, on the file’s clock. One chain; nothing that reads files, listens, loads plugins, makes sound or changes the timing. -
eq: bands, in order,{ type, hz, db?, q?, keys? }:highpass,lowpass,bandpass(qdefault 0.707),peak(qdefault 1),lowshelf,highshelf(qthe slope, default 1).keys: [{ at, hz?, db?, q? }]move a band over the film. -
compress: a compressor after the bands,{ threshold, ratio?, knee?, attack?, release?, window? }, its threshold dB over the music’s own loudness; the restCOMPRESS_DEFAULTS(ratio 2, knee 6 dB, attack 0.01 s, release 0.15 s, window 0.01 s). -
A moment of
gainandkeys(at) is film seconds, an anchor of any part ("@drop","@drop-0.2","@/drop"), a part’s key (its start), or{ part, at }, a word reference or seconds in that part (speech.md). One that is not there is AK-T004. -
A beat, a bar,
snaporendwithout a tempo (nobpmand none measured) plays the file once, or ends it with the film, and snaps nothing; a fact says so. -
Music shorter than the film leaves its end silent unless it loops; the
music-lengthfact gives both lengths. -
The music is levelled as shaped:
filter,eqandcompresschange how it sounds,mixandgainhow loud. The values areMUSIC_DEFAULTS(defaults.md).
Sound effects
Section titled “Sound effects”A part places its sounds: part({ sounds: [{ at, sound, gainDb?, shape?, what? }] }).
at: a word reference, an anchor or seconds from the part’s start. A built-in sound’s transient lands there; a file starts there.sound: a name (the built-in synthesizedwhoosh,tickorpop, or a name of the look’ssfx), or an audio file: an import, or a path (a string with.or/in it). A video with sound plays its sound.gainDb: −60 to 24, over the mix’s sound-effect level; default 0.shape:{ filter?, eq?, compress? }, as the music’s, its bands withoutkeys.what: a note for people, what it marks (allyhub inspect timelineprints it).
The look’s sounds are named in allyhub.config.ts: defineConfig({ look: { sfx: { swoosh: "assets/media/sfx/swoosh.wav" } } }), each path from the project’s root; a name there replaces a built-in one. A name that is neither a sound nor a file, or a file that is not in the project, is AK-A001; a file with no sound AK-A003.
A file is levelled by its integrated loudness to mix.sfx; a sound too short or quiet for that (a click) by its K-weighted level over its whole length. The built-in sounds move by as much as mix.sfx moves from its default.
Footage sound
Section titled “Footage sound”A footage layer’s own sound is mixed only with footage(file, { audio: true }); gainDb (default 0) changes its level over the mix’s footage level. A take’s own footage (footage() without a file) sounds as the part’s speech (footage.md).
The mix
Section titled “The mix”video({ mix }): each stem’s loudness target in LUFS, and the music’s dip under words.
| Field | Default | What it is |
|---|---|---|
speech |
−16 | Each part’s speech, with its room tone |
music |
−20 | The music while speech plays |
musicAlone |
−16 | The music where nobody speaks |
sfx |
−22 | Sound-effect files; the built-in sounds move by the same amount from −22 |
footage |
−16 | Footage’s own sound where nobody speaks |
footageUnderSpeech |
−22 | Footage’s own sound while speech plays |
duck |
{ db: −8, attack: 0.3, release: 1, hold: 2 } |
How far (dB) the music dips under words, over attack seconds before a word and release after one, staying down through pauses shorter than hold |
musicFade |
{ in: 0.3, out: 1.5 } |
The music’s fades at the start and the end, seconds (the fade out at most a quarter of the video) |
The values are MIX_DEFAULTS. A part’s own speech level is speechGainDb (speech.md).
The master
Section titled “The master”video({ master }): a loudness in LUFS (−70 to 0), { lufs?, truePeak? } (truePeak −9 to 0 dBTP), or false. The default is −14 LUFS, with a −3 dBTP true peak (MASTER_DEFAULTS).
The mix’s integrated loudness is measured and one gain brings it to lufs; a limiter at 4 times the sample rate holds the true peak; what came out is measured again and the gain corrected by what the limiter took, at most 4 times, until within 0.1 LU (MASTER_MECHANICS). Nothing rides the level over time: the mix keeps its loudness range. false (or lufs: false) keeps the mix’s own level and only limits its peaks.
allyhub audio measures a video’s mix as the render makes it: its loudness, range and true peak, what the master did, silences, onsets against cuts, each part’s speech. allyhub render --type wav writes the mastered mix.
How the voice was recorded
Section titled “How the voice was recorded”video({ mic }) and part({ mic }) say what heard the voice: a microphone of some pattern and distance, in a room, over the room’s own sound, through what a device does by itself, then the finishing done to it after. The speech is played through it in the mix, the same every time. A part’s mic replaces the video’s for that part; parts with the same setting are one recording, with one room tone under them while they play.
- Nothing asked is
"studio": a cardioid 0.2 m away, no room, no room tone, finished ("finish": a high-pass, a gentle de-esser and compressor).mic: falseplays the files as they are. - A setup by name:
"studio","phone"(a phone held in front, a little below the face: two omni microphones, its band, its automatic gain, a small dead room and its room tone),"lav"(a clip-on on the chest, in a dead room, finished). Their values areSETUP_PRESETS, the rooms’ROOM_PRESETSand the finishing’sPOST_PRESETS(defaults.md). - Any value of a setup changes:
import { part, video } from "@allyhub/sdk";
export default video({ format: "16x9", mic: { setup: "lav", room: { preset: "office", rt60: 0.3 }, noise: { below: 36, hum: { hz: 50, db: -50 } }, post: { preset: "finish", highpass: 80 }, also: [{ setup: "phone", distance: 0.8, gain: -10 }], }, parts: () => [part({ key: "x", dur: 2, mic: "phone" })],});setup: the setup whose values fill what is not given; default"studio".pattern:omni,subcardioid,cardioid,supercardioid,hypercardioidorfigure8.distance,angle,below: mouth to microphone (m); how far round from where the talker faces, and how far below the mouth (degrees).response: the microphone’s own response,[Hz, dB]points, linear in log frequency.band:[low, high]Hz it records; under and over them it falls away.stereo: two microphones this far apart (m), orfalse.room: a name (bedroom,living-room,office,kitchen,treated,car,outdoor), a box{ preset?, size: [w, h, d], walls, floor, ceiling, rt60, talker }, orfalse. A surface is a material (hard,plasterboard,plasterboard-ceiling,tiles,concrete,wood,carpet,carpet-thin,glass,curtains,ceiling-tiles,foam,upholstery,grass) or its absorption by octave from 125 Hz to 8 kHz.rt60(s, at 500 Hz–1 kHz) scales the surfaces’ absorption until the room has it.talkeris the mouth, m from the room’s corner, facing +z.noise: the room tone,{ below, tilt, rumble, top, hum }(LU under the voice at the setup’s own distance; dB per octave over 1 kHz; dB more below 120 Hz; the highest frequency, Hz; a mains hum{ hz, db }), orfalse. Further away, the voice reaches the microphone weaker against the same room tone.agc: the device’s automatic gain,{ target, most, attack, release, window, gate }(target dB over the voice’s loudness; undergatedB the gain holds), orfalse.post: the finishing,"finish","phone-app",{ preset?, highpass, eq, deess, compress }(thresholds dB over the voice’s loudness), orfalse.source: what recorded the file already,{ pattern, distance }; default a cardioid 0.2 m away (a generated voice). The setup adds only what differs.also: more microphones that heard the same voice,[{ setup, …, gain }], mixed in atgaindB (default −12). The finishing is the first’s, on the sum.
The voice’s loudness is kept: a setting changes how it sounds, not how loud it is. A part’s mic that is not valid throws where part() is called, naming the field.
allyhub audio recording measures how a video’s voice is recorded (the room, the room tone, the spectrum, the dynamics), beside a reference file with --file, and --fit fits the video’s setting to that file and prints it for video({ mic }).