Skip to content
Get started

SDK: Sound

Nothing makes a sound unless the piece puts it there: each part's speech (speech.md), the music, footage with audio: true and the sound effects a part places. Each stem is shaped, levelled to a loudness, the music dips under speech, and the sum is mastered. The render's mix and the preview's are the same computation, sample by sample. This file covers music, sound effects, the mix, the master and how the voice was recorded.

import { part, take, video } from "@allyhub/sdk";
import chime from "./sfx/chime.wav";
export default video({
format: "16x9",
music: {
file: "./music/song.mp3",
bpm: 113,
downbeat: 0.42,
play: [
{ from: { bar: 0 }, to: { bar: 7 } },
{ from: { bar: 3 }, to: { bar: 7 } },
],
end: "bar",
snap: true,
eq: [
{ type: "highpass", hz: 60 },
{
type: "lowpass",
hz: 20000,
keys: [
{ at: 0, hz: 1100 },
{ at: "@drop", hz: 1100 },
{ at: "@drop+0.3", hz: 20000 },
],
},
],
compress: { threshold: -6, ratio: 1.5 },
gain: [
{ at: 0, db: -4 },
{ at: "@drop", db: -4 },
{ at: "@drop+0.1", db: 0 },
],
},
mix: { music: -24, duck: { db: -6, hold: 1 }, musicFade: { out: 3 } },
master: { lufs: -16, truePeak: -1.5 },
mic: { setup: "phone", distance: 1.3, room: "bedroom" },
parts: ({ s }) => [
part({
key: "how",
speech: take("./vo/en.mp3").saying(s("launch.film.how.say")),
anchors: { drop: 2 },
sounds: [
{ at: "paste", sound: "tick", what: "step 1" },
{ at: 2.4, sound: "./sfx/ding.wav", gainDb: -3 },
{
at: "free-0.05",
sound: chime,
shape: { eq: [{ type: "highpass", hz: 200 }] },
},
],
}),
],
});

A path written as a string (the music’s file, a sound’s) is relative to the module that wrote it (project.md).

video({ music }): an audio file (an import, or a path), or MusicOptions. Any audio file is music: a song, or a bed of a room’s sound.

  • file: the music, or one per language ({ en: a, "zh-cn": b }); a language without an entry has none.

  • from: where in the file it starts, seconds; default 0.

  • play: stretches of the file, [{ from, to }], back to back; each place in seconds, { beat: n } or { bar: n }, and no to runs to the file’s end. The film moves on by each stretch’s exact length, so stretches cut on the beat keep the beat. Default the file from from to its end.

  • loop: true plays the whole file again and again, each seam crossed over 0.25 s; { from, to } plays that stretch again and again after play. Default false: it plays once.

  • seam: each seam’s equal-power cross-fade, seconds, before the join, so a downbeat there is heard whole; default 0.03 (0.25 with loop: true).

  • bpm: beats per minute, for snap, beats and bars; default the tempo the build measured.

  • downbeat: the file’s second of bar 0; bars and beats count from 0. Default the first beat measured, or 0 with a given bpm.

  • meter: beats a bar, for { bar } places and end: "bar"; default 4.

  • end: where it stops, "film" (the video’s end, the default), "bar" or "beat" (the last bar line or beat before it); its fade out ends there.

  • snap: parts without speech end on the music’s beats as the stretches lay them; default false.

  • gain: its level over the film, [{ at, db }], dB over its mix level, linear in dB between points and held outside them, added to its dip under speech.

  • filter: an ffmpeg audio filter chain run on the file as it is decoded, on the file’s clock. One chain; nothing that reads files, listens, loads plugins, makes sound or changes the timing.

  • eq: bands, in order, { type, hz, db?, q?, keys? }: highpass, lowpass, bandpass (q default 0.707), peak (q default 1), lowshelf, highshelf (q the slope, default 1). keys: [{ at, hz?, db?, q? }] move a band over the film.

  • compress: a compressor after the bands, { threshold, ratio?, knee?, attack?, release?, window? }, its threshold dB over the music’s own loudness; the rest COMPRESS_DEFAULTS (ratio 2, knee 6 dB, attack 0.01 s, release 0.15 s, window 0.01 s).

  • A moment of gain and keys (at) is film seconds, an anchor of any part ("@drop", "@drop-0.2", "@/drop"), a part’s key (its start), or { part, at }, a word reference or seconds in that part (speech.md). One that is not there is AK-T004.

  • A beat, a bar, snap or end without a tempo (no bpm and none measured) plays the file once, or ends it with the film, and snaps nothing; a fact says so.

  • Music shorter than the film leaves its end silent unless it loops; the music-length fact gives both lengths.

  • The music is levelled as shaped: filter, eq and compress change how it sounds, mix and gain how loud. The values are MUSIC_DEFAULTS (defaults.md).

A part places its sounds: part({ sounds: [{ at, sound, gainDb?, shape?, what? }] }).

  • at: a word reference, an anchor or seconds from the part’s start. A built-in sound’s transient lands there; a file starts there.
  • sound: a name (the built-in synthesized whoosh, tick or pop, or a name of the look’s sfx), or an audio file: an import, or a path (a string with . or / in it). A video with sound plays its sound.
  • gainDb: −60 to 24, over the mix’s sound-effect level; default 0.
  • shape: { filter?, eq?, compress? }, as the music’s, its bands without keys.
  • what: a note for people, what it marks (allyhub inspect timeline prints it).

The look’s sounds are named in allyhub.config.ts: defineConfig({ look: { sfx: { swoosh: "assets/media/sfx/swoosh.wav" } } }), each path from the project’s root; a name there replaces a built-in one. A name that is neither a sound nor a file, or a file that is not in the project, is AK-A001; a file with no sound AK-A003.

A file is levelled by its integrated loudness to mix.sfx; a sound too short or quiet for that (a click) by its K-weighted level over its whole length. The built-in sounds move by as much as mix.sfx moves from its default.

A footage layer’s own sound is mixed only with footage(file, { audio: true }); gainDb (default 0) changes its level over the mix’s footage level. A take’s own footage (footage() without a file) sounds as the part’s speech (footage.md).

video({ mix }): each stem’s loudness target in LUFS, and the music’s dip under words.

Field Default What it is
speech −16 Each part’s speech, with its room tone
music −20 The music while speech plays
musicAlone −16 The music where nobody speaks
sfx −22 Sound-effect files; the built-in sounds move by the same amount from −22
footage −16 Footage’s own sound where nobody speaks
footageUnderSpeech −22 Footage’s own sound while speech plays
duck { db: −8, attack: 0.3, release: 1, hold: 2 } How far (dB) the music dips under words, over attack seconds before a word and release after one, staying down through pauses shorter than hold
musicFade { in: 0.3, out: 1.5 } The music’s fades at the start and the end, seconds (the fade out at most a quarter of the video)

The values are MIX_DEFAULTS. A part’s own speech level is speechGainDb (speech.md).

video({ master }): a loudness in LUFS (−70 to 0), { lufs?, truePeak? } (truePeak −9 to 0 dBTP), or false. The default is −14 LUFS, with a −3 dBTP true peak (MASTER_DEFAULTS).

The mix’s integrated loudness is measured and one gain brings it to lufs; a limiter at 4 times the sample rate holds the true peak; what came out is measured again and the gain corrected by what the limiter took, at most 4 times, until within 0.1 LU (MASTER_MECHANICS). Nothing rides the level over time: the mix keeps its loudness range. false (or lufs: false) keeps the mix’s own level and only limits its peaks.

allyhub audio measures a video’s mix as the render makes it: its loudness, range and true peak, what the master did, silences, onsets against cuts, each part’s speech. allyhub render --type wav writes the mastered mix.

video({ mic }) and part({ mic }) say what heard the voice: a microphone of some pattern and distance, in a room, over the room’s own sound, through what a device does by itself, then the finishing done to it after. The speech is played through it in the mix, the same every time. A part’s mic replaces the video’s for that part; parts with the same setting are one recording, with one room tone under them while they play.

  • Nothing asked is "studio": a cardioid 0.2 m away, no room, no room tone, finished ("finish": a high-pass, a gentle de-esser and compressor). mic: false plays the files as they are.
  • A setup by name: "studio", "phone" (a phone held in front, a little below the face: two omni microphones, its band, its automatic gain, a small dead room and its room tone), "lav" (a clip-on on the chest, in a dead room, finished). Their values are SETUP_PRESETS, the rooms’ ROOM_PRESETS and the finishing’s POST_PRESETS (defaults.md).
  • Any value of a setup changes:
import { part, video } from "@allyhub/sdk";
export default video({
format: "16x9",
mic: {
setup: "lav",
room: { preset: "office", rt60: 0.3 },
noise: { below: 36, hum: { hz: 50, db: -50 } },
post: { preset: "finish", highpass: 80 },
also: [{ setup: "phone", distance: 0.8, gain: -10 }],
},
parts: () => [part({ key: "x", dur: 2, mic: "phone" })],
});
  • setup: the setup whose values fill what is not given; default "studio".
  • pattern: omni, subcardioid, cardioid, supercardioid, hypercardioid or figure8.
  • distance, angle, below: mouth to microphone (m); how far round from where the talker faces, and how far below the mouth (degrees).
  • response: the microphone’s own response, [Hz, dB] points, linear in log frequency. band: [low, high] Hz it records; under and over them it falls away.
  • stereo: two microphones this far apart (m), or false.
  • room: a name (bedroom, living-room, office, kitchen, treated, car, outdoor), a box { preset?, size: [w, h, d], walls, floor, ceiling, rt60, talker }, or false. A surface is a material (hard, plasterboard, plasterboard-ceiling, tiles, concrete, wood, carpet, carpet-thin, glass, curtains, ceiling-tiles, foam, upholstery, grass) or its absorption by octave from 125 Hz to 8 kHz. rt60 (s, at 500 Hz–1 kHz) scales the surfaces’ absorption until the room has it. talker is the mouth, m from the room’s corner, facing +z.
  • noise: the room tone, { below, tilt, rumble, top, hum } (LU under the voice at the setup’s own distance; dB per octave over 1 kHz; dB more below 120 Hz; the highest frequency, Hz; a mains hum { hz, db }), or false. Further away, the voice reaches the microphone weaker against the same room tone.
  • agc: the device’s automatic gain, { target, most, attack, release, window, gate } (target dB over the voice’s loudness; under gate dB the gain holds), or false.
  • post: the finishing, "finish", "phone-app", { preset?, highpass, eq, deess, compress } (thresholds dB over the voice’s loudness), or false.
  • source: what recorded the file already, { pattern, distance }; default a cardioid 0.2 m away (a generated voice). The setup adds only what differs.
  • also: more microphones that heard the same voice, [{ setup, …, gain }], mixed in at gain dB (default −12). The finishing is the first’s, on the sum.

The voice’s loudness is kept: a setting changes how it sounds, not how loud it is. A part’s mic that is not valid throws where part() is called, naming the field.

allyhub audio recording measures how a video’s voice is recorded (the room, the room tone, the spectrum, the dynamics), beside a reference file with --file, and --fit fits the video’s setting to that file and prints it for video({ mic }).

Was this page helpful?