SDK: 3D sceneries
A scenery is three.js the agent writes, in a module whose default export is defineScenery(…): AllyHub builds it once, poses it from time alone, draws it from cameras and measures it. This file covers that contract, cameras and their moves, and the scenery() layer with its screens. The commands are in allyhub help 3d.
A scenery is an item
Section titled “A scenery is an item”A scenery a person sees is listed in allyhub.yaml as kind: scenery (project.md). A person looks round it in Studio and allyhub dev’s page at scenery time 0, selecting a placed object (the outermost named one) and its named cameras, never editing it; a note on it names the object, its size, the point clicked and the view as an allyhub scenery view command. A scenery module the list does not name (one a video imports, a setting several share) is code, yours alone; the stage knows a scenery module by what it is (it calls defineScenery), not by its name.
- A video imports a scenery item by its path, as any module:
import room from "../room/room.ts". - A path a scenery reads that starts with
./or../(w.texture("./wood.jpg"),w.load("./desk.glb"), a model’s image) is from the scenery’s own module; any other is from the project’s root. An import (import desk from "./desk.glb") is the same. Kept in its folder and named from there, a scenery works wherever the folder goes. three(r186, 0.186.1),three/addons/…,three/webgpuandthree/tslare imported by name with nopackage.json: the CLI lends them, unless the project declares its ownthree. So are@allyhub/scenery-host/layer(everything in this file) and@allyhub/model(modeling.md). Their types come with the project’sallyhubonceallyhub/clientis in tsconfig.json’stypes, beside@allyhub/sdk/client; three.js’s are@types/three, which the project installs at the version lent (npm install -D @types/three@0.186.0).- glTF and GLB load (
.glb,.gltf,.bin,.ktx2), environments are.hdror.exr; all are media imports. CC0 models, HDRIs and textures:allyhub add search --type @polyhaven,allyhub add @polyhaven/<id>. AllyHub’s own 3D models and sceneries:allyhub add search --type 3d-model(orscenery),allyhub add 3d-model/<name>.
Units and axes. Right-handed, Y up, metres, angles in degrees everywhere in AllyHub’s API (three.js itself takes radians). A camera looks down its own −Z. A glTF model’s front is its +Z. Positions and sizes AllyHub prints are world space. three.js’s lights are physical: point and spot lights in candela, directional lights in lux, an emissive surface in nits.
defineScenery
Section titled “defineScenery”import * as THREE from "three";import { defineScenery, path } from "@allyhub/scenery-host/layer";import sofaModel from "./sofa.glb";
export default defineScenery({ async build(w) { w.renderer.toneMapping = THREE.ACESFilmicToneMapping; const sofa = await w.load(sofaModel); sofa.name = "sofa"; w.scene.add(sofa); const mixer = new THREE.AnimationMixer(sofa); const clips = sofa.userData["animations"] as THREE.AnimationClip[]; for (const clip of clips) mixer.clipAction(clip).play(); return { mixer }; }, update(made, t, w) { w.seek(made.mixer, t); }, cameras: { door: { position: [0, 1.5, 2.8], target: "sofa", lens: 26 } }, moves: { walk: path( [ [0, 1.5, 3], [0.4, 1.2, -0.2], ], { dur: 6, target: "sofa" }, ), },});build(w) runs once, awaited with every load it starts (60 s at most; in a video it counts in the piece’s loadLimit, AK-P002); what it returns is made. update(made, t, w) poses it at t, seconds of scenery time. cameras (checked when defined, AK-W004) and moves are by name, for the commands and for layers. bake is in Baked light. title is the item’s name where allyhub.yaml gives it none, read from the module as written (a string, not computed); without either, its file’s name.
From t alone. The same moment must draw the same pixels in any order (AK-D002). Date, performance.now, Math.random, crypto.getRandomValues and requestAnimationFrame follow the moment drawn; timers run only while it builds. Pose an AnimationMixer with w.seek(mixer, t) (every action reset, then setTime(t)), not a bare setTime. Name what is referred to (Object3D.name): cameras, moves, measurements and projectScenery take names, and a name no object has, or two have, is AK-W005.
renderer (all optional; SCENERY_RENDERER_DEFAULTS, the fastest drawing):
| Option | Default | What it does |
|---|---|---|
kind |
"webgl" |
"webgpu": WebGPURenderer (node materials, TSL) through its WebGL 2 back end; not baked yet |
antialias |
false |
Multisampled edges |
transmissionScale |
0.25 | The extra pass transmissive materials (glass) need, a share of the frame’s size, (0, 2] |
outputBufferType |
three’s | The buffer effects draw into (HalfFloatType for HDR effects) |
logarithmicDepthBuffer, reversedDepthBuffer |
false |
|
focus |
{ samples: 64, maxRadius: 64 } |
A camera’s depth of field: samples per pixel, the widest blur’s radius in px as drawn, and supersample (1–4; default 2 with antialias, else 1) |
Alpha, size and pixel ratio are AllyHub’s; set the rest on w.renderer in build (tone mapping, shadows, setEffects). A material whose shader does not compile, or needs more texture units than the renderer has (maps, environment, shadow-casting lights, screens lighting it), draws nothing: AK-W006, a warning over 16 units where the renderer has more.
w, the scenery’s host (SceneryHost):
| Field | What it is |
|---|---|
scene, renderer |
The THREE.Scene, and the renderer AllyHub made |
load(asset) |
A glTF or GLB as a Group, with the lent decoders (Draco, meshopt, KTX2); its animations on userData.animations. One that does not load is AK-W003 |
texture(asset, { colorSpace? }) |
A picture as a texture: sRGB, or "linear" for a data map (normal, roughness, AO, displacement) |
environment(asset, { background? }) |
An .hdr or .exr as the scene’s environment, and its background when asked |
seek(mixer, t) |
Poses an animation mixer at t |
object(name) |
The object of that name, undefined for none |
positionOf(name), boundsOf(name) |
Its box’s centre now; its box, from its vertices |
asset(path) |
A project file’s URL |
time(ref) |
In update: the scenery time of a word or anchor reference ("@tap", "@tap+0.2") of the part that draws it. Throws in build |
screen(name) |
A screen’s face now (its first mesh’s +Z side): point(x, y) from shares of its content, center, normal, up, width, height |
screenPicture(name) |
In update: a screen’s picture as a mipmapped texture, for a material of the scenery’s own; undefined for a screen drawn over the scenery |
An asset is an import or a path. Where there is no part to time a reference by (the allyhub scenery commands), its moments come from the video that draws the scenery (the one video of the project that does, or --piece’s), its scenery layers finding them as they do when drawing; --times @tap=2.4 gives them by hand, over the video’s.
Cameras
Section titled “Cameras”A CameraState points one of three ways, or looks down −Z, level:
{ position: [0, 1.5, 2.8], target: "sofa", lens: 26 } // at an object's box centre, followed as it moves{ position: [0, 1.5, 2.8], target: [0, 0.6, -1.4], fov: 40 } // at a point; a vertical fov in degrees{ position: [0, 1.5, 2.8], yaw: 30, pitch: -10 } // 30° left of −Z, 10° down{ position: [0, 1.5, 2.8], direction: [0, -0.2, -1], roll: 5 }| Field | Default | What it is |
|---|---|---|
position |
required | Metres |
target, direction, or yaw/pitch |
down −Z | yaw about +Y (90 looks down −X); pitch above the horizon |
up |
[0, 1, 0] |
A view straight along up cannot be drawn (AK-W004) |
roll |
0 | Degrees about the view, counterclockwise |
lens |
26 | 35 mm-equivalent focal length, mm |
lensFrame |
"diagonal" |
What the lens is measured on: the frame’s diagonal (CIPA), "long", "short", "phone" ({ aspect: 4/3, keep: "long" }, a phone’s “26 mm”), or { aspect, keep } |
fov |
Vertical degrees, instead of a lens | |
near, far |
0.05, 500 | Metres |
aperture |
none (a pinhole) | An f-number in the same 35 mm-equivalent terms as lens (a phone’s f/1.8 is about f/8). Needs focus |
focus |
A distance along the view (m), a point, or an object’s name | |
ev |
none | Exposure, EV at ISO 100: the light is scaled by exposureOf(ev), 1 / (1.2 · 2^ev), before the tone curve, in place of toneMappingExposure |
CAMERA_DEFAULTS holds them. The aspect is always the frame’s. fieldOfView, diagonalLens and lensForVertical convert (three.js’s setFocalLength measures on another frame). allyhub scenery solve finds the camera that put points of the scenery at pixels of a picture.
A move (Move) is (t, view?) => CameraState, a function of scenery time alone; view (SceneryView) gives pointOf(name), time(ref), the frame’s aspect and screen(name). Every moment a move takes (MoveTime) is seconds or a word or anchor reference of the part that draws the scenery.
| Function | What it makes |
|---|---|
keys([{ t, position?, target?, yaw?, pitch?, lens?, fov?, roll?, aperture?, focus?, ev?, ease? }], { lensSpace? }) |
Each key at its time: position along a curve, speed continuous, a repeated position holds; ease runs the stretch into its key; lens in log focal length (or lensSpace: "fov"); aperture and ev in every key or none |
path(points, { dur | speed, start?, ease?, target?, ahead?, frame?, alpha? }) |
Through the points at constant speed; target a point, a name or "ahead" (default); frame: "rmf" carries up untwisted (PATH_DEFAULTS) |
orbit({ around, radius, height?, from, to, dur, start?, ease?, target? }) |
Round a point or a name; 0° on its +Z side, 90° on +X, not wrapped |
shots([{ from, camera }, …]) |
Shots joined by cuts, each a camera or a move; measurements of motion stop at its cuts (cutsOf) |
blend(a, b, { from, to, ease? }) |
a before from, b after to, mixed between (lens at a steady zoom ratio, ev in stops); ease default "smooth" |
focusScreen(name, [x, y, w, h], { fill?, lens?, fov?, yaw?, pitch?, ev?, lensFrame? }) |
A move framing a rectangle of a screen’s content (shares from its top left), square on or yaw/pitch off it |
fitCamera({ points | boxes | sphere, aspect, direction | yaw/pitch, lens | fov, fill?, margins?, adjust?, … }) |
A CameraState framing them exactly; adjust: "lens" keeps position and solves the lens (FIT_DEFAULTS) |
mixCameras(a, b, x, view, what) |
The camera x of the way from a to b, as blend mixes them |
withCuts(camera, move) |
move with camera’s cuts, so a move of the agent’s own wrapped round shots() keeps them |
Ease is "linear", "in", "out", "in-out", "smooth" or a function on 0–1 (easeValue). aimed({ overshoot?, peak?, primary?, vary?, seed?, index? }), an aimed movement’s progress (past 1 and back), is an ease too; aimAt(shape, x) and rise(s, peak) are its parts (AIM_DEFAULTS).
A shot holds a camera state or a move, not a name: a scenery’s own are room.cameras?.["door"] and room.moves?.["walk"].
camera: shots([ { from: 0, camera: { position: [0, 1.5, 2.8], target: "sofa", lens: 26 } }, { from: "@demo", camera: blend(wide, focusScreen("monitor-screen", [0.62, 0.05, 0.1, 0.06]), { from: "@demo", to: "@click" }) },]),handheld
Section titled “handheld”handheld(camera, opts) is a camera or a move held in a hand: a Move of t alone, the same for a seed, another hand in each shot of shots() (whose cuts it keeps), its lens and focus kept. Amounts are standard deviations. Defaults: HANDHELD_DEFAULTS, STABILIZER_DEFAULTS, AUTOFOCUS_DEFAULTS, AUTO_EXPOSURE_DEFAULTS (defaults.md).
| Option | What it is |
|---|---|
angle, roll, shift, sway |
The sway: yaw and pitch degrees, roll as a share of them, metres on each axis; its spectrum round sway Hz, falling as 1/f² |
tremor |
{ angle, frequency, spread, vary, swell, period }: a band round frequency Hz that swells |
drift |
{ angle, shift, period }: the framing’s slow drift |
stabilizer |
{ strength, cutoff }: takes strength of the turning out above cutoff Hz. Absent: none |
reframes |
[{ at, to?, turn?, shift?, duration?, overshoot? }]: an aimed move to a camera, or by a Turn [yaw, pitch, roll?] and Shift [right, up, forward?]; reframe sets defaults |
jolts |
[{ at, turn? }]: the hand knocked, ringing out; jolt sets their defaults |
autofocus |
With an aperture and focus: rests miss off and hunts after cuts, reframes and jolts. Absent: none; {} the defaults |
exposure |
With an ev: follows a reframe’s ev late, is off at a cut and settles, breathes. Absent: none |
seed |
Another seed is another hand |
A phone
Section titled “A phone”PHONE_CAMERA (@allyhub/sdk/scenery) names a phone’s main camera as the options of each mechanism; phoneCamera(over) changes any value and keeps the rest. lens goes into a camera state, sensor into scenery({ sensor }), hand into handheld(), and the marks it leaves on its picture (grain, softness, sharpening) into video()’s capture (video.md). Its motionBlur is null: each sample draws a frame again (8 made a phone-filmed short take three times as long to render), and a hand’s moves barely show it. The sensor’s grade follows what fills the frame, as a phone’s exposure does: on a room or a person its room grade (a gentle contrast, a little more colour); as a screen comes to fill the frame, its screen grade (the phone exposes for the lit screen: its white a neutral light grey, its middle tones and colours deep), eased in between share[0] and share[1] of the frame filled (default 50% and 80%), frame by frame. Each grade is a brightness curve (curve, hues kept), then saturation and hue; it grades the scenery layer alone, not captions laid over it; grade: null for none. The screen’s grade and the capture’s marks are measured against phone photos of a laptop’s screen (fine grain kept in the whites, crisp sharpened text). Each is a starting point, not a look.
import { part, video } from "@allyhub/sdk";import { phoneCamera } from "@allyhub/sdk/scenery";import { handheld, scenery, shots } from "@allyhub/scenery-host/layer";import room from "../room/room.ts";
const phone = phoneCamera({ hand: { tremor: { angle: 0.15 } } });const camera = handheld( shots([ { from: 0, camera: { ...phone.lens, position: [0, 1.4, 1.2], target: "sofa", focus: "sofa", ev: 7.4, }, }, ]), { ...phone.hand, reframes: [{ at: 2, turn: [-2.5, 1] }] },);
export default video({ format: "9x16", capture: phone.capture, parts: () => [ part({ key: "look", dur: 5, layers: [scenery(room, { camera, sensor: phone.sensor })], }), ],});The scenery() layer
Section titled “The scenery() layer”scenery(scenery, opts) is a graphic layer. The scenery is built once when the output loads and posed at each frame’s scenery time.
import { custom, part, video } from "@allyhub/sdk";import { projectScenery, scenery, screen,} from "@allyhub/scenery-host/layer";import room from "../room/room.ts";import demo from "../demo/demo.ts";
export default video({ format: "16x9", parts: () => [ part({ key: "walk", dur: 6, layers: [ scenery(room, { camera: "walk", name: "room", screens: { "tv-screen": screen(demo, { start: 1 }) }, }), custom({ render(el, t) { const sofa = projectScenery("room", "sofa", t, el); if (sofa) el.style.transform = `translate(${String(sofa.x)}px, ${String(sofa.y)}px)`; }, }), ], }), ],});| Option | Default | What it does |
|---|---|---|
camera |
required | A CameraState, the name of one of the scenery’s cameras or moves, or a move |
box |
the frame | [x, y, w, h] as fractions of the frame; the camera’s aspect is the box’s |
background |
the scenery’s own | A colour, or "transparent"; without it what build set as the scene’s background, else transparent |
time |
{ from: 0 } |
The scenery’s time from the film’s: { from } at the film’s start, or (filmT) => number. It runs on across parts and cuts |
quality |
{ scale: 1, focusLevels: 8 } |
scale: the share of the layer’s size each way it is drawn at (above 1 supersamples; 0.5 for a quicker draft, softer). focusLevels: Gaussian levels a screen’s depth of field is mixed from. SCENERY_LAYER_DEFAULTS |
sensor |
none | A phone’s sensor filming it: the moiré its Bayer grid makes of the screens’ pixels. { photosites, lens, moire, chroma, distortion }, SENSOR_DEFAULTS |
screens |
none | { <object name>: screen(…) } (below) |
props |
none | SceneryProps this layer brings in (below) |
name |
"scenery" |
What projectScenery and the facts call it |
at, until, dur, z |
As every layer (video.md) |
projectScenery(layer, target, t, el), called in a layer’s render with its own element, gives where a point or a named object of the scenery layer called layer in the same part is at part time t: { x, y } in frame px, inView, depth (m); undefined when no such layer shows then.
Props (SceneryProp: { name, build(w), update?(object, t, w) }) are things a layer brings in (a stand-in, a pointing rod): built after the scenery and its bake (never in it), posed from t after the scenery’s update, shown only in the layers that add them. One a hand moves takes its path from the SDK’s handPath, tap or drag (layers.md, “A person’s input”), as the registry’s pointing-prop does.
A scenery inside another video. A video item whose picture is a scenery can be shown in a film: take(scene).range(from, to) plays a stretch of it and footage({ camera }) draws its scenery layers from a camera of the film’s (cameras by layer name for several, quality: { scale } for this film only), on the item’s clock. See footage.md and speech.md.
Screens
Section titled “Screens”screen(content, opts) puts content on a named object of the scenery (its first mesh’s +Z face, the content’s top left at the geometry’s −x, +y). The content is a video item of the project (its piece, imported: its parts, timing, anchors and components its own; pictures only, not its sound, and not footage yet) or graphic layers drawn on size px, whose windows and time props are seconds of the screen’s time or references of the part.
| Option | Default | What it does |
|---|---|---|
start |
0 | When the screen’s time is 0: scenery seconds, or a reference of the part ("@demo") |
cue |
0 | The screen’s own moment that plays at start: seconds, or (a video) its own reference ("@click", a part’s key) |
cues |
Instead: { <its moment>: <film reference or scenery seconds> }; between two cues it plays at the speed that joins them |
|
time |
the scenery’s | Instead: { from } or (t) => number from scenery time |
size |
[1920, 1080] |
Layers only: the content’s pixels (a video uses its format’s) |
dur |
to the layer’s end | Layers only |
background |
"#000000" |
Behind the content, or "transparent" |
pixels |
{ density: 2, rows: 0.15, mesh: 0.09 } |
Its physical pixels, for a sensor’s moiré, and mesh (0–1, 0 none): the fine grid of faint lines a phone close to a screen shows, every three of its pixels (PANEL_DEFAULTS) |
glass |
none | { reflectance?, roughness?, scale? }: what its glass reflects of the scenery over its picture, by Fresnel; {} the defaults (GLASS_DEFAULTS) |
inScenery |
true |
Drawn in the scenery: { brightness?, light? } (below); false: laid over the scenery |
import { custom, part, video } from "@allyhub/sdk";
// Made only to be shown on a screen.export default video({ format: "16x9", deliverable: false, parts: () => [ part({ key: "open", dur: 4, layers: [ custom({ render: (el, t) => (el.textContent = `${String(Math.round(t))} new`), }), ], }), ],});Before start a video holds its first frame, after its end its last. A video that shows itself through any chain is AK-M010; one made only for a screen is video({ deliverable: false }).
In the scenery (where the browser has Chrome’s HTML-in-Canvas, as AllyHub’s has) the content is the mesh’s texture, drawn with the exposure, depth of field, glass and moiré, and it lights the scenery’s standard and physical materials (at most 2 screens). brightness (default 1) is its white, scene-linear: nits in a scenery lit in physical units with an ev. light: false lights nothing; light: { cells, specular, base } sets the cells of its diffuse light ([8, 5]), leaves out its reflection (specular: false), and with base: { brightness?, color? } adds only what differs from steady light the scenery already has from it (SCREEN_LIGHT_DEFAULTS). material.userData.screenLight = false (or { specular: false }) keeps a material out. Without HTML-in-Canvas (or with ALLYHUB_HTML_IN_CANVAS=0) the mesh is a hole with the content laid over it at its own colours: AK-W007, unless asked with inScenery: false. An element it cannot show (in the browser’s top layer, preserve-3d planes it cannot order, over the largest texture) is AK-W008.
Text on a screen is left out of the frame’s text facts; scenery-screen reports its smallest text instead.
Baked light
Section titled “Baked light”allyhub scenery bake <item> bakes the scenery’s light (direct, shadows, bounces, emissive surfaces, the environment) into a lightmap once, kept in .allyhub/cache/bake/ by the scene’s digest and the renderer that made it, and laid on the scenery each time it is built. A bake made on a GPU is laid only by that GPU; one made on the CPU by every renderer. A scene that changed is drawn unbaked until baked again. How it bakes is defineScenery({ bake }) (BakeOptions: density, samples, bounces, weight, skip, ignore, posedAt, probe, …; BAKE_DEFAULTS).
Measuring
Section titled “Measuring”allyhub scenery <item> (objects, sizes, costs, lights, cameras, moves, bake), scenery view, map, see, measure, motion, solve and bake draw or measure the scenery module alone, without its screens (allyhub frames on a video shows them); inspect fidelity compares each screen drawn in the scenery with its own element.
The check reports each scenery layer’s facts, printed by allyhub inspect: scenery-contact, scenery-support, mesh-health, camera-clearance, camera-motion, scenery-screen (each screen’s share, scale, angle, hidden share, smallest text, moiré, cue speeds), scenery-screen-fidelity and scenery-bake. frames cuts sheets a scenery camera’s cuts.
Rules a problem is reported under: AK-W001 (it does not build, or update throws), AK-W002 (WebGL cannot be made, or is not SwiftShader where the CPU was asked for), AK-W003 (a file does not load), AK-W004 (a camera cannot be drawn), AK-W005 (a name names no object, or two), AK-W006 (a shader does not compile, or needs too many texture units), AK-W007 (a screen drawn over the scenery, a warning), AK-W008 (a screen in the scenery cannot show its element).
Frames are drawn on the GPU on a Mac and on the CPU (SwiftShader, the same pixels on every computer) elsewhere; ALLYHUB_GPU=0 or 1 chooses.