Skip to content
Get started

SDK: 3D sceneries

A scenery is three.js the agent writes, in a module whose default export is defineScenery(…): AllyHub builds it once, poses it from time alone, draws it from cameras and measures it. This file covers that contract, cameras and their moves, and the scenery() layer with its screens. The commands are in allyhub help 3d.

A scenery a person sees is listed in allyhub.yaml as kind: scenery (project.md). A person looks round it in Studio and allyhub dev’s page at scenery time 0, selecting a placed object (the outermost named one) and its named cameras, never editing it; a note on it names the object, its size, the point clicked and the view as an allyhub scenery view command. A scenery module the list does not name (one a video imports, a setting several share) is code, yours alone; the stage knows a scenery module by what it is (it calls defineScenery), not by its name.

  • A video imports a scenery item by its path, as any module: import room from "../room/room.ts".
  • A path a scenery reads that starts with ./ or ../ (w.texture("./wood.jpg"), w.load("./desk.glb"), a model’s image) is from the scenery’s own module; any other is from the project’s root. An import (import desk from "./desk.glb") is the same. Kept in its folder and named from there, a scenery works wherever the folder goes.
  • three (r186, 0.186.1), three/addons/…, three/webgpu and three/tsl are imported by name with no package.json: the CLI lends them, unless the project declares its own three. So are @allyhub/scenery-host/layer (everything in this file) and @allyhub/model (modeling.md). Their types come with the project’s allyhub once allyhub/client is in tsconfig.json’s types, beside @allyhub/sdk/client; three.js’s are @types/three, which the project installs at the version lent (npm install -D @types/three@0.186.0).
  • glTF and GLB load (.glb, .gltf, .bin, .ktx2), environments are .hdr or .exr; all are media imports. CC0 models, HDRIs and textures: allyhub add search --type @polyhaven, allyhub add @polyhaven/<id>. AllyHub’s own 3D models and sceneries: allyhub add search --type 3d-model (or scenery), allyhub add 3d-model/<name>.

Units and axes. Right-handed, Y up, metres, angles in degrees everywhere in AllyHub’s API (three.js itself takes radians). A camera looks down its own −Z. A glTF model’s front is its +Z. Positions and sizes AllyHub prints are world space. three.js’s lights are physical: point and spot lights in candela, directional lights in lux, an emissive surface in nits.

import * as THREE from "three";
import { defineScenery, path } from "@allyhub/scenery-host/layer";
import sofaModel from "./sofa.glb";
export default defineScenery({
async build(w) {
w.renderer.toneMapping = THREE.ACESFilmicToneMapping;
const sofa = await w.load(sofaModel);
sofa.name = "sofa";
w.scene.add(sofa);
const mixer = new THREE.AnimationMixer(sofa);
const clips = sofa.userData["animations"] as THREE.AnimationClip[];
for (const clip of clips) mixer.clipAction(clip).play();
return { mixer };
},
update(made, t, w) {
w.seek(made.mixer, t);
},
cameras: { door: { position: [0, 1.5, 2.8], target: "sofa", lens: 26 } },
moves: {
walk: path(
[
[0, 1.5, 3],
[0.4, 1.2, -0.2],
],
{ dur: 6, target: "sofa" },
),
},
});

build(w) runs once, awaited with every load it starts (60 s at most; in a video it counts in the piece’s loadLimit, AK-P002); what it returns is made. update(made, t, w) poses it at t, seconds of scenery time. cameras (checked when defined, AK-W004) and moves are by name, for the commands and for layers. bake is in Baked light. title is the item’s name where allyhub.yaml gives it none, read from the module as written (a string, not computed); without either, its file’s name.

From t alone. The same moment must draw the same pixels in any order (AK-D002). Date, performance.now, Math.random, crypto.getRandomValues and requestAnimationFrame follow the moment drawn; timers run only while it builds. Pose an AnimationMixer with w.seek(mixer, t) (every action reset, then setTime(t)), not a bare setTime. Name what is referred to (Object3D.name): cameras, moves, measurements and projectScenery take names, and a name no object has, or two have, is AK-W005.

renderer (all optional; SCENERY_RENDERER_DEFAULTS, the fastest drawing):

Option Default What it does
kind "webgl" "webgpu": WebGPURenderer (node materials, TSL) through its WebGL 2 back end; not baked yet
antialias false Multisampled edges
transmissionScale 0.25 The extra pass transmissive materials (glass) need, a share of the frame’s size, (0, 2]
outputBufferType three’s The buffer effects draw into (HalfFloatType for HDR effects)
logarithmicDepthBuffer, reversedDepthBuffer false
focus { samples: 64, maxRadius: 64 } A camera’s depth of field: samples per pixel, the widest blur’s radius in px as drawn, and supersample (1–4; default 2 with antialias, else 1)

Alpha, size and pixel ratio are AllyHub’s; set the rest on w.renderer in build (tone mapping, shadows, setEffects). A material whose shader does not compile, or needs more texture units than the renderer has (maps, environment, shadow-casting lights, screens lighting it), draws nothing: AK-W006, a warning over 16 units where the renderer has more.

w, the scenery’s host (SceneryHost):

Field What it is
scene, renderer The THREE.Scene, and the renderer AllyHub made
load(asset) A glTF or GLB as a Group, with the lent decoders (Draco, meshopt, KTX2); its animations on userData.animations. One that does not load is AK-W003
texture(asset, { colorSpace? }) A picture as a texture: sRGB, or "linear" for a data map (normal, roughness, AO, displacement)
environment(asset, { background? }) An .hdr or .exr as the scene’s environment, and its background when asked
seek(mixer, t) Poses an animation mixer at t
object(name) The object of that name, undefined for none
positionOf(name), boundsOf(name) Its box’s centre now; its box, from its vertices
asset(path) A project file’s URL
time(ref) In update: the scenery time of a word or anchor reference ("@tap", "@tap+0.2") of the part that draws it. Throws in build
screen(name) A screen’s face now (its first mesh’s +Z side): point(x, y) from shares of its content, center, normal, up, width, height
screenPicture(name) In update: a screen’s picture as a mipmapped texture, for a material of the scenery’s own; undefined for a screen drawn over the scenery

An asset is an import or a path. Where there is no part to time a reference by (the allyhub scenery commands), its moments come from the video that draws the scenery (the one video of the project that does, or --piece’s), its scenery layers finding them as they do when drawing; --times @tap=2.4 gives them by hand, over the video’s.

A CameraState points one of three ways, or looks down −Z, level:

{ position: [0, 1.5, 2.8], target: "sofa", lens: 26 } // at an object's box centre, followed as it moves
{ position: [0, 1.5, 2.8], target: [0, 0.6, -1.4], fov: 40 } // at a point; a vertical fov in degrees
{ position: [0, 1.5, 2.8], yaw: 30, pitch: -10 } // 30° left of −Z, 10° down
{ position: [0, 1.5, 2.8], direction: [0, -0.2, -1], roll: 5 }
Field Default What it is
position required Metres
target, direction, or yaw/pitch down −Z yaw about +Y (90 looks down −X); pitch above the horizon
up [0, 1, 0] A view straight along up cannot be drawn (AK-W004)
roll 0 Degrees about the view, counterclockwise
lens 26 35 mm-equivalent focal length, mm
lensFrame "diagonal" What the lens is measured on: the frame’s diagonal (CIPA), "long", "short", "phone" ({ aspect: 4/3, keep: "long" }, a phone’s “26 mm”), or { aspect, keep }
fov Vertical degrees, instead of a lens
near, far 0.05, 500 Metres
aperture none (a pinhole) An f-number in the same 35 mm-equivalent terms as lens (a phone’s f/1.8 is about f/8). Needs focus
focus A distance along the view (m), a point, or an object’s name
ev none Exposure, EV at ISO 100: the light is scaled by exposureOf(ev), 1 / (1.2 · 2^ev), before the tone curve, in place of toneMappingExposure

CAMERA_DEFAULTS holds them. The aspect is always the frame’s. fieldOfView, diagonalLens and lensForVertical convert (three.js’s setFocalLength measures on another frame). allyhub scenery solve finds the camera that put points of the scenery at pixels of a picture.

A move (Move) is (t, view?) => CameraState, a function of scenery time alone; view (SceneryView) gives pointOf(name), time(ref), the frame’s aspect and screen(name). Every moment a move takes (MoveTime) is seconds or a word or anchor reference of the part that draws the scenery.

Function What it makes
keys([{ t, position?, target?, yaw?, pitch?, lens?, fov?, roll?, aperture?, focus?, ev?, ease? }], { lensSpace? }) Each key at its time: position along a curve, speed continuous, a repeated position holds; ease runs the stretch into its key; lens in log focal length (or lensSpace: "fov"); aperture and ev in every key or none
path(points, { dur | speed, start?, ease?, target?, ahead?, frame?, alpha? }) Through the points at constant speed; target a point, a name or "ahead" (default); frame: "rmf" carries up untwisted (PATH_DEFAULTS)
orbit({ around, radius, height?, from, to, dur, start?, ease?, target? }) Round a point or a name; 0° on its +Z side, 90° on +X, not wrapped
shots([{ from, camera }, …]) Shots joined by cuts, each a camera or a move; measurements of motion stop at its cuts (cutsOf)
blend(a, b, { from, to, ease? }) a before from, b after to, mixed between (lens at a steady zoom ratio, ev in stops); ease default "smooth"
focusScreen(name, [x, y, w, h], { fill?, lens?, fov?, yaw?, pitch?, ev?, lensFrame? }) A move framing a rectangle of a screen’s content (shares from its top left), square on or yaw/pitch off it
fitCamera({ points | boxes | sphere, aspect, direction | yaw/pitch, lens | fov, fill?, margins?, adjust?, … }) A CameraState framing them exactly; adjust: "lens" keeps position and solves the lens (FIT_DEFAULTS)
mixCameras(a, b, x, view, what) The camera x of the way from a to b, as blend mixes them
withCuts(camera, move) move with camera’s cuts, so a move of the agent’s own wrapped round shots() keeps them

Ease is "linear", "in", "out", "in-out", "smooth" or a function on 0–1 (easeValue). aimed({ overshoot?, peak?, primary?, vary?, seed?, index? }), an aimed movement’s progress (past 1 and back), is an ease too; aimAt(shape, x) and rise(s, peak) are its parts (AIM_DEFAULTS).

A shot holds a camera state or a move, not a name: a scenery’s own are room.cameras?.["door"] and room.moves?.["walk"].

camera: shots([
{ from: 0, camera: { position: [0, 1.5, 2.8], target: "sofa", lens: 26 } },
{ from: "@demo", camera: blend(wide, focusScreen("monitor-screen", [0.62, 0.05, 0.1, 0.06]), { from: "@demo", to: "@click" }) },
]),

handheld(camera, opts) is a camera or a move held in a hand: a Move of t alone, the same for a seed, another hand in each shot of shots() (whose cuts it keeps), its lens and focus kept. Amounts are standard deviations. Defaults: HANDHELD_DEFAULTS, STABILIZER_DEFAULTS, AUTOFOCUS_DEFAULTS, AUTO_EXPOSURE_DEFAULTS (defaults.md).

Option What it is
angle, roll, shift, sway The sway: yaw and pitch degrees, roll as a share of them, metres on each axis; its spectrum round sway Hz, falling as 1/f²
tremor { angle, frequency, spread, vary, swell, period }: a band round frequency Hz that swells
drift { angle, shift, period }: the framing’s slow drift
stabilizer { strength, cutoff }: takes strength of the turning out above cutoff Hz. Absent: none
reframes [{ at, to?, turn?, shift?, duration?, overshoot? }]: an aimed move to a camera, or by a Turn [yaw, pitch, roll?] and Shift [right, up, forward?]; reframe sets defaults
jolts [{ at, turn? }]: the hand knocked, ringing out; jolt sets their defaults
autofocus With an aperture and focus: rests miss off and hunts after cuts, reframes and jolts. Absent: none; {} the defaults
exposure With an ev: follows a reframe’s ev late, is off at a cut and settles, breathes. Absent: none
seed Another seed is another hand

PHONE_CAMERA (@allyhub/sdk/scenery) names a phone’s main camera as the options of each mechanism; phoneCamera(over) changes any value and keeps the rest. lens goes into a camera state, sensor into scenery({ sensor }), hand into handheld(), and the marks it leaves on its picture (grain, softness, sharpening) into video()’s capture (video.md). Its motionBlur is null: each sample draws a frame again (8 made a phone-filmed short take three times as long to render), and a hand’s moves barely show it. The sensor’s grade follows what fills the frame, as a phone’s exposure does: on a room or a person its room grade (a gentle contrast, a little more colour); as a screen comes to fill the frame, its screen grade (the phone exposes for the lit screen: its white a neutral light grey, its middle tones and colours deep), eased in between share[0] and share[1] of the frame filled (default 50% and 80%), frame by frame. Each grade is a brightness curve (curve, hues kept), then saturation and hue; it grades the scenery layer alone, not captions laid over it; grade: null for none. The screen’s grade and the capture’s marks are measured against phone photos of a laptop’s screen (fine grain kept in the whites, crisp sharpened text). Each is a starting point, not a look.

import { part, video } from "@allyhub/sdk";
import { phoneCamera } from "@allyhub/sdk/scenery";
import { handheld, scenery, shots } from "@allyhub/scenery-host/layer";
import room from "../room/room.ts";
const phone = phoneCamera({ hand: { tremor: { angle: 0.15 } } });
const camera = handheld(
shots([
{
from: 0,
camera: {
...phone.lens,
position: [0, 1.4, 1.2],
target: "sofa",
focus: "sofa",
ev: 7.4,
},
},
]),
{ ...phone.hand, reframes: [{ at: 2, turn: [-2.5, 1] }] },
);
export default video({
format: "9x16",
capture: phone.capture,
parts: () => [
part({
key: "look",
dur: 5,
layers: [scenery(room, { camera, sensor: phone.sensor })],
}),
],
});

scenery(scenery, opts) is a graphic layer. The scenery is built once when the output loads and posed at each frame’s scenery time.

import { custom, part, video } from "@allyhub/sdk";
import {
projectScenery,
scenery,
screen,
} from "@allyhub/scenery-host/layer";
import room from "../room/room.ts";
import demo from "../demo/demo.ts";
export default video({
format: "16x9",
parts: () => [
part({
key: "walk",
dur: 6,
layers: [
scenery(room, {
camera: "walk",
name: "room",
screens: { "tv-screen": screen(demo, { start: 1 }) },
}),
custom({
render(el, t) {
const sofa = projectScenery("room", "sofa", t, el);
if (sofa)
el.style.transform = `translate(${String(sofa.x)}px, ${String(sofa.y)}px)`;
},
}),
],
}),
],
});
Option Default What it does
camera required A CameraState, the name of one of the scenery’s cameras or moves, or a move
box the frame [x, y, w, h] as fractions of the frame; the camera’s aspect is the box’s
background the scenery’s own A colour, or "transparent"; without it what build set as the scene’s background, else transparent
time { from: 0 } The scenery’s time from the film’s: { from } at the film’s start, or (filmT) => number. It runs on across parts and cuts
quality { scale: 1, focusLevels: 8 } scale: the share of the layer’s size each way it is drawn at (above 1 supersamples; 0.5 for a quicker draft, softer). focusLevels: Gaussian levels a screen’s depth of field is mixed from. SCENERY_LAYER_DEFAULTS
sensor none A phone’s sensor filming it: the moiré its Bayer grid makes of the screens’ pixels. { photosites, lens, moire, chroma, distortion }, SENSOR_DEFAULTS
screens none { <object name>: screen(…) } (below)
props none SceneryProps this layer brings in (below)
name "scenery" What projectScenery and the facts call it
at, until, dur, z As every layer (video.md)

projectScenery(layer, target, t, el), called in a layer’s render with its own element, gives where a point or a named object of the scenery layer called layer in the same part is at part time t: { x, y } in frame px, inView, depth (m); undefined when no such layer shows then.

Props (SceneryProp: { name, build(w), update?(object, t, w) }) are things a layer brings in (a stand-in, a pointing rod): built after the scenery and its bake (never in it), posed from t after the scenery’s update, shown only in the layers that add them. One a hand moves takes its path from the SDK’s handPath, tap or drag (layers.md, “A person’s input”), as the registry’s pointing-prop does.

A scenery inside another video. A video item whose picture is a scenery can be shown in a film: take(scene).range(from, to) plays a stretch of it and footage({ camera }) draws its scenery layers from a camera of the film’s (cameras by layer name for several, quality: { scale } for this film only), on the item’s clock. See footage.md and speech.md.

screen(content, opts) puts content on a named object of the scenery (its first mesh’s +Z face, the content’s top left at the geometry’s −x, +y). The content is a video item of the project (its piece, imported: its parts, timing, anchors and components its own; pictures only, not its sound, and not footage yet) or graphic layers drawn on size px, whose windows and time props are seconds of the screen’s time or references of the part.

Option Default What it does
start 0 When the screen’s time is 0: scenery seconds, or a reference of the part ("@demo")
cue 0 The screen’s own moment that plays at start: seconds, or (a video) its own reference ("@click", a part’s key)
cues Instead: { <its moment>: <film reference or scenery seconds> }; between two cues it plays at the speed that joins them
time the scenery’s Instead: { from } or (t) => number from scenery time
size [1920, 1080] Layers only: the content’s pixels (a video uses its format’s)
dur to the layer’s end Layers only
background "#000000" Behind the content, or "transparent"
pixels { density: 2, rows: 0.15, mesh: 0.09 } Its physical pixels, for a sensor’s moiré, and mesh (0–1, 0 none): the fine grid of faint lines a phone close to a screen shows, every three of its pixels (PANEL_DEFAULTS)
glass none { reflectance?, roughness?, scale? }: what its glass reflects of the scenery over its picture, by Fresnel; {} the defaults (GLASS_DEFAULTS)
inScenery true Drawn in the scenery: { brightness?, light? } (below); false: laid over the scenery
import { custom, part, video } from "@allyhub/sdk";
// Made only to be shown on a screen.
export default video({
format: "16x9",
deliverable: false,
parts: () => [
part({
key: "open",
dur: 4,
layers: [
custom({
render: (el, t) => (el.textContent = `${String(Math.round(t))} new`),
}),
],
}),
],
});

Before start a video holds its first frame, after its end its last. A video that shows itself through any chain is AK-M010; one made only for a screen is video({ deliverable: false }).

In the scenery (where the browser has Chrome’s HTML-in-Canvas, as AllyHub’s has) the content is the mesh’s texture, drawn with the exposure, depth of field, glass and moiré, and it lights the scenery’s standard and physical materials (at most 2 screens). brightness (default 1) is its white, scene-linear: nits in a scenery lit in physical units with an ev. light: false lights nothing; light: { cells, specular, base } sets the cells of its diffuse light ([8, 5]), leaves out its reflection (specular: false), and with base: { brightness?, color? } adds only what differs from steady light the scenery already has from it (SCREEN_LIGHT_DEFAULTS). material.userData.screenLight = false (or { specular: false }) keeps a material out. Without HTML-in-Canvas (or with ALLYHUB_HTML_IN_CANVAS=0) the mesh is a hole with the content laid over it at its own colours: AK-W007, unless asked with inScenery: false. An element it cannot show (in the browser’s top layer, preserve-3d planes it cannot order, over the largest texture) is AK-W008.

Text on a screen is left out of the frame’s text facts; scenery-screen reports its smallest text instead.

allyhub scenery bake <item> bakes the scenery’s light (direct, shadows, bounces, emissive surfaces, the environment) into a lightmap once, kept in .allyhub/cache/bake/ by the scene’s digest and the renderer that made it, and laid on the scenery each time it is built. A bake made on a GPU is laid only by that GPU; one made on the CPU by every renderer. A scene that changed is drawn unbaked until baked again. How it bakes is defineScenery({ bake }) (BakeOptions: density, samples, bounces, weight, skip, ignore, posedAt, probe, …; BAKE_DEFAULTS).

allyhub scenery <item> (objects, sizes, costs, lights, cameras, moves, bake), scenery view, map, see, measure, motion, solve and bake draw or measure the scenery module alone, without its screens (allyhub frames on a video shows them); inspect fidelity compares each screen drawn in the scenery with its own element.

The check reports each scenery layer’s facts, printed by allyhub inspect: scenery-contact, scenery-support, mesh-health, camera-clearance, camera-motion, scenery-screen (each screen’s share, scale, angle, hidden share, smallest text, moiré, cue speeds), scenery-screen-fidelity and scenery-bake. frames cuts sheets a scenery camera’s cuts.

Rules a problem is reported under: AK-W001 (it does not build, or update throws), AK-W002 (WebGL cannot be made, or is not SwiftShader where the CPU was asked for), AK-W003 (a file does not load), AK-W004 (a camera cannot be drawn), AK-W005 (a name names no object, or two), AK-W006 (a shader does not compile, or needs too many texture units), AK-W007 (a screen drawn over the scenery, a warning), AK-W008 (a screen in the scenery cannot show its element).

Frames are drawn on the GPU on a Mac and on the CPU (SwiftShader, the same pixels on every computer) elsewhere; ALLYHUB_GPU=0 or 1 chooses.

Was this page helpful?