# SDK: 3D sceneries

> A scenery is three.js the agent writes, in a module whose default export is defineScenery(…): AllyHub builds it once, poses it from time alone, draws it from cameras and measures it. This file covers that contract, cameras and their moves, and the scenery() layer with its screens. The commands are in allyhub help 3d.

Source: https://allyhub.com/docs/reference/sdk/scenery · AllyHub 0.10.0
Index of all docs: https://allyhub.com/llms.txt

## A scenery is an item

A scenery a person sees is listed in `allyhub.yaml` as `kind: scenery` ([project.md](https://allyhub.com/docs/reference/sdk/project.md#items)). A person looks round it in Studio and `allyhub dev`'s page at scenery time 0, selecting a placed object (the outermost named one) and its named `cameras`, never editing it; a note on it names the object, its size, the point clicked and the view as an `allyhub scenery view` command. A scenery module the list does not name (one a video imports, a setting several share) is code, yours alone; the stage knows a scenery module by what it is (it calls `defineScenery`), not by its name.

- A video imports a scenery item by its path, as any module: `import room from "../room/room.ts"`.
- A path a scenery reads that starts with `./` or `../` (`w.texture("./wood.jpg")`, `w.load("./desk.glb")`, a model's image) is from the scenery's own module; any other is from the project's root. An import (`import desk from "./desk.glb"`) is the same. Kept in its folder and named from there, a scenery works wherever the folder goes.
- `three` (r186, 0.186.1), `three/addons/…`, `three/webgpu` and `three/tsl` are imported by name with no `package.json`: the CLI lends them, unless the project declares its own `three`. So are `@allyhub/scenery-host/layer` (everything in this file) and `@allyhub/model` ([modeling.md](https://allyhub.com/docs/reference/sdk/modeling.md)). Their types come with the project's `allyhub` once `allyhub/client` is in tsconfig.json's `types`, beside `@allyhub/sdk/client`; three.js's are `@types/three`, which the project installs at the version lent (`npm install -D @types/three@0.186.0`).
- glTF and GLB load (`.glb`, `.gltf`, `.bin`, `.ktx2`), environments are `.hdr` or `.exr`; all are media imports. CC0 models, HDRIs and textures: `allyhub add search --type @polyhaven`, `allyhub add @polyhaven/<id>`. AllyHub's own 3D models and sceneries: `allyhub add search --type 3d-model` (or `scenery`), `allyhub add 3d-model/<name>`.

**Units and axes.** Right-handed, **Y up**, **metres**, angles in **degrees** everywhere in AllyHub's API (three.js itself takes radians). A camera looks down its own −Z. A glTF model's front is its +Z. Positions and sizes AllyHub prints are world space. three.js's lights are physical: point and spot lights in candela, directional lights in lux, an emissive surface in nits.

## `defineScenery`

```ts file=room/room.ts

  async build(w) {
    w.renderer.toneMapping = THREE.ACESFilmicToneMapping;
    const sofa = await w.load(sofaModel);
    sofa.name = "sofa";
    w.scene.add(sofa);
    const mixer = new THREE.AnimationMixer(sofa);
    const clips = sofa.userData["animations"] as THREE.AnimationClip[];
    for (const clip of clips) mixer.clipAction(clip).play();
    return { mixer };
  },
  update(made, t, w) {
    w.seek(made.mixer, t);
  },
  cameras: { door: { position: [0, 1.5, 2.8], target: "sofa", lens: 26 } },
  moves: {
    walk: path(
      [
        [0, 1.5, 3],
        [0.4, 1.2, -0.2],
      ],
      { dur: 6, target: "sofa" },
    ),
  },
});
```

`build(w)` runs once, awaited with every load it starts (60 s at most; in a video it counts in the piece's `loadLimit`, AK-P002); what it returns is `made`. `update(made, t, w)` poses it at `t`, seconds of scenery time. `cameras` (checked when defined, AK-W004) and `moves` are by name, for the commands and for layers. `bake` is in [Baked light](#baked-light). `title` is the item's name where `allyhub.yaml` gives it none, read from the module as written (a string, not computed); without either, its file's name.

**From `t` alone.** The same moment must draw the same pixels in any order (AK-D002). `Date`, `performance.now`, `Math.random`, `crypto.getRandomValues` and `requestAnimationFrame` follow the moment drawn; timers run only while it builds. Pose an `AnimationMixer` with `w.seek(mixer, t)` (every action reset, then `setTime(t)`), not a bare `setTime`. Name what is referred to (`Object3D.name`): cameras, moves, measurements and `projectScenery` take names, and a name no object has, or two have, is AK-W005.

`renderer` (all optional; `SCENERY_RENDERER_DEFAULTS`, the fastest drawing):

| Option | Default | What it does |
| --- | --- | --- |
| `kind` | `"webgl"` | `"webgpu"`: `WebGPURenderer` (node materials, TSL) through its WebGL 2 back end; not baked yet |
| `antialias` | `false` | Multisampled edges |
| `transmissionScale` | 0.25 | The extra pass transmissive materials (glass) need, a share of the frame's size, (0, 2] |
| `outputBufferType` | three's | The buffer effects draw into (`HalfFloatType` for HDR effects) |
| `logarithmicDepthBuffer`, `reversedDepthBuffer` | `false` | |
| `focus` | `{ samples: 64, maxRadius: 64 }` | A camera's depth of field: samples per pixel, the widest blur's radius in px as drawn, and `supersample` (1–4; default 2 with `antialias`, else 1) |

Alpha, size and pixel ratio are AllyHub's; set the rest on `w.renderer` in `build` (tone mapping, shadows, `setEffects`). A material whose shader does not compile, or needs more texture units than the renderer has (maps, environment, shadow-casting lights, screens lighting it), draws nothing: AK-W006, a warning over 16 units where the renderer has more.

`w`, the scenery's host (`SceneryHost`):

| Field | What it is |
| --- | --- |
| `scene`, `renderer` | The `THREE.Scene`, and the renderer AllyHub made |
| `load(asset)` | A glTF or GLB as a `Group`, with the lent decoders (Draco, meshopt, KTX2); its animations on `userData.animations`. One that does not load is AK-W003 |
| `texture(asset, { colorSpace? })` | A picture as a texture: sRGB, or `"linear"` for a data map (normal, roughness, AO, displacement) |
| `environment(asset, { background? })` | An `.hdr` or `.exr` as the scene's environment, and its background when asked |
| `seek(mixer, t)` | Poses an animation mixer at `t` |
| `object(name)` | The object of that name, undefined for none |
| `positionOf(name)`, `boundsOf(name)` | Its box's centre now; its box, from its vertices |
| `asset(path)` | A project file's URL |
| `time(ref)` | In `update`: the scenery time of a word or anchor reference (`"@tap"`, `"@tap+0.2"`) of the part that draws it. Throws in `build` |
| `screen(name)` | A screen's face now (its first mesh's +Z side): `point(x, y)` from shares of its content, `center`, `normal`, `up`, `width`, `height` |
| `screenPicture(name)` | In `update`: a screen's picture as a mipmapped texture, for a material of the scenery's own; undefined for a screen drawn over the scenery |

An asset is an import or a path. Where there is no part to time a reference by (the `allyhub scenery` commands), its moments come from the video that draws the scenery (the one video of the project that does, or `--piece`'s), its scenery layers finding them as they do when drawing; `--times @tap=2.4` gives them by hand, over the video's.

## Cameras

A `CameraState` points one of three ways, or looks down −Z, level:

```ts fragment
{ position: [0, 1.5, 2.8], target: "sofa", lens: 26 }        // at an object's box centre, followed as it moves
{ position: [0, 1.5, 2.8], target: [0, 0.6, -1.4], fov: 40 } // at a point; a vertical fov in degrees
{ position: [0, 1.5, 2.8], yaw: 30, pitch: -10 }             // 30° left of −Z, 10° down
{ position: [0, 1.5, 2.8], direction: [0, -0.2, -1], roll: 5 }
```

| Field | Default | What it is |
| --- | --- | --- |
| `position` | required | Metres |
| `target`, `direction`, or `yaw`/`pitch` | down −Z | `yaw` about +Y (90 looks down −X); `pitch` above the horizon |
| `up` | `[0, 1, 0]` | A view straight along `up` cannot be drawn (AK-W004) |
| `roll` | 0 | Degrees about the view, counterclockwise |
| `lens` | 26 | 35 mm-equivalent focal length, mm |
| `lensFrame` | `"diagonal"` | What the lens is measured on: the frame's diagonal (CIPA), `"long"`, `"short"`, `"phone"` (`{ aspect: 4/3, keep: "long" }`, a phone's "26 mm"), or `{ aspect, keep }` |
| `fov` | | Vertical degrees, instead of a lens |
| `near`, `far` | 0.05, 500 | Metres |
| `aperture` | none (a pinhole) | An f-number in the same 35 mm-equivalent terms as `lens` (a phone's f/1.8 is about f/8). Needs `focus` |
| `focus` | | A distance along the view (m), a point, or an object's name |
| `ev` | none | Exposure, EV at ISO 100: the light is scaled by `exposureOf(ev)`, 1 / (1.2 · 2^ev), before the tone curve, in place of `toneMappingExposure` |

`CAMERA_DEFAULTS` holds them. The aspect is always the frame's. `fieldOfView`, `diagonalLens` and `lensForVertical` convert (three.js's `setFocalLength` measures on another frame). `allyhub scenery solve` finds the camera that put points of the scenery at pixels of a picture.

## Moves

A move (`Move`) is `(t, view?) => CameraState`, a function of scenery time alone; `view` (`SceneryView`) gives `pointOf(name)`, `time(ref)`, the frame's `aspect` and `screen(name)`. Every moment a move takes (`MoveTime`) is seconds or a word or anchor reference of the part that draws the scenery.

| Function | What it makes |
| --- | --- |
| `keys([{ t, position?, target?, yaw?, pitch?, lens?, fov?, roll?, aperture?, focus?, ev?, ease? }], { lensSpace? })` | Each key at its time: position along a curve, speed continuous, a repeated position holds; `ease` runs the stretch into its key; lens in log focal length (or `lensSpace: "fov"`); `aperture` and `ev` in every key or none |
| `path(points, { dur \| speed, start?, ease?, target?, ahead?, frame?, alpha? })` | Through the points at constant speed; `target` a point, a name or `"ahead"` (default); `frame: "rmf"` carries `up` untwisted (`PATH_DEFAULTS`) |
| `orbit({ around, radius, height?, from, to, dur, start?, ease?, target? })` | Round a point or a name; 0° on its +Z side, 90° on +X, not wrapped |
| `shots([{ from, camera }, …])` | Shots joined by cuts, each a camera or a move; measurements of motion stop at its cuts (`cutsOf`) |
| `blend(a, b, { from, to, ease? })` | `a` before `from`, `b` after `to`, mixed between (lens at a steady zoom ratio, `ev` in stops); `ease` default `"smooth"` |
| `focusScreen(name, [x, y, w, h], { fill?, lens?, fov?, yaw?, pitch?, ev?, lensFrame? })` | A move framing a rectangle of a screen's content (shares from its top left), square on or `yaw`/`pitch` off it |
| `fitCamera({ points \| boxes \| sphere, aspect, direction \| yaw/pitch, lens \| fov, fill?, margins?, adjust?, … })` | A `CameraState` framing them exactly; `adjust: "lens"` keeps `position` and solves the lens (`FIT_DEFAULTS`) |
| `mixCameras(a, b, x, view, what)` | The camera `x` of the way from `a` to `b`, as `blend` mixes them |
| `withCuts(camera, move)` | `move` with `camera`'s cuts, so a move of the agent's own wrapped round `shots()` keeps them |

`Ease` is `"linear"`, `"in"`, `"out"`, `"in-out"`, `"smooth"` or a function on 0–1 (`easeValue`). `aimed({ overshoot?, peak?, primary?, vary?, seed?, index? })`, an aimed movement's progress (past 1 and back), is an `ease` too; `aimAt(shape, x)` and `rise(s, peak)` are its parts (`AIM_DEFAULTS`).

A shot holds a camera state or a move, not a name: a scenery's own are `room.cameras?.["door"]` and `room.moves?.["walk"]`.

```ts fragment
camera: shots([
  { from: 0, camera: { position: [0, 1.5, 2.8], target: "sofa", lens: 26 } },
  { from: "@demo", camera: blend(wide, focusScreen("monitor-screen", [0.62, 0.05, 0.1, 0.06]), { from: "@demo", to: "@click" }) },
]),
```

### `handheld`

`handheld(camera, opts)` is a camera or a move held in a hand: a `Move` of t alone, the same for a `seed`, another hand in each shot of `shots()` (whose cuts it keeps), its lens and focus kept. Amounts are standard deviations. Defaults: `HANDHELD_DEFAULTS`, `STABILIZER_DEFAULTS`, `AUTOFOCUS_DEFAULTS`, `AUTO_EXPOSURE_DEFAULTS` ([defaults.md](https://allyhub.com/docs/reference/sdk/defaults.md)).

| Option | What it is |
| --- | --- |
| `angle`, `roll`, `shift`, `sway` | The sway: yaw and pitch degrees, roll as a share of them, metres on each axis; its spectrum round `sway` Hz, falling as 1/f² |
| `tremor` | `{ angle, frequency, spread, vary, swell, period }`: a band round `frequency` Hz that swells |
| `drift` | `{ angle, shift, period }`: the framing's slow drift |
| `stabilizer` | `{ strength, cutoff }`: takes `strength` of the turning out above `cutoff` Hz. Absent: none |
| `reframes` | `[{ at, to?, turn?, shift?, duration?, overshoot? }]`: an aimed move to a camera, or by a `Turn` `[yaw, pitch, roll?]` and `Shift` `[right, up, forward?]`; `reframe` sets defaults |
| `jolts` | `[{ at, turn? }]`: the hand knocked, ringing out; `jolt` sets their defaults |
| `autofocus` | With an `aperture` and `focus`: rests `miss` off and hunts after cuts, reframes and jolts. Absent: none; `{}` the defaults |
| `exposure` | With an `ev`: follows a reframe's `ev` late, is off at a cut and settles, breathes. Absent: none |
| `seed` | Another seed is another hand |

### A phone

`PHONE_CAMERA` (`@allyhub/sdk/scenery`) names a phone's main camera as the options of each mechanism; `phoneCamera(over)` changes any value and keeps the rest. `lens` goes into a camera state, `sensor` into `scenery({ sensor })`, `hand` into `handheld()`, and the marks it leaves on its picture (grain, softness, sharpening) into `video()`'s `capture` ([video.md](https://allyhub.com/docs/reference/sdk/video.md)). Its `motionBlur` is null: each sample draws a frame again (8 made a phone-filmed short take three times as long to render), and a hand's moves barely show it. The sensor's `grade` follows what fills the frame, as a phone's exposure does: on a room or a person its `room` grade (a gentle contrast, a little more colour); as a screen comes to fill the frame, its `screen` grade (the phone exposes for the lit screen: its white a neutral light grey, its middle tones and colours deep), eased in between `share[0]` and `share[1]` of the frame filled (default 50% and 80%), frame by frame. Each grade is a brightness curve (`curve`, hues kept), then `saturation` and `hue`; it grades the scenery layer alone, not captions laid over it; `grade: null` for none. The screen's grade and the capture's marks are measured against phone photos of a laptop's screen (fine grain kept in the whites, crisp sharpened text). Each is a starting point, not a look.

```ts file=phone/phone.ts

const phone = phoneCamera({ hand: { tremor: { angle: 0.15 } } });
const camera = handheld(
  shots([
    {
      from: 0,
      camera: {
        ...phone.lens,
        position: [0, 1.4, 1.2],
        target: "sofa",
        focus: "sofa",
        ev: 7.4,
      },
    },
  ]),
  { ...phone.hand, reframes: [{ at: 2, turn: [-2.5, 1] }] },
);

  format: "9x16",
  capture: phone.capture,
  parts: () => [
    part({
      key: "look",
      dur: 5,
      layers: [scenery(room, { camera, sensor: phone.sensor })],
    }),
  ],
});
```

## The `scenery()` layer

`scenery(scenery, opts)` is a graphic layer. The scenery is built once when the output loads and posed at each frame's scenery time.

```ts file=tour/tour.ts
  projectScenery,
  scenery,
  screen,
} from "@allyhub/scenery-host/layer";

  format: "16x9",
  parts: () => [
    part({
      key: "walk",
      dur: 6,
      layers: [
        scenery(room, {
          camera: "walk",
          name: "room",
          screens: { "tv-screen": screen(demo, { start: 1 }) },
        }),
        custom({
          render(el, t) {
            const sofa = projectScenery("room", "sofa", t, el);
            if (sofa)
              el.style.transform = `translate(${String(sofa.x)}px, ${String(sofa.y)}px)`;
          },
        }),
      ],
    }),
  ],
});
```

| Option | Default | What it does |
| --- | --- | --- |
| `camera` | required | A `CameraState`, the name of one of the scenery's cameras or moves, or a move |
| `box` | the frame | `[x, y, w, h]` as fractions of the frame; the camera's aspect is the box's |
| `background` | the scenery's own | A colour, or `"transparent"`; without it what `build` set as the scene's background, else transparent |
| `time` | `{ from: 0 }` | The scenery's time from the film's: `{ from }` at the film's start, or `(filmT) => number`. It runs on across parts and cuts |
| `quality` | `{ scale: 1, focusLevels: 8 }` | `scale`: the share of the layer's size each way it is drawn at (above 1 supersamples; 0.5 for a quicker draft, softer). `focusLevels`: Gaussian levels a screen's depth of field is mixed from. `SCENERY_LAYER_DEFAULTS` |
| `sensor` | none | A phone's sensor filming it: the moiré its Bayer grid makes of the screens' pixels. `{ photosites, lens, moire, chroma, distortion }`, `SENSOR_DEFAULTS` |
| `screens` | none | `{ <object name>: screen(…) }` (below) |
| `props` | none | `SceneryProp`s this layer brings in (below) |
| `name` | `"scenery"` | What `projectScenery` and the facts call it |
| `at`, `until`, `dur`, `z` | | As every layer ([video.md](https://allyhub.com/docs/reference/sdk/video.md)) |

`projectScenery(layer, target, t, el)`, called in a layer's `render` with its own element, gives where a point or a named object of the scenery layer called `layer` in the same part is at part time `t`: `{ x, y }` in frame px, `inView`, `depth` (m); undefined when no such layer shows then.

**Props** (`SceneryProp`: `{ name, build(w), update?(object, t, w) }`) are things a layer brings in (a stand-in, a pointing rod): built after the scenery and its bake (never in it), posed from t after the scenery's update, shown only in the layers that add them. One a hand moves takes its path from the SDK's `handPath`, `tap` or `drag` ([layers.md](https://allyhub.com/docs/reference/sdk/layers.md), "A person's input"), as the registry's `pointing-prop` does.

**A scenery inside another video.** A video item whose picture is a scenery can be shown in a film: `take(scene).range(from, to)` plays a stretch of it and `footage({ camera })` draws its scenery layers from a camera of the film's (`cameras` by layer name for several, `quality: { scale }` for this film only), on the item's clock. See [footage.md](https://allyhub.com/docs/reference/sdk/footage.md) and [speech.md](https://allyhub.com/docs/reference/sdk/speech.md).

## Screens

`screen(content, opts)` puts content on a named object of the scenery (its first mesh's +Z face, the content's top left at the geometry's −x, +y). The content is a video item of the project (its piece, imported: its parts, timing, anchors and components its own; pictures only, not its sound, and not footage yet) or graphic layers drawn on `size` px, whose windows and time props are seconds of the screen's time or references of the part.

| Option | Default | What it does |
| --- | --- | --- |
| `start` | 0 | When the screen's time is 0: scenery seconds, or a reference of the part (`"@demo"`) |
| `cue` | 0 | The screen's own moment that plays at `start`: seconds, or (a video) its own reference (`"@click"`, a part's key) |
| `cues` | | Instead: `{ <its moment>: <film reference or scenery seconds> }`; between two cues it plays at the speed that joins them |
| `time` | the scenery's | Instead: `{ from }` or `(t) => number` from scenery time |
| `size` | `[1920, 1080]` | Layers only: the content's pixels (a video uses its format's) |
| `dur` | to the layer's end | Layers only |
| `background` | `"#000000"` | Behind the content, or `"transparent"` |
| `pixels` | `{ density: 2, rows: 0.15, mesh: 0.09 }` | Its physical pixels, for a sensor's moiré, and `mesh` (0–1, 0 none): the fine grid of faint lines a phone close to a screen shows, every three of its pixels (`PANEL_DEFAULTS`) |
| `glass` | none | `{ reflectance?, roughness?, scale? }`: what its glass reflects of the scenery over its picture, by Fresnel; `{}` the defaults (`GLASS_DEFAULTS`) |
| `inScenery` | `true` | Drawn in the scenery: `{ brightness?, light? }` (below); `false`: laid over the scenery |

```ts file=demo/demo.ts

// Made only to be shown on a screen.
  format: "16x9",
  deliverable: false,
  parts: () => [
    part({
      key: "open",
      dur: 4,
      layers: [
        custom({
          render: (el, t) => (el.textContent = `${String(Math.round(t))} new`),
        }),
      ],
    }),
  ],
});
```

Before `start` a video holds its first frame, after its end its last. A video that shows itself through any chain is AK-M010; one made only for a screen is `video({ deliverable: false })`.

**In the scenery** (where the browser has Chrome's HTML-in-Canvas, as AllyHub's has) the content is the mesh's texture, drawn with the exposure, depth of field, glass and moiré, and it lights the scenery's standard and physical materials (at most 2 screens). `brightness` (default 1) is its white, scene-linear: nits in a scenery lit in physical units with an `ev`. `light: false` lights nothing; `light: { cells, specular, base }` sets the cells of its diffuse light (`[8, 5]`), leaves out its reflection (`specular: false`), and with `base: { brightness?, color? }` adds only what differs from steady light the scenery already has from it (`SCREEN_LIGHT_DEFAULTS`). `material.userData.screenLight = false` (or `{ specular: false }`) keeps a material out. Without HTML-in-Canvas (or with `ALLYHUB_HTML_IN_CANVAS=0`) the mesh is a hole with the content laid over it at its own colours: AK-W007, unless asked with `inScenery: false`. An element it cannot show (in the browser's top layer, `preserve-3d` planes it cannot order, over the largest texture) is AK-W008.

Text on a screen is left out of the frame's text facts; `scenery-screen` reports its smallest text instead.

## Baked light

`allyhub scenery bake <item>` bakes the scenery's light (direct, shadows, bounces, emissive surfaces, the environment) into a lightmap once, kept in `.allyhub/cache/bake/` by the scene's digest and the renderer that made it, and laid on the scenery each time it is built. A bake made on a GPU is laid only by that GPU; one made on the CPU by every renderer. A scene that changed is drawn unbaked until baked again. How it bakes is `defineScenery({ bake })` (`BakeOptions`: `density`, `samples`, `bounces`, `weight`, `skip`, `ignore`, `posedAt`, `probe`, …; `BAKE_DEFAULTS`).

## Measuring

`allyhub scenery <item>` (objects, sizes, costs, lights, cameras, moves, bake), `scenery view`, `map`, `see`, `measure`, `motion`, `solve` and `bake` draw or measure the scenery module alone, without its screens (`allyhub frames` on a video shows them); `inspect fidelity` compares each screen drawn in the scenery with its own element.

The check reports each scenery layer's facts, printed by `allyhub inspect`: `scenery-contact`, `scenery-support`, `mesh-health`, `camera-clearance`, `camera-motion`, `scenery-screen` (each screen's share, scale, angle, hidden share, smallest text, moiré, cue speeds), `scenery-screen-fidelity` and `scenery-bake`. `frames cuts` sheets a scenery camera's cuts.

Rules a problem is reported under: AK-W001 (it does not build, or `update` throws), AK-W002 (WebGL cannot be made, or is not SwiftShader where the CPU was asked for), AK-W003 (a file does not load), AK-W004 (a camera cannot be drawn), AK-W005 (a name names no object, or two), AK-W006 (a shader does not compile, or needs too many texture units), AK-W007 (a screen drawn over the scenery, a warning), AK-W008 (a screen in the scenery cannot show its element).

Frames are drawn on the GPU on a Mac and on the CPU (SwiftShader, the same pixels on every computer) elsewhere; `ALLYHUB_GPU=0` or `1` chooses.
