Skip to content

CLAUDE.md

Latest commit

 

History

History
50 lines (32 loc) · 24.9 KB

CLAUDE.md

File metadata and controls

50 lines (32 loc) · 24.9 KB
Raw

Animation Factory — Project Standard

A code-driven, Kurzgesagt-style 2.5D animation pipeline: a story script in → a deterministic video out. No manual animation tool; everything is code + offline-generated assets.

Authoritative design: docs/superpowers/specs/2026-06-22-animation-factory-design.md. Read it before non-trivial work. This file is the short, durable standard loaded every session; detail lives in the spec and in .claude/skills/.

Golden rules (non-negotiable)

  1. Determinism. Same IR ⇒ byte-identical MP4. Seeded RNG only — never Date.now(), Math.random(), or wall-clock. Everything is a function of frame (+ seed, params). Re-rendering twice must match. Recipe: render with no gl backend (CPU raster) — deterministic + disk-safe; gl:'angle' (the Iris Xe iGPU) is non-deterministic for blur-heavy SVG and software GL balloons the disk. Verify fast on stills (render <proj> --frames auto, ~14s) before the slow full video; for AUDIO, determinism is the decoded-stream md5 of a short full render (ffmpeg -map 0:v -f md5 AND -map 0:a -f md5, both byte-identical cross-process). Offline-generated artifacts are the deterministic record, not the generator: anything stochastic (a TTS engine, an AI asset step) runs ONCE OFFLINE at build into a content-addressed cache (hash of inputs, skip-if-exists); the render replays the FIXED cached file, so the output is byte-deterministic even though the generator is not. See verify-render.
  2. Separation of identity vs motion. Art is fixed assets; code only changes transforms and part-swaps. AI never touches frames or runtime — only the offline asset library and (later) the script→IR front-end.
  3. Reuse over invent. Before writing anything non-trivial, check for an existing library/standard. Adopted standards — do not re-implement: Remotion (host/render/audio/mux/batch), DragonBones (skeletal rigs + IK + mesh deform, via Pixi canvas), Lottie superset (Scene IR property/keyframe model), Zod (single source → types + validation + JSON-Schema), OTIO-aligned (sequencing concepts), plus d3-shape/d3-scale (the chart generator), simplex-noise, @remotion/shapes+@remotion/paths+flubber (the shape layer: primitives + path morph; flubber also bridges match-cut transitions), @remotion/transitions (scene-boundary presentations), @remotion/lottie+<OffthreadVideo> (the footage layer), culori (OKLab color-script interpolation), @remotion/motion-blur, object-hash. Runtime vs compiler boundary: Remotion is the RUNTIME (frame model + interpolate/spring/Easing/Sequence/Series/TransitionSeries + render backend); our Scene IR is the compiler TARGET, passed verbatim as inputProps. The render layer is a thin IR→Remotion adapter — never reimplement a Remotion primitive (interpolation, easing, springs, sequencing, transitions); borrow Remotion + its ecosystem. The wrapper exists ONLY for what Remotion has no native answer to: a serializable/validatable IR, the script→IR→frames compiler, the content library, and the AI-emits-data boundary.
  4. Compiler architecture. script → Story IR → Scene IR → frames → MP4. Each stage is a pure IR_n → IR_{n+1} pass; validate (Zod) at every boundary; content-hash cache; golden-diff. Cache-invalidation rule: the pipeline cache key = the script contents + each pass's PASS_VERSION (+ cacheKeyExtra), NOT the pass source. So changing a pass's CODE (e.g. lower.ts) without bumping its PASS_VERSION serves a STALE Scene IR — after editing a pass, bump its PASS_VERSION or rm -rf .cache before rendering/verifying (this bit the footage muted fix). Likewise the key omits library state: after editing library/*, clear .cache.
  5. Scope discipline. Build by milestone, one at a time; verify + commit each before the next. The full roadmap (M1→M9 + the "cover all of it" tail, docs/factory/REMAINING-ROADMAP.md) is DONE as of 2026-06-23 — so this rule now governs NEW work: don't build ahead of an agreed milestone, and reserve IR fields for future stages rather than implementing them early. When a capability is genuinely complete, say so plainly (don't manufacture scope).
  6. Reuse has TWO levels — never conflate them. (a) Project-internal reuse is the DEFAULT (the After-Effects precomp/asset model): a unit — a rig/object, an animated shape, an imported image — is defined ONCE inside a project and reused across its scenes; it stays PART OF the project and never leaves it. This is the common case and does NOT touch library/. (b) The shared library/ is OPT-IN PUBLIC SHARING ONLY — reach for it only when you deliberately publish a unit so OTHER projects/users can import it (the npm-package layer: library : project :: npm package : app). A published unit is content-addressed (name@version→hash), cataloged in library/index.json, and pinned via animation.lock. Do NOT dump every reusable thing into library/ — that's like publishing every local component to npm. The new-library-entry skill is for the publish/share path; project-local reuse lives in the project. (NOTE: the project-local reusable-definition authoring surface is the in-progress architecture redesign — see the project-local-first / four-layer spec; until it lands, demos used library entries as a stopgap, which is exactly the conflation being corrected.)
  7. Style is selectable DATA; the quality floor is the DEFAULT stylekit, not a mandate (ADR-008). A "house style" is a library stylekit entry (library/stylekits/*.json); core owns only the StyleKitSchema + easing helpers + a STYLE-CLEAN NEUTRAL_STYLEKIT fallback (NO Kurzgesagt hex / liveness magic numbers in src/ as source). Kurzgesagt is the DEFAULT preset (story style defaults to "kurzgesagt"): no motion ever linear, motion blur on fast moves, liveness (idle + breathing + blink + spring follow-through), parallax depth, limited palette. The kit's floor{liveness,parallax,shading,nonLinearMotion} toggles are opt-out — style: plain turns them off for a flat/technical look. An OPTIONAL stylekit paint sub-table (ADR-009; absent → flat, so the default look is unchanged) carries the PAINTED look: a generic mechanism (src/render/paint.ts, gated by floor.shading) turns every shape's solid fill into an AUTO culori shade-ramp form-shading gradient (+ a lit-side rim), reuses core-effects grain/glow, and paints scene atmosphere (dark backdrop + focal pool + vignette + per-layer depth grade) — so the §11.1 shading now does real form-shading; the painted VALUES live in DATA (library/stylekits/kurzgesagt-nature.json), src/ keeps only the schema. Paint follow-ons (M8b, DONE — all DATA/mechanism): a softer 4-stop ramp (highlight→base→mid→shadow) + per-silhouette ramp orientation (round/blobby shapes bias to a radial volume, elongated shapes keep a linear ramp along their major axis), and AERIAL PERSPECTIVE — far (low-parallax) layers haze their WHOLE shade-ramp fill TOWARD the atmosphere colour (mixToward/atmosphereTintColor, silhouette-perfect since it tints the fill itself, not a rectangular overlay), so the outer depth grade only adds a gentle desaturate. The resolved kit travels in Scene IR defs.stylekit; render reads style from the IR, never a core constant. Camera moves are DATA (library/camera/presets.json) + arbitrary {a,k} keyframes the story may pass through. A non-default look = changing DATA, never a core edit.

Layer taxonomy (one graph for characters and objects)

asset/text (fixed art, animate transforms) · rig (provider-rendered generic layer — provider id + opaque spec; e.g. blob-creature/object/dragonbones) · shape (morph/path) · generator (procedural: water, fire, particles, crowd, chart) · footage (DONE: external time-based media — a VIDEO via <OffthreadVideo> or a LOTTIE via @remotion/lottie, frame-seeked → deterministic) · clip (reusable nested pre-composition — DONE: shared defs.clips def + clip-layer refs + Essential-Graphics params/args + Remotion <Sequence>; per-instance id-namespacing + seed derivation keeps two instances distinct yet byte-identical; recurses to any depth) · scene-template/environment (DONE: a library clip used as a full-scene backdrop — a lowering convention reusing the clip machinery, no new nesting). Compose intra-rig (DONE parts/skins — opaque per-rig variant selection the provider interprets) and inter-rig (DONE scene-graph attach — a child layer parented onto a parent rig MOUNT via the rig-manifest mounts DATA; pure per-frame world-anchor in src/render/attach.ts, inherit selects which parent channels propagate). Per-layer compositing (DONE): a blend mode (CSS mix-blend-mode) + a track matte/mask (SVG <mask> / CSS mask-image), applied generically in the layer wrapper on top of shading + effects[] + parallax. Reuse compounds: parts → rigs → presets → clips → scenes → videos.

Audio (M3, DONE) + captions + interop: the full audio stack runs OFFLINE at BUILD (golden rule 2: voice/sfx/music never touch frames/runtime — like factory:gen/the font), caches content-addressed wavs, and Remotion <Audio> muxes them (h264+aac) — never reimplement mixing/looping/ducking. Narration: a beat's say → TTS → cached wav → a kind:"narration" audio[] cue → <Audio>; engine swappable (src/cli/narrate.ts): espeak-ng default (deterministic), Coqui XTTS v2 via an isolated venv (.venv-tts/) with espeak-ng fallback. Captions (A1 + M4a word-sync, DONE): narration-synced on-screen subtitles DERIVED from the same say text + cue window → a captions[] cue → src/render/CaptionTrack.tsx (bottom pill; line/words cadence); default-on (--no-captions/--caption-mode). In words mode, faster-whisper (.venv-whisper + scripts/tts/align_whisper.py) force-aligns the cached wav to its transcript OFFLINE → REAL per-word timings (alignNarration/timedWordsFromAlignment → CaptionCue wordsTimed[]), CACHED content-addressed (assets/audio/align/<hash>.json, skip-if-exists → byte-deterministic replay); missing whisper/alignment-error → the deterministic even-split fallback (never fails; --no-word-align skips the slow first-run step). SFX (A2): show[].sfx/beat.sfx[] → kind:"sfx" cues at the event frame; palette synthesized by ffmpeg lavfi into library/sfx/ (src/cli/sfx.ts). Music + ducking (A3): a music bed (library/music/ ffmpeg beds or asset ref) → one kind:"music" cue, <Loop>-tiled, ducked per-frame from gain→duck while VO speaks (pure duckingAt in Composition.tsx; the reserved <Audio volume> hook is now live). Layered audio tracks (DONE): a story-level audio[] — each {src, at, duration, volume, speed, loop, fade_in, fade_out, trim{before,after}} — lowers (src/cli/audio-pass.ts) to generic audio[] cues that ONE GenericAudioTrack renders: narration/sfx/ambience/foley/extra beds are all independent <Audio> muxed by Remotion, freely overlapping, with per-cue controls mapping 1:1 to NATIVE <Audio> props — volume(fade envelope)/playbackRate(speed)/trimBefore+trimAfter(crop)/loop (golden rule 3, never reimplemented). AudioCue gained volume/playback_rate/trim_before/trim_after/fade_in/fade_out (all optional → omitted = byte-identical to before). Built-in bed/sfx NAMES synth+cache; asset:///path refs pass through. Pure → deterministic (cross-process byte-identical), audible-gated. Track-matte clipPath (A4): a SHAPE-source layer matte builds a real geometry stencil (shapeClipGeometry) composited as a STATIC mask-image data-URI (NOT live clip-path — it drifted across cold renderMedia processes; see DECISIONS 2026-06-23). Generator presets (B, ADR-004): a generator-preset library entry (named gen + locked params, library/generators/*.preset.json) the resolver expands when a show[].generator names it (layer args win); factory:gen-preset registers one. Interop: factory:bundle (self-contained shareable bundle), factory:list (library + projects), export:otio (OTIO-aligned timeline JSON). render runs narration+sfx+music on the compile path (--no-audio master switch). VERIFY GATE: audio must be AUDIBLE — a build with audio is checked for cross-process byte-identity (decoded video md5 AND decoded audio md5) AND for loudness (ffmpeg volumedetect mean well above −60 dB; a silent track is a regression). Post grade (M8a, DONE): an optional story-level post[] (carried verbatim to Scene IR post[]) is a FILM-LEVEL effect stack (color_grade/vignette/grain/…) wrapping the WHOLE composited frame in Composition.tsx, REUSING the same core-effects ops the per-layer effects[] use (resolveEffects/applyEffects) — absent/empty → strict no-op (byte-identical to before), dropped on --alpha, pure → deterministic. Lip-sync visemes (M4b, DONE): the narrate pass derives a per-frame MOUTH-OPENNESS track OFFLINE from each cached narration wav (ffmpeg → mono PCM → short-time RMS energy envelope → 0..1, attack/decay smoothed; mouthTrackForNarration/opennessEnvelope in narrate.ts), CACHED content-addressed under assets/audio/mouth/ (skip-if-exists → deterministic replay; ffmpeg missing → no track, never fails), and attaches it to the SPEAKER rig layer (first on-screen actor) as a GENERIC, core-OPAQUE mouth channel on the Scene-IR rig layer ({fps, open[], viseme?} — core never reads a sample's meaning). The blob-creature provider samples it by scene-local frame to open/close its mouth (mouthMarkup); openness 0 (no track / beat w/o narration / --no-lip-sync) draws the original resting smile (byte-identical to pre-M4b). Optional coarse viseme[] labels (word/gap classes) come FOR FREE when a whisper caption alignment already ran (no extra whisper run) for part-swap providers. Pure (track, frame)→openness. Director (M5, DONE): src/pipeline/director.ts — a Director interface + pure applyPlan fold, runs BEFORE the lite layout/camera passes, AUTHOR ALWAYS WINS (fills only FREE placements + unset camera intent); plans are FRACTIONAL coords (aspect-independent). HeuristicDirector (DEFAULT, pure/local) scores rule-of-thirds slots in an aspect-scaled safe area by focal weight + headroom + balance and picks a camera preset NAME by structural beat intent (first=establishing, 1 focal=push-in, ≥3=pull-out, else hold) — MECHANISM here, move RECIPES stay DATA in library/camera/presets.json. LlmDirector (opt-in kind:"llm") shells to claude -p --output-format json (keyless, no-tools) ONCE, validates vs the schema + live preset table, caches the plan content-addressed under .cache/director/ → render replays the FIXED plan (byte-deterministic + offline, the TTS pattern); any failure → heuristic fallback. Same seam is the future script→IR expander's spine. Tier-B GPU (M6, DONE — the ONE non-byte-exact tier): plugins/gpu-effects adopts pixi-filters (CRT/bloom/shockwave/godray/glitch) + a gl-transition dissolve into the generic effects/transitions registries; DOUBLE-GATED (webpack-DefinePlugin LOAD gate on render.ts --gpu + a PixiHost.gpuActive() real-hardware-WebGL RUNTIME self-gate) so the CPU default never loads it (stays byte-exact); each draw is a pure fn of frame, GPU the only non-determinism — verified PERCEPTUALLY with VMAF (ffmpeg libvmaf), NEVER cmp. AI asset-gen (M9, DONE — mechanism): src/cli/imagegen.ts (factory:imagegen) content-addresses {prompt,seed,model,steps,size,negative,guidance} → reuses a cached PNG OR runs .venv-sd OpenVINO SD on the Iris Xe iGPU ONCE → registers an asset catalog entry the render replays (AI touches ONLY the offline library, no provider needed → core stays plugin-free); HONEST env note: the OV SD models hit a transformers/CLIPFeatureExtractor mismatch on this box → imagegen writes a DETERMINISTIC placeholder PNG (records it in provenance, build never fails); re-run after pinning transformers / re-exporting a base/turbo SD with optimum export=True for real generations. Capability gaps (NOT yet built): articulated phoneme→viseme lip-sync (we approximate via the energy envelope, not per-phoneme shapes) · whisper-PRECISION caption page-grouping · LLM script→IR front-end (the M5 LLM seam is its spine) · real generated SD assets (pending the .venv-sd model fix). (DONE since: the ADR-001 remote LibraryResolver — factory:publish-library/factory:fetch-library — and the specified bundle/export format in docs/factory/BUNDLE-FORMAT.md.)

Repo layout

src/engine (ADR-005/006/007 plugin core: extension-point registries + EngineAPI + plugin/manifest contract + loader + the generic generator socket generator.ts; specializes in NOTHING — no domain entity, front-AND-back) · src/ir (Zod IR; Story IR is generic cast + show[] — no characters entity; rig def is {uri, provider, spec(opaque z.record(unknown))} — no CharacterSpec) · src/library (registry/resolver/lockfile; provider comes from catalog DATA, never guessed in core) · src/project (project bundle: manifest + scene + lock) · src/pipeline (passes; lower.ts is GENERIC — builds layers ENTIRELY from each beat's show[], nothing force-injected, no demo baked in) · src/render (generic compositor + the GeneratorLayer/AssetLayer/ShapeLayer sockets; dispatches a rig layer via providers.get(rigDef.provider)) · src/cli (incl. thin factory:gen → active provider plugin's generator; lockRefsForScene walks the scene) · render-entry.tsx + plugins/enabled.ts (composition root + plugin manifest — OUTSIDE src/ so core names no plugin) · plugins/<id>/ (capability = code: plugin.json + index.ts register(api); built-ins = core-generators[the generator implementations] / core-rigs[dragonbones provider + DragonBones runtime] / blob-creature[blob-creature provider — OWNS CharacterSpec + builder + generator]) · library/ (shared asset catalog; characters/ is just one namespace) · projects/<id>/ (reproducible video bundles) · examples/ (source scripts + specs) · docs/ (spec, ADRs, factory standard). NOTE: src/factory (ADR-006), src/generators + src/rig (ADR-007) are deleted from core — generator/rig CODE now lives in plugins/core-generators + plugins/core-rigs; the delete-the-plugin test (move a plugin dir aside → src/ still typechecks) keeps the arrow plugin→core.

Three pillars (ADR-001): the engine (generic; renders any provider) · the library (shared, content-addressed, versioned reusable assets) · the project (projects/<id>/: one reproducible video = manifest + scene.json + project.lock + media). library : project :: npm package : app — i.e. the library is the OPT-IN PUBLIC-SHARING layer (publish a unit out / import shared units in), NOT the default home for a project's own reusable units: those are defined + reused PROJECT-LOCALLY and never leave the project unless deliberately published (golden rule 6). Providers (procedural / dragonbones / …) sit behind one AssetProvider-style interface; procedural (SVG/DOM) is preferred — deterministic on any GL backend, no software-GL disk bloat. ADR-001 boundary contracts are now FORMAL TS interfaces (M7a, src/library/interfaces.ts): the storage seam LibraryResolver (catalog name@version→hash + lockfile pin/verify) + the library-side ref→def adapter AssetRefResolver (toRigDef/toAssetDef/toStyleKit/toClip/toPalette/toGeneratorPreset/expandGeneratorRef, keyed on URI scheme); Library implements both (compile-time assertion, ZERO behavior change — an alternate/remote backend is a drop-in). The render-time provider half (instantiate/render/dispose) lives in the provider plugins (providers.get(id)), not core. Icon library (M7b): factory:ingest-icons ingests open-license SVGs (lucide-static, ISC) OFFLINE into the core-objects ObjectSpec (kind:"icon", geometry only) under the icons catalog namespace, content-addressed + provenance/license recorded; renders as a form-shadeable badge+glyph (the stylekit shades it for free).

Plugin model (ADR-005/006/007, DONE): the engine core is minimal and specializes in NOTHING — back-end AND front-end AND code location — it owns generic extension-point registries + an EngineAPI; capability is contributed by plugins (code), content stays in the library (data). Live extension points: generators (built-ins ship as core-generators [scatter/water/particles/fire/crowd] + core-dataviz [the chart generator: bar/line/pie via d3-shape/d3-scale, draw-on; + the map generator (M7c): geographic maps via d3-geo projections+geoPath + topojson-client — geometry is DATA (inline TopoJSON/GeoJSON in params, selected by a generator-preset like world-map→library/maps/), domain-clean (no country hardcoded), choropleth + draw-on, pure→deterministic]), providers (core-rigs[dragonbones] / blob-creature[blob-creature] / core-objects[object — simple flat-vector props]), effects (the per-layer effects[] stack = the core-effects plugin: blur/glow/drop_shadow/color_grade/turbulence/displace/vignette/grain/motion_blur via SVG/CSS filters + @remotion/motion-blur, composed in order on top of §11.1 shading + parallax; ADR-003 #2), transitions (the scene-boundary presentations = the core-transitions plugin: fade/wipe/slide/iris via @remotion/transitions, mask via SVG reveal, match-cut/morph-match via flubber shared-element, camera-continuous — core Composition.tsx resolves transitions.get(kind), NO hardcoded switch); stubs reserved: layerTypes/passes. A rig is a provider-rendered GENERIC layer: core sees only rigDef.provider (id) + an OPAQUE spec; the named provider validates/interprets the spec with its OWN schema. blob-creature is a PROVIDER PLUGIN that owns CharacterSpec + builder + source-material generator — "character" is NOT a core entity (no characterStyles, no CharacterSpec in core; future chart/widget/diagram are peer providers). ADR-007: the FRONT-END is generic too — lower.ts builds layers ENTIRELY from each beat's show[] (no demo baked in), the Story IR is a generic cast (no characters), and generator/rig CODE lives in plugins (src/generators/src/rig deleted from core). New capability (providers, effects, transitions, text, color, IR passes) is authored AS A PLUGIN — never a core edit; loadPlugins() validates each plugin.json, orders by deps, calls register(api) before render. A plugin obeys the golden rules and is gated by verify-render — incl. the domain-clean grep (src/ code names no demo/subject/provider word) + the delete-the-plugin test (move a plugin dir aside → src/ still typechecks; arrow is plugin→core).

How we work (process)

  • New capability → superpowers:brainstorming → spec → writing-plans or a Workflow. Don't implement before design is approved.
  • Build/author/organize a SCENE or project (compose objects, place + animate them, the file roles, the two reuse levels, the four separated concerns, the flat-shape look) → .claude/skills/building-scenes. Read this before creating or editing a project's content.
  • Produce a news / geopolitics EXPLAINER REEL (Indian-language/Hinglish 9:16 short: native voiceover, animated maps with routes/markers/fly-to, cinematic-dark look, music + SFX, coherent narrative) → .claude/skills/producing-news-reels. Covers the voice engine (Sarvam Bulbul), Devanagari/Mukta typography, the OFFICIAL Government-of-India map (world-in, legally required), the map toolkit, story STRUCTURE, the audio-design layer, and the stale-artifact gotchas.
  • Produce a ~5-min short DOCUMENTARY (multi-ACT Hinglish geopolitics/defense piece built from journalism EVIDENCE: real news-article screenshots + a ubiquity montage, real news video clips, Wikimedia archival stills with Ken Burns, cinematic maps, stage-aware music/SFX) → .claude/skills/producing-documentaries. Builds on producing-news-reels; adds act structure + the evidence CLIs (factory:photo/newsshot/newsclip) + Ken Burns + stage-aware audio + the 5-min render reality. Spec: docs/superpowers/specs/2026-07-08-documentary-v1-design.md.
  • Add a generator → .claude/skills/add-generator.
  • Publish/share a reusable unit to the library (opt-in, cross-project) → .claude/skills/new-library-entry.
  • Before claiming a render works → .claude/skills/verify-render.
  • After any workflow/milestone → .claude/skills/refine-standard (folds learnings back into this standard).

Self-refining standard

This standard grows (add skills + decisions) and self-corrects (delete/fix disproven rules). After each milestone, run refine-standard: append to docs/factory/DECISIONS.md, update these golden rules only for durable lessons, and create skills for new repeatable processes. Keep CLAUDE.md tight — push detail into skills/spec.