AI Puppet Theatre — Native CrewAI 3D Pipeline

This project replaces the 2D portrait/AnimateDiff/Wav2Lip path with persistent MPFB humanoid actors, NVIDIA Audio2Face-3D facial animation, semantic body and camera direction, and deterministic Blender Eevee rendering.

The active production order is:

Planning Crew -> screenplay HITL
  -> generate/resolve persistent actors -> actor HITL
  -> XTTS -> measure WAV durations
  -> specialized 3D Production Crew semantic plans + Production Supervisor
  -> Stable Audio (optional) -> release ComfyUI
  -> Audio2Face batch -> stop Audio2Face
  -> Blender scene build -> Eevee render -> preview HITL
  -> final encode -> QC

Creative responsibility is intentionally split:

Planning:
Showrunner -> Casting Director -> Screenwriter -> Dramaturg

Production:
Production Designer -> Performance Director -> Cinematographer
-> Lighting Designer -> Foley Designer -> Render Editor
-> Production Supervisor

The production specialists return typed SetDesignPlan, PerformancePlan, CameraPlan, LightingPlan, AudioCuePlan, and RenderPlan outputs. Python assembles the final ScenePlan deterministically. The Production Supervisor reviews the complete semantic plan before rendering and can trigger bounded production-only revisions; it never replaces Pydantic or deterministic asset validation.

Agents select validated semantic primitives. They cannot emit Blender coordinates, bone names, arbitrary asset paths, or invoke GPU tools. The persisted Flow owns the serialized media queue and unloads Ollama for every media window, then reloads it before the next CrewAI task.

Local setup

The controller stays lightweight:

uv sync --extra dev
cp .env.example .env
uv run aithtr-assets preflight
uv run aithtr-assets smoke-a2f --audio /absolute/path/to/16-bit-pcm-speech.wav

Heavy dependencies live outside .venv:

• Blender 4.2+ (pinned LTS recommended), MPFB, MakeHuman system assets, and the faceunits01 ARKit shape-key pack;
• one curated actor-assets.json mapping supported hair, wardrobe, and skin semantics to installed MPFB assets;
• an approved v2 .blend motion package containing imported/retargeted actions for every performance semantic on the MPFB Mixamo-compatible rig;
• NVIDIA Audio2Face-3D NIM v2.0 and the official v2.0 sample client;
• XTTS, ComfyUI only for optional SFX, FFmpeg, Docker Compose, NVIDIA Container Toolkit, and an externally supplied NGC_API_KEY.

Do not put the NGC key in .env, command arguments, logs, or the repository.

Production does not generate a stage or prop from primitives. Import an authored theatre package, inspect its immutable preview, then approve it:

uv run aithtr-assets import-stage --source-blend /absolute/authored/theatre_master.blend --metadata /absolute/authored/theatre.metadata.json
uv run aithtr-assets preview-stage asset://stages/theatre_classic/v1/metadata.json
uv run aithtr-assets approve asset://stages/theatre_classic/v1/metadata.json --reviewer "$USER"

uv run aithtr-assets import-prop --source-blend /absolute/authored/old_radio.blend --metadata /absolute/authored/old_radio.metadata.json
uv run aithtr-assets preview-prop asset://props/old_radio/v1/metadata.json
uv run aithtr-assets approve asset://props/old_radio/v1/metadata.json --reviewer "$USER"

uv run aithtr-assets import-motion --source-blend /absolute/retargeted/actions.blend --metadata /absolute/retargeted/motions.metadata.json --preview-actor /absolute/approved/character.blend
uv run aithtr-assets preview-motion-library asset://motions/core/v2/metadata.json
uv run aithtr-assets approve asset://motions/core/v2/metadata.json --reviewer "$USER"

uv run aithtr-assets preflight exits nonzero until the authored theatre, v2 motions, actor catalog, Audio2Face service, and required prop families are all present, structurally valid, checksum-stable, and approved. Bootstrap plane, cube, and procedural-Euler assets are test fixtures only and are never a production fallback.

Actor candidates are generated deterministically from ActorSpec during the Flow. At the actor gate, inspect each package's turntable.mp4, then approve it:

uv run aithtr-assets list --category actors
uv run aithtr-assets approve asset://actors/ACTOR/VERSION/metadata.json --reviewer "$USER"

Only approved assets may enter a Blender scene. Candidate package content is immutable; approval is stored separately and does not change its checksum.

Run

One command owns Ollama handoff and all local child services:

uv run aithtr --theme "a broken lunar radio" --mood funny --duration 30

For a new terminal session:

tmux new-session -s aithtr 'cd /home/zodrak/crewai/aithtr_crewai && uv run aithtr --theme "a broken lunar radio" --mood funny --duration 30'

The three human gates accept:

approve optional notes
revise screenplay: make the ending less abrupt
revise cast: change the wardrobe
revise animation: use calmer gestures
abort optional reason

Resume with uv run aithtr --resume FLOW_ID.

Tests

uv run pytest

The default suite is hardware-independent. It validates safe and immutable asset URIs, explicit approvals, timeline closure, specialized production-plan schemas and assembly, Production Supervisor retry behavior, Audio2Face output, 30-to-24 FPS mapping, idempotent serialized jobs, GPU handoff, and the complete 3D Flow contract. GitHub Actions runs the same full suite on pull requests and main. Live Blender/MPFB and Audio2Face smoke tests are blocked until uv run aithtr-assets preflight passes.

See docs/3d-pipeline.md for service and package contracts.