Skip to content

Workers And Jobs

Last refreshed: 2026-07-22

Worker Scripts

  • pnpm rag:worker: runs scripts/rag-worker.ts.
  • pnpm presentation:worker: runs scripts/presentation-worker.ts.
  • pnpm anuva:worker: runs scripts/anuva-worker.ts.

Most scripts load .env through dotenv-cli.

The local Compose stack starts the RAG and presentation workers as separate services after the web service is healthy. Both use the Compose-local Postgres connection with Payload schema push disabled. The upgrade-path Anuva worker remains a manual process.

RAG Worker

The RAG worker polls rag_ingestion_jobs where status = queued.

For each job it:

  1. Marks the job inProgress.
  2. Runs processRagSource.
  3. Updates the source and job to completed or failed state.

It also calls ensureRagVectorIndex at startup.

stateDiagram-v2
  [*] --> queued
  queued --> inProgress: worker claims job
  inProgress --> completed: artifacts saved
  inProgress --> failed: error recorded
  failed --> queued: explicit requeue
  completed --> [*]

Presentation Worker

The presentation worker polls presentation_generation_jobs where status = queued.

It handles:

  • image generation for scenes that need generated visuals.
  • voiceover audio generation.
  • progress calculation from audio/image totals and done/failed counts.
  • job status messages.
  • generation failure state.

It uses OpenAI image generation, ElevenLabs timing/audio helpers, RAG retrieval, prompt resolution, and Payload writes.

Anuva Worker

scripts/anuva-worker.ts is the upgrade-path worker entry point. Confirm the current queue semantics in src/lib/anuva/worker-queue.ts before extending it. Upgrade-path worker state is stored through src/lib/anuva/repository.ts; Payload no longer registers V1 upgrade workflow collections.

flowchart LR
  queue["worker queue"] --> anuvaWorker["scripts/anuva-worker.ts"]
  anuvaWorker --> orchestration["Anuva orchestration service"]
  orchestration --> repository["In-memory AnuvaRepository"]
  repository -. future persistence gap .-> durable["Durable repository store"]

Job Collections

  • rag_ingestion_jobs: RAG ingestion queue.
  • presentation_generation_jobs: audio/image generation queue for Presentation Videos.
  • video_generation_jobs: render/final video queue and worker-visible status tracking.
  • render_workers: latest external render-worker heartbeat and active-job state.
  • render_worker_reports: append-only external render-worker diagnostics.

Python Render Worker

The Anuva Python Server now talks to a dedicated worker API under /api/worker/v1.

Core routes:

  • GET /api/worker/v1/jobs/next
  • GET /api/worker/v1/jobs/available
  • GET /api/worker/v1/jobs/{jobId}
  • POST /api/worker/v1/jobs/{jobId}/claim
  • GET /api/worker/v1/jobs/{jobId}/package
  • PATCH /api/worker/v1/jobs/{jobId}/status
  • POST /api/worker/v1/workers/{workerId}/heartbeat
  • POST /api/worker/v1/jobs/{jobId}/uploads/signed
  • POST /api/worker/v1/jobs/{jobId}/uploads/confirm
  • POST /api/worker/v1/jobs/{jobId}/complete
  • POST /api/worker/v1/jobs/{jobId}/fail
  • POST /api/worker/v1/jobs/{jobId}/cancel

video_generation_jobs.progress remains persisted as 0..100, while the worker API translates to 0.0..1.0.

configSnapshot.publicConfig is the exact validated PresentationConfig v2 aggregate revision selected for the job. Package creation validates the aggregate and exact consumer compatibility again.

The worker receives an immutable anuva/bento-render-package containing config and root checksums, exact contract/protocol/player/Bento/renderer pins, the vendored player file manifest, three surface documents, timeline, and content-addressed package-relative asset descriptors. Every player entry includes byte count and SHA-256.

Signed or expiring asset URLs are returned only under transport.assetDownloads. They are not part of PresentationConfig or the package checksum. Missing assets, compatibility mismatch, invalid checksum, unsafe locator, query string, credential, or traversal input fails package assembly before a worker can render it.

stateDiagram-v2
  [*] --> queued
  queued --> claimed: POST claim
  queued --> cancelled: POST cancel
  claimed --> inProgress: PATCH status
  claimed --> failed: POST fail
  claimed --> cancelled: POST cancel
  inProgress --> uploading: PATCH status or upload confirm
  inProgress --> uploadFailed: PATCH status
  inProgress --> failed: POST fail
  inProgress --> cancelled: POST cancel
  uploading --> completed: POST complete
  uploading --> uploadFailed: PATCH status
  uploadFailed --> uploading: retry upload
  uploadFailed --> failed: POST fail
  completed --> [*]
  failed --> [*]
  cancelled --> [*]

Status And Retry Notes

RAG and presentation workers poll in-process and update Payload state directly. The external Python render worker uses /api/worker/v1, while upgrade-path worker state is repository-owned and retry behavior is partly represented in src/lib/anuva/retry-policy.ts.

When adding worker behavior, keep:

  • explicit status values.
  • startedAt, completedAt, errorMessage, and progress fields current.
  • idempotency and duplicate-processing behavior tested.
  • workspace and presentation relationships validated.
  • immutable package identity separate from transport authorization.
  • the last accepted compilation current when package validation fails.

Required Environment

Workers commonly need:

  • DATABASE_URI
  • PAYLOAD_SECRET
  • NEXT_PUBLIC_SERVER_URL
  • S3_*
  • ANUVA_WORKER_API_TOKEN for the external render worker API
  • AI provider keys for generation tasks
  • ELEVENLABS_API_KEY for TTS