Skip to content

Workers And Jobs

Last refreshed: 2026-05-21

Worker Scripts

  • pnpm rag:worker: runs scripts/rag-worker.ts.
  • pnpm presentation:worker: runs scripts/presentation-worker.ts.
  • pnpm anuva:worker: runs scripts/anuva-worker.ts.

Most scripts load .env through dotenv-cli.

RAG Worker

The RAG worker polls rag_ingestion_jobs where status = queued.

For each job it:

  1. Marks the job inProgress.
  2. Runs processRagSource.
  3. Updates the source and job to completed or failed state.

It also calls ensureRagVectorIndex at startup.

stateDiagram-v2
  [*] --> queued
  queued --> inProgress: worker claims job
  inProgress --> completed: artifacts saved
  inProgress --> failed: error recorded
  failed --> queued: explicit requeue
  completed --> [*]

Presentation Worker

The presentation worker polls presentation_generation_jobs where status = queued.

It handles:

  • image generation for scenes that need generated visuals.
  • voiceover audio generation.
  • progress calculation from audio/image totals and done/failed counts.
  • job status messages.
  • generation failure state.

It uses OpenAI image generation, ElevenLabs timing/audio helpers, RAG retrieval, prompt resolution, and Payload writes.

Anuva Worker

scripts/anuva-worker.ts is the upgrade-path worker entry point. Confirm the current queue semantics in src/lib/anuva/worker-queue.ts before extending it. Upgrade-path worker state is stored through src/lib/anuva/repository.ts; Payload no longer registers V1 upgrade workflow collections.

flowchart LR
  queue["worker queue"] --> anuvaWorker["scripts/anuva-worker.ts"]
  anuvaWorker --> orchestration["Anuva orchestration service"]
  orchestration --> repository["In-memory AnuvaRepository"]
  repository -. future persistence gap .-> durable["Durable repository store"]

Job Collections

  • rag_ingestion_jobs: RAG ingestion queue.
  • presentation_generation_jobs: audio/image generation queue for Presentation Videos.
  • video_generation_jobs: render/final video queue and worker-visible status tracking.
  • render_workers: latest external render-worker heartbeat and active-job state.
  • render_worker_reports: append-only external render-worker diagnostics.

Python Render Worker

The Anuva Python Server now talks to a dedicated worker API under /api/worker/v1.

Core routes:

  • GET /api/worker/v1/jobs/next
  • GET /api/worker/v1/jobs/available
  • GET /api/worker/v1/jobs/{jobId}
  • POST /api/worker/v1/jobs/{jobId}/claim
  • GET /api/worker/v1/jobs/{jobId}/package
  • PATCH /api/worker/v1/jobs/{jobId}/status
  • POST /api/worker/v1/workers/{workerId}/heartbeat
  • POST /api/worker/v1/jobs/{jobId}/uploads/signed
  • POST /api/worker/v1/jobs/{jobId}/uploads/confirm
  • POST /api/worker/v1/jobs/{jobId}/complete
  • POST /api/worker/v1/jobs/{jobId}/fail
  • POST /api/worker/v1/jobs/{jobId}/cancel

video_generation_jobs.progress remains persisted as 0..100, while the worker API translates to 0.0..1.0.

RenderJobPackage.config is the persisted PresentationConfigPublic snapshot from configSnapshot.publicConfig.

stateDiagram-v2
  [*] --> queued
  queued --> claimed: POST claim
  queued --> cancelled: POST cancel
  claimed --> inProgress: PATCH status
  claimed --> failed: POST fail
  claimed --> cancelled: POST cancel
  inProgress --> uploading: PATCH status or upload confirm
  inProgress --> uploadFailed: PATCH status
  inProgress --> failed: POST fail
  inProgress --> cancelled: POST cancel
  uploading --> completed: POST complete
  uploading --> uploadFailed: PATCH status
  uploadFailed --> uploading: retry upload
  uploadFailed --> failed: POST fail
  completed --> [*]
  failed --> [*]
  cancelled --> [*]

Status And Retry Notes

RAG and presentation workers poll in-process and update Payload state directly. The external Python render worker uses /api/worker/v1, while upgrade-path worker state is repository-owned and retry behavior is partly represented in src/lib/anuva/retry-policy.ts.

When adding worker behavior, keep:

  • explicit status values.
  • startedAt, completedAt, errorMessage, and progress fields current.
  • idempotency and duplicate-processing behavior tested.
  • workspace and presentation relationships validated.

Required Environment

Workers commonly need:

  • DATABASE_URI
  • PAYLOAD_SECRET
  • NEXT_PUBLIC_SERVER_URL
  • S3_*
  • ANUVA_WORKER_API_TOKEN for the external render worker API
  • AI provider keys for generation tasks
  • ELEVENLABS_API_KEY for TTS