Skip to content

Anuva Python Server API Implementation Plan

Created: 2026-05-21 13:30 UTC

Goal

Implement a worker-facing /api/worker/v1 API for the Anuva Python Server that fully replaces the current /api/render-jobs renderer API and matches the contract defined in docs/AnuvaPythonServer_SystemDescription.md.

The resulting backend must let the Python worker:

  1. Discover or claim one render job at a time
  2. Fetch a complete render package
  3. Report status, heartbeat, and failures
  4. Upload the render result through signed S3 upload
  5. Finalize the job and update presentation-level final video pointers

Non-Goals

  • Changing Python worker code beyond the contract assumptions already documented
  • Changing Unity runtime behavior beyond consuming the documented package contract
  • Reworking presentation generation, scene generation, or Mastra orchestration flows
  • Adding a legacy compatibility layer that preserves /api/render-jobs
  • Shipping direct multipart upload fallback in V1

Current Behavior

  • Render jobs are persisted in video_generation_jobs.
  • The current renderer surface is limited to:
  • GET /api/render-jobs
  • PATCH /api/render-jobs/{jobId}
  • Current auth uses x-renderer-secret, not bearer token worker auth.
  • src/app/(frontend)/app/presentation/actions.ts creates render jobs with configSnapshot.publicConfig, which is the closest existing renderer-facing config snapshot.
  • There is no dedicated claim endpoint, package endpoint, heartbeat endpoint, signed upload issuance/confirmation flow, or failure diagnostics persistence model.
  • video_generation_jobs.progress is stored as 0..100, while the Python worker contract expects 0.0..1.0.
  • video_generation_jobs currently lacks worker-native statuses claimed and uploadFailed.

Target Behavior

  • The backend exposes the full worker contract under /api/worker/v1.
  • /api/render-jobs is removed.
  • RenderJobPackage.config is exactly PresentationConfigPublic.
  • The package endpoint returns a normalized asset manifest with signed URLs and safe basenames.
  • Internal progress remains 0..100 in video_generation_jobs, but the worker API translates progress to and from 0.0..1.0.
  • Worker auth uses Authorization: Bearer <ANUVA_WORKER_API_TOKEN>.
  • Worker state is split into:
  • render_workers for latest heartbeat and worker state
  • render_worker_reports for append-only failure diagnostics and related reports
  • Final render upload uses presigned S3 upload and confirmation only.

Planned Flow

flowchart TD
  queued["video_generation_jobs: queued"] --> next["GET /api/worker/v1/jobs/next"]
  next --> claim["POST /jobs/{jobId}/claim"]
  claim --> package["GET /jobs/{jobId}/package"]
  package --> config["config = PresentationConfigPublic"]
  package --> assets["signed asset manifest"]
  config --> worker["Python worker rewrites to localhost URLs"]
  assets --> worker
  worker --> status["PATCH /jobs/{jobId}/status"]
  worker --> heartbeat["POST /workers/{workerId}/heartbeat"]
  worker --> signed["POST /jobs/{jobId}/uploads/signed"]
  signed --> s3["PUT to S3"]
  s3 --> confirm["POST /jobs/{jobId}/uploads/confirm"]
  confirm --> complete["POST /jobs/{jobId}/complete"]
  worker --> fail["POST /jobs/{jobId}/fail"]
  worker --> cancel["POST /jobs/{jobId}/cancel"]

Assumptions

  • /api/worker/v1 fully replaces /api/render-jobs in one change.
  • RenderJobPackage.config is PresentationConfigPublic with no wrapper.
  • Signed asset URLs are generated on demand and are not persisted in Payload documents.
  • Signed upload confirmation creates or resolves the final media_assets record before job completion.
  • Direct multipart upload fallback is intentionally omitted from V1.

Affected Files And Modules

Worker API routes

  • Remove or replace:
  • src/app/api/render-jobs/route.ts
  • src/app/api/render-jobs/[jobId]/route.ts
  • Add:
  • src/app/api/worker/v1/jobs/next/route.ts
  • src/app/api/worker/v1/jobs/available/route.ts
  • src/app/api/worker/v1/jobs/[jobId]/route.ts
  • src/app/api/worker/v1/jobs/[jobId]/claim/route.ts
  • src/app/api/worker/v1/jobs/[jobId]/package/route.ts
  • src/app/api/worker/v1/jobs/[jobId]/status/route.ts
  • src/app/api/worker/v1/jobs/[jobId]/uploads/signed/route.ts
  • src/app/api/worker/v1/jobs/[jobId]/uploads/confirm/route.ts
  • src/app/api/worker/v1/jobs/[jobId]/complete/route.ts
  • src/app/api/worker/v1/jobs/[jobId]/fail/route.ts
  • src/app/api/worker/v1/jobs/[jobId]/cancel/route.ts
  • src/app/api/worker/v1/workers/[workerId]/heartbeat/route.ts

Shared render-worker helpers

  • Add a shared worker/render helper area under src/lib, for example:
  • worker auth validation
  • render job serializers
  • progress conversion helpers
  • claim and transition helpers
  • render package builder
  • S3 presign/confirm helpers

Schema and job-creation touchpoints

  • src/collections/anuvax/index.ts
  • src/payload-types.ts
  • generated import map output
  • src/app/(frontend)/app/presentation/actions.ts
  • src/lib/presentation-config/schema.ts
  • optional shared media/storage helpers if package assembly or upload confirmation needs them

Test harness and coverage

  • tests/integration/helpers/fake-payload.ts
  • tests/integration/helpers/presentation-fixtures.ts
  • new worker API integration tests
  • existing render-job integration tests that assume old route semantics

Affected Docs

  • docs/AnuvaPythonServer_SystemDescription.md
  • docs/core/WorkersAndJobs.md
  • docs/core/SystemArchitecture.md
  • docs/core/PresentationConfig.md
  • docs/state/RouteMap.md
  • docs/state/CollectionMap.md
  • docs/state/CurrentImplementationMap.md
  • docs/state/KnownGaps.md
  • docs/changes/2026-05-21-1330-anuva-python-server-api/ImplementationLog.md

Data And Schema Impact

video_generation_jobs

  • Add statuses:
  • claimed
  • uploadFailed
  • Keep persisted progress as 0..100.
  • Add worker-facing metadata fields:
  • workerId
  • claimedAt
  • errorCode
  • statusMessage
  • Keep existing core relationships:
  • workspace
  • presentation
  • compilation
  • resultMedia

render_workers

  • New workspace-scoped collection for latest worker state
  • Proposed fields:
  • workspace
  • workerId
  • status
  • activeJob
  • details
  • lastHeartbeatAt

render_worker_reports

  • New workspace-scoped append-only collection for diagnostics and worker reports
  • Proposed fields:
  • workspace
  • workerId
  • job
  • reportType
  • errorCode
  • errorMessage
  • diagnostics
  • timestamps

Generated artifacts

  • Regenerate:
  • src/payload-types.ts
  • Payload import map

Implementation Phases

Phase 1. Contract and schema foundation

  • Add worker auth helper for bearer-token validation
  • Define shared route-level serializers and parsers for:
  • RenderJob
  • RenderJobSummary
  • RenderJobPackage
  • WorkerHeartbeat
  • signed-upload responses
  • UploadedMedia
  • Update video_generation_jobs
  • Add render_workers
  • Add render_worker_reports
  • Regenerate Payload types and import map

Phase 2. Core job lifecycle APIs

  • Implement next, available, and summary endpoints
  • Implement atomic claim logic
  • Implement shared status transition rules
  • Implement progress conversion at the API boundary
  • Remove old route-specific assumptions tied to x-renderer-secret

Phase 3. Render package and asset manifest

  • Build package assembly from existing render-job state
  • Use configSnapshot.publicConfig as RenderJobPackage.config
  • Dynamically collect referenced media assets and emit signed URLs plus safe basenames
  • Ensure the package is fully sufficient for worker-side local materialization

Phase 4. Worker operational APIs

  • Implement status, heartbeat, fail, and cancel
  • Persist latest worker state in render_workers
  • Persist diagnostics history in render_worker_reports
  • Ensure invalid terminal rewrites are rejected

Phase 5. Upload and completion flow

  • Add S3 presign helper using existing AWS SDK dependencies
  • Implement uploads/signed
  • Implement uploads/confirm
  • Create or resolve the final media_assets record during confirmation
  • Implement complete so presentation final-video pointers update only after confirmed upload

Phase 6. Cleanup, tests, and docs

  • Remove /api/render-jobs
  • Update integration harness and add contract coverage
  • Update all impacted docs
  • Record final implementation details in ImplementationLog.md

Ordered Tasks

  • [x] 1. Add shared worker auth and contract serialization helpers.
  • [x] 2. Update video_generation_jobs schema and add render_workers plus render_worker_reports.
  • [x] 3. Regenerate Payload types and import map.
  • [x] 4. Extract shared claim, status-transition, and progress-conversion helpers.
  • [x] 5. Implement GET /api/worker/v1/jobs/next.
  • [x] 6. Implement GET /api/worker/v1/jobs/available.
  • [x] 7. Implement GET /api/worker/v1/jobs/{jobId}.
  • [x] 8. Implement POST /api/worker/v1/jobs/{jobId}/claim.
  • [x] 9. Implement GET /api/worker/v1/jobs/{jobId}/package.
  • [x] 10. Implement PATCH /api/worker/v1/jobs/{jobId}/status.
  • [x] 11. Implement POST /api/worker/v1/workers/{workerId}/heartbeat.
  • [x] 12. Implement POST /api/worker/v1/jobs/{jobId}/fail.
  • [x] 13. Implement POST /api/worker/v1/jobs/{jobId}/cancel.
  • [x] 14. Implement POST /api/worker/v1/jobs/{jobId}/uploads/signed.
  • [x] 15. Implement POST /api/worker/v1/jobs/{jobId}/uploads/confirm.
  • [x] 16. Implement POST /api/worker/v1/jobs/{jobId}/complete.
  • [x] 17. Update render-job creation to remain compatible with the new worker package contract.
  • [x] 18. Remove the old /api/render-jobs route surface and update any internal references.
  • [x] 19. Update integration test harnesses and add worker API contract tests.
  • [x] 20. Update docs and write final notes to ImplementationLog.md.

Verification Plan

Run:

  • pnpm generate:types
  • pnpm generate:importmap
  • pnpm lint
  • pnpm typecheck
  • pnpm test:int

Test Cases And Scenarios

  1. GET /jobs/next returns null or 204 when no queued jobs exist.
  2. GET /jobs/available returns only safe summary data and does not leak config or signed asset URLs.
  3. POST /claim atomically claims a queued job and returns 409 on conflict.
  4. GET /package returns PresentationConfigPublic exactly in config.
  5. GET /package returns a valid asset manifest with unique asset IDs and safe basenames.
  6. Progress conversion is correct:
  7. API input 0.42 persists as 42
  8. persisted 50 returns as 0.5
  9. PATCH /status accepts valid transitions and rejects invalid terminal rewrites.
  10. POST /heartbeat upserts latest worker state into render_workers.
  11. POST /fail stores diagnostics in render_worker_reports and marks the job failed.
  12. POST /uploads/signed returns a valid signed upload response shape.
  13. POST /uploads/confirm creates or resolves media_assets and returns mediaId.
  14. POST /complete marks the job completed and updates presentation final-video pointers only after confirmed upload.
  15. POST /cancel transitions the job to cancelled.
  16. Existing render-job creation still produces a worker-consumable queued job with configSnapshot.publicConfig.

Docs Parity Checklist

  • [ ] Replace /api/render-jobs with /api/worker/v1 in route documentation.
  • [ ] Document the Python worker lifecycle in docs/core/WorkersAndJobs.md.
  • [ ] Document that PresentationConfigPublic is the exact RenderJobPackage.config.
  • [ ] Add render_workers and render_worker_reports to docs/state/CollectionMap.md.
  • [ ] Update system architecture and implementation maps to reflect worker-facing render APIs.
  • [ ] Update known gaps if any intentional V1 limitations remain after implementation.
  • [ ] Update ImplementationLog.md with tasks completed, files changed, tests run, and deviations.

Acceptance Criteria

  • A Python worker using the documented bearer-token contract can acquire, claim, render, upload, and finalize a job using only /api/worker/v1.
  • /api/render-jobs no longer exists in code or docs.
  • RenderJobPackage.config is exactly the persisted PresentationConfigPublic snapshot.
  • The render package includes all required asset references as signed URLs plus safe basenames.
  • video_generation_jobs supports claimed and uploadFailed.
  • API progress uses 0.0..1.0 while persistence remains 0..100.
  • Worker heartbeats persist in render_workers.
  • Failure diagnostics persist in render_worker_reports.
  • Signed upload confirmation returns normalized media data with mediaId.
  • Presentation final-video state updates only after confirmed upload and successful completion.

Rollback And Recovery Notes

  • If the new worker API fails in non-production verification, restore the previous route surface from git before rollout because this plan intentionally removes /api/render-jobs.
  • If schema changes land but worker routes are incomplete, block rollout until the new route family and generated Payload artifacts are in sync.
  • If signed upload confirmation fails after upload, preserve uploadFailed state and diagnostics so the job can be retried without losing the render artifact context.