Skip to content

Testing And Verification

Last refreshed: 2026-07-28

Bento Verification

Focused Bento tests cover reviewed artifact provenance, manifest-listed file serving, per-file integrity, traversal rejection, exact template catalog load, stable scene focus, immutable save, idempotent retry, conflict pointer preservation, durable draft state, structured AI target scope, and executable content rejection. The regular integration suite continues to cover revision compilation and render-package compatibility.

Browser verification should use the real vendored editor/player with E2E_BYPASS_AUTH=1 only in the test server. Production route behavior still requires authenticated workspace context.

Commands

  • git diff --check: whitespace and patch sanity check.
  • mkdocs build --strict: docs build check when MkDocs is available.
  • pnpm lint: Biome checks.
  • pnpm typecheck: TypeScript strict check.
  • pnpm test:int: Vitest integration/unit suite configured by vitest.config.mts.
  • pnpm test:e2e: Playwright e2e suite.
  • pnpm test: integration plus e2e.
  • pnpm stack:config: secret-free Compose model validation.
  • pnpm stack:verify: containerized Payload, Mastra, RAG vector, and worker API smoke. It runs as a separate no-schema-push process after the health-gated web service has initialized Payload.
  • pnpm local-stack:db -- verify ...: verify the ignored export SHA-256 manifest.
  • pnpm local-stack:db:migrate -- parity ...: compare application counts, hashed keys, schema signatures, sequences, and workspace/RAG state.
  • pnpm local-stack:db:migrate -- probe ...: run targeted Payload, workspace, Mastra, queue, RAG, and vector-query checks on a loopback target.

Test Layout

  • tests/unit: contract, billing, metering, source adapter, web normalization, and Mastra web tool tests.
  • tests/integration: Presentation Video server action integration tests with fake Payload/runtime harness.
  • tests/e2e: route-level Presentation Video flow tests.
  • tests/fixtures: fixture data such as Zoho webhook examples.

Presentation Integration Harness

Read tests/integration/README.md before adding presentation action tests.

Shared helpers:

  • helpers/presentation-fixtures.ts
  • helpers/fake-payload.ts
  • helpers/mock-runtime.ts
  • helpers/presentation-harness.ts

Tests should assert both returned action results and persisted fake DB state.

PresentationConfig v2 Matrix

ANU-13 coverage includes:

  • exact vendored artifact provenance, player manifest, license, and notices
  • Bento schema and CMS compatibility validation
  • complementary required surfaces and equal document structure
  • stable IDs and checksums across unchanged recompilation
  • zero/many items, preserve/replace/clear, fragments, and transient emphasis
  • deterministic seeded arbitrary scene counts and monotonic event ordering
  • actual-duration-estimate timing precedence and pause behavior in integration
  • immutable parent-linked revisions, expected revision/checksum conflicts, last-good pointer preservation, and workspace isolation
  • supported media/checksum enforcement, missing assets, unsafe IDs/locators, executable content, path traversal, and credential/query exclusion
  • config, player-file, asset, and root-package checksums plus tamper detection
  • v2-only presentation fixture export/import validation

Active-path scanning should find no presentation-generation markup path or legacy public-config fixture. UI uses of the English word "reveal" for animation, password controls, or historical docs are unrelated to the compilation contract.

E2E Setup

pnpm test:e2e clears .next and starts its own non-reused dev server on 127.0.0.1:3100, so a developer server on port 3000 and its public flags cannot be inherited. It uses:

  • E2E_BYPASS_AUTH=1
  • NEXT_PUBLIC_E2E_PRESENTATION_MOCK=1
  • PAYLOAD_DB_PUSH=0

Use e2e tests for route-level workflow behavior such as setup validation, scene generation, preview controls, stale-state blocking, and per-scene assistants.

When To Run What

  • Docs-only changes: run git diff --check; run mkdocs build --strict if MkDocs is available; run no implementation tests unless implementation code changed.
  • Server action changes: pnpm lint, pnpm typecheck, and pnpm test:int.
  • Presentation flow changes: add or run pnpm test:e2e coverage when route-level behavior changes.
  • Payload schema changes: pnpm generate:types, optionally pnpm generate:importmap, then typecheck and targeted tests.
  • Billing changes: run tests/unit/zoho-billing.test.ts and tests/unit/ai-coin-metering.test.ts through pnpm test:int.
  • RAG/source changes: run RAG/source unit tests and relevant presentation integration tests.
  • Local stack changes: run docker compose config, clean build/start with health waits, version/extension queries, persistence checks, stack:verify, local-stack guard tests, the full repository checks, and relevant Playwright coverage.

Local Stack Verification

Static checks do not replace Docker runtime evidence. A complete local-stack verification records:

  1. pnpm stack:config without printing resolved secrets.
  2. pnpm stack:up from clean images/containers and pnpm stack:status showing all three services healthy.
  3. loopback HTTP/TCP probes for ports 3000, 5432, and 5050.
  4. running PostgreSQL, pgvector, pgAdmin, and Node versions plus image digests.
  5. pnpm stack:verify for application/database integration.
  6. sentinel state before and after restart and routine down/up to prove named volume persistence.
  7. database-tool plan/refusal tests; never run a real export or restore in ANU-6.
  8. pnpm lint, pnpm typecheck, pnpm test:int, relevant Playwright smoke, strict MkDocs build, and recursive docs index validation.

If Docker Desktop is unavailable, record every runtime item as skipped and keep the static Compose and unit evidence distinct.

Database Cutover Verification

A Supabase-to-local acceptance run additionally records:

  1. read-only source and target inventories without row contents or credentials.
  2. Supabase CLI version, dump dry run, export completion, checksum verification, ignore rules, and absence from Git.
  3. reviewed schema preparation with exact removed ownership/ACL statement counts.
  4. transactional restore into a fresh loopback database.
  5. semantic parity of table counts, hashed primary keys, constraints, indexes, sequences, Payload migrations, Mastra tables, RAG tables, and workspace-null counts despite expected PostgreSQL/pgvector patch-version differences.
  6. targeted cross-workspace mismatch counts, empty queue state, migrated vector index presence, and a successful nearest-neighbor result.
  7. container health, Payload migration status, public/admin/login probes, stack:verify, bounded worker startup, restart and down/up persistence.
  8. a no-schema-push Payload status check through the retained Supabase URI, followed by confirmation that the running CMS still targets local Postgres.
flowchart TD
  change["Change type"] --> docsOnly{"Docs only?"}
  docsOnly -->|yes| docsChecks["git diff --check and MkDocs build"]
  docsOnly -->|no| codeArea{"Touched area"}
  codeArea --> actions["Server actions: lint, typecheck, test:int"]
  codeArea --> presentation["Presentation flow: integration or e2e coverage"]
  codeArea --> payload["Payload schema: generate types/import map"]
  codeArea --> billing["Billing: targeted unit tests"]
  codeArea --> rag["RAG/source: RAG tests and integration coverage"]

Docs Parity

Every behavior-changing implementation should update:

  • docs/core for durable system behavior.
  • docs/state for paths, commands, maps, gaps, or operational state.
  • docs/changes/<change> for change-specific reasoning when using the large-change workflow.

When local work affects product behavior, repository ownership, cross-repository contracts, release readiness, or publishing status, include product-main sync notes in the GitHub issue, PR completion report, or final implementation report.

Skipped Checks

When a check cannot be run, report:

  • which check was skipped
  • why it was skipped
  • what evidence remains