Skip to content

RAG And Sources

Last refreshed: 2026-05-18

Source Types

Two layers currently exist.

Product RAG collections support source type values on rag_sources such as uploaded/document/video/image-oriented app sources.

Anuva v1 source adapters in src/lib/anuva/source-adapters.ts define canonical upgrade source types:

  • uploaded
  • website
  • web_research
  • rss

Feature flags for these source types live in src/lib/anuva/feature-flags.ts.

Product Ingestion Flow

Main product ingestion uses:

  • src/app/(frontend)/app/rag/actions.ts
  • src/app/(frontend)/app/rag/upload/route.ts
  • src/lib/rag/ingestion/process-rag-source.ts
  • scripts/rag-worker.ts

Flow:

  1. User uploads or creates a RAG source.
  2. App creates rag_sources and rag_ingestion_jobs.
  3. pnpm rag:worker polls queued jobs.
  4. Worker extracts media/document text, chunks it, embeds it, writes rag_chunks, and upserts vectors.
  5. Source status moves to ready or error.
flowchart LR
  source["📄 Upload or web source"] --> ragSource["rag_sources"]
  ragSource --> job["rag_ingestion_jobs"]
  job --> worker["⚙️ rag:worker"]
  worker --> extract["🧹 Extract and chunk"]
  extract --> chunks["rag_chunks"]
  chunks --> vectors["🔎 Vector index"]
  worker --> status["✅ ready or ❌ error"]

Vector Store

src/lib/rag/vector-store.ts manages vector index operations.

Constants:

  • index: anuvax_rag_chunks_v1.
  • embedding dimension: 1536.
  • embedding model: text-embedding-3-small.

Retrieval Flow

src/lib/rag/retrieve.ts retrieves persisted chunks for presentation generation. retrievePresentationRagContextTool merges persisted chunks into Mastra workflow context in presentationVideoWorkflow.

Upgrade-path retrieval helpers also exist in src/lib/anuva/retrieval.ts.

flowchart TD
  presentation["🎬 Presentation generation"] --> selected["selectedRagSources"]
  selected --> retrieve["retrieve.ts"]
  retrieve --> chunks["rag_chunks"]
  chunks --> context["📚 Knowledge context"]
  web["🌐 Described web sources"] --> context
  context --> narrative["📝 Narrative, visuals, slides, direction"]

Web Sources

Web source normalization and ingestion helpers:

  • src/lib/rag/web-source-normalization.ts.
  • src/lib/rag/web-source-ingestion.ts.
  • src/mastra/tools/web/index.ts.

Tests cover normalization of URLs, bare domains, RSS feeds, duplicate/invalid inputs, denied hosts, and research notes.

Workspace Scoping

RAG sources, chunks, and ingestion jobs are workspace-scoped. Retrieval and ingestion paths must preserve workspace IDs and avoid cross-workspace source selection.

Source Usage In Presentation Video

Presentations store selected RAG sources through selectedRagSources. Generation merges selected source context and newly described web sources before narrative, visuals, slides, and direction are produced.