The Four RAG Levels — A Decision Framework for Enterprise Systems

In plain terms: L1 guesses, L2 narrows, L3 reasons, L4 debates with itself until it's confident. Pick your complexity based on what the problem actually needs — not what the architecture diagram looks coolest.

A note on the retrieval assumption: All four levels above assume that retrieval works by similarity — embed the query, embed document chunks, find the nearest vectors. This is the right default for corpus-level search across thousands of documents. But for structured professional documents (financial filings, clinical guidelines, legal agreements, regulatory disclosures), there is an emerging alternative: reasoning-based retrieval, where the LLM navigates a document's structure directly instead of searching a vector space. Section 6.4 introduces this paradigm, and Section 10 applies it across all four domains.

Architectural framing: The levels are not milestones to progress through linearly — they are tools. A production system at a bank might use L1 for FAQ deflection, L2 for product search, L3 for compliance checks, and L4 for portfolio incident analysis. The architectural decision is: which level of retrieval sophistication does this specific problem require, and can the organization afford the ontology and latency cost of higher levels?

Use this framework when designing a RAG system for any enterprise problem:

Question: Does the answer exist in a single document?
  YES → L1 is sufficient
  NO  → continue

Question: Does the query contain exact identifiers (codes, tickers, drug names, amounts)?
  YES → L2 minimum (hybrid required)
  NO  → L1 may suffice if purely semantic

Question: Does the answer require reasoning across multiple facts in relationship to each other?
  YES → L3 (GraphRAG) if relationships are pre-definable
  NO  → L2 sufficient

Question: Is the relationship structure pre-known and consistent?
  YES → L3 (invest in ontology)
  NO  → L4 (let the agent discover the retrieval path)

Question: Does answering require iterative refinement — "I need more context before I can answer"?
  YES → L4 (agentic loop)
  NO  → L3 sufficient

Question: Is the latency tolerance under 2 seconds AND context per turn under 4,000 tokens?
  YES → L1 or L2 with any backend
  NO  → Evaluate Mamba-backed L3/L4 before concluding infeasible
        (Mamba's 5× throughput + constant memory enables 3-5s L4 responses
         in configurations where Transformers require 15-20s due to KV cache pressure)

Question: Does answering require ingesting a document longer than 8,000 tokens in a single pass?
  (examples: full offering memorandum, complete EHR summary, full insurance policy)
  YES → Consider Mamba-based backend; chunk + average with Transformer will lose coherence
  NO  → Standard Transformer backend sufficient

Question: Is the answer in a specific known document with logical section structure?
  (examples: SEC filing, clinical guideline, fare manual, mortgage agreement)
  YES → Consider reasoning-based retrieval (PageIndex) instead of or alongside vector search
        — eliminates chunking artifacts, follows cross-references, provides audit trail
  NO  → Vector/hybrid retrieval is the right mechanism

Question: Is accuracy life-critical or regulatory-binding?
  YES → L3 minimum; L4 with human-in-the-loop for final decision
  NO  → L1/L2 with appropriate hedging

Decision Matrix by Domain and Use Case

This post is the map. The rest of the series is the territory.