Beyond Basic RAG: Hybrid Search, Reranking and Agentic Retrieval

Retrieval-Augmented Generation (RAG) is how most business AI features stay grounded in company knowledge: find relevant passages, give them to the model, and ask it to answer with citations. The first version is easy. Making it reliable on messy, real documents is where most projects stall. Here are the upgrades, in the order we usually apply them.

Why naive RAG disappoints

The classic pipeline — split documents every 500 tokens, embed, retrieve the top five by cosine similarity — fails in predictable ways:

• Exact terms get lost. Product codes, error messages and names often embed poorly; semantic search returns something _similar_ instead of the exact match.
• Chunks lose context. A paragraph that says "this limit is 30 days" is useless if the chunk doesn't say _which_ limit.
• Questions don't look like answers. "Can I return a sale item?" may not be semantically close to the policy text that answers it.
• Multi-part questions need several retrievals, not one.

1. Chunk by structure, keep context

Split on the document's own structure — headings, sections, list items, table rows — rather than fixed token counts. Then attach context to every chunk:

• The document title and section path ("Returns Policy › Sale Items")
• Metadata for filtering: product, region, version, date, access level
• Optionally a one-sentence, model-generated summary of where the chunk sits in the document (_contextual retrieval_)

chunk_text = f"{doc.title} > {' > '.join(section_path)}\n\n{body}"

This small change often improves retrieval more than switching embedding models.

2. Hybrid search: vectors plus keywords

Combine semantic search (embeddings) with lexical search (BM25 / full-text) and merge the results. Semantic search finds paraphrases; keyword search nails exact identifiers. Reciprocal Rank Fusion (RRF) is a simple, robust way to merge the two ranked lists.

You don't need a separate search cluster to start. PostgreSQL with pgvector and built-in full-text search handles both in one query:

WITH semantic AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS rank
  FROM chunks WHERE tenant_id = $3
  ORDER BY embedding <=> $1 LIMIT 40
), keyword AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank_cd(tsv, query) DESC) AS rank
  FROM chunks, websearch_to_tsquery('english', $2) query
  WHERE tenant_id = $3 AND tsv @@ query
  ORDER BY rank LIMIT 40
)
SELECT id, SUM(1.0 / (60 + rank)) AS score
FROM (SELECT * FROM semantic UNION ALL SELECT * FROM keyword) r
GROUP BY id ORDER BY score DESC LIMIT 20;

Note the tenant_id filter: enforce access control in retrieval, never by asking the model to ignore documents.

3. Rerank the candidates

Retrieve generously (20–50 candidates), then use a reranker — a cross-encoder model that scores each query–passage pair together — to pick the best handful. Rerankers are slower per item than embeddings but far more precise, and they work on a small candidate set, so latency stays reasonable. This is usually the single biggest quality jump after hybrid search.

4. Rewrite the query

User questions are often vague, conversational or depend on earlier messages ("what about for Dubai?"). Before retrieving:

• Condense the conversation into a standalone question.
• Expand with synonyms or likely terminology.
• Decompose multi-part questions into sub-queries and retrieve for each.
• Extract filters — dates, products, regions — into structured metadata filters.

A fast, inexpensive model handles this step well.

5. Agentic retrieval: let the model search iteratively

For complex questions, give the model search as a tool and let it decide what to look up, read results, and search again if needed:

1. Plan: "I need the return window and the sale-item exception."
2. Search for each; read results.
3. Notice a reference to "regional terms" and search for the Dubai policy.
4. Answer with citations, or say what couldn't be found.

Agentic retrieval costs more and takes longer, so route to it only when a quick classifier decides a question is complex. Simple lookups stay on the fast single-pass pipeline.

6. Answer with citations — and permission to say "I don't know"

Instruct the model to answer only from the provided passages, cite them, and state clearly when the answer isn't present. Show the citations in the UI so users can verify. A system that admits gaps earns far more trust than one that confidently improvises.

7. Evaluate retrieval and answers separately

When an answer is wrong, you need to know whether retrieval failed or generation did. Build a test set of questions with known source passages and measure:

Run the suite whenever you change chunking, models, prompts or ranking — and add every real-world failure as a new test.

8. Keep the index fresh

Stale knowledge is a silent failure. Re-index on change (webhooks or change feeds from your CMS, drive or database), store a content hash per chunk to skip unchanged text, and record document versions so answers can say _as of_ when.

Putting it together

A production RAG pipeline in 2026 typically looks like: structured chunking with context → hybrid retrieval with access filters → reranking → grounded generation with citations, with query rewriting in front and agentic retrieval for hard questions. None of the steps is exotic. Doing all of them, and measuring each, is what makes the difference.

_Building search or an assistant over your company knowledge? Talk to us about AI software & automation._