Start free
Document · rag-and-vector-databases
GuidesNov 20, 2026 · 8 min readBy

Part 3: RAG and Vector Databases

Part 3: RAG and Vector Databases

A support bot needs to answer a question about your help center. Its model may never have seen those pages, and a confident answer from memory can still be wrong. Give it the relevant passages before it writes.

That lookup is , or RAG: retrieve relevant documents at question time and add them to the . You can inspect the passages separately from the answer, which gives you a useful place to start when something goes wrong.

Fast review (60 seconds)

  • RAG adds evidence from your documents to the prompt at question time.
  • Ingestion splits documents into chunks, embeds them, and stores their text and metadata alongside the vectors.
  • Querying retrieves nearby vectors, optionally reranks the results, and asks the model to answer from that with .
  • Chunk size changes what the search can find. Large chunks can bury an answer; small ones can lose its context.
  • combines BM25 keyword matches with search by meaning.
  • Evaluate retrieval with recall@k and ranking metrics, then evaluate the answer for faithfulness and relevance.

Key terms

  • RAG: Retrieve relevant documents at query time and add them to the prompt.
  • : Splitting documents into passages sized for retrieval; overlap keeps context across cuts.
  • Embedding: A vector of numbers representing meaning; similar text gets nearby vectors.
  • : A store that indexes embeddings for fast nearest-neighbor search.
  • Cosine similarity: How closely two vectors point the same way; the usual similarity score.
  • ANN / HNSW: Approximate nearest neighbor search; HNSW is the common graph-based index.
  • BM25: A classic keyword ranking function based on term frequency.
  • Hybrid search: Combining vector and keyword results, often with reciprocal rank fusion (RRF).
  • Reranker: A cross-encoder that rescores query and passage pairs for precision.
  • Metadata filtering: Restricting search by fields like client, date or permissions.
  • recall@k / precision@k: Share of relevant chunks found in the / share of the top k that are relevant.
  • MRR / nDCG: Ranking metrics that reward putting the right result near the top.
  • Faithfulness / groundedness: Whether every claim in the answer is supported by the retrieved context.

Why look up the facts first

Suppose a help-center article contains the answer a customer needs. Your system first finds that passage. It then includes the passage in the prompt and asks the model to answer only from the supplied context, with a citation.

This gives the model evidence it may not have learned during . It also makes the evidence available for you to check. RAG can still produce unsupported claims, so the retrieval step doesn't remove the need to evaluate the answer.

A student answering an exam question by finding the right page in an open book, like RAG looking up facts before answering.
Find the passage first, then write an answer you can trace back to it.

The open-book exam analogy helps here. A student finds the right pages, writes an answer from them, and cites them. A vector database is like an index organized by meaning rather than exact words. The mechanism is still a search followed by a model call.

Ingestion: chunk, embed, store

Ingestion prepares your documents ahead of time. Split each document into passages, turn each passage into an embedding, and store it with the original text. Keep metadata too, such as its source, date, client, and permissions.

An embedding is a list of numbers that represents meaning. Text with similar meaning gets nearby vectors, so you can retrieve passages even when a question uses different words.

Documents cut into cards that become dots in a 3D space, where similar text clusters together.
Chunking sets the unit of retrieval; embeddings let you search those chunks by meaning.
  1. Chunk
  2. Embed
  3. Similar meaning = nearby points

Choose chunk with the source material in mind. A large chunk may include the answer among unrelated text. A small chunk may cut away the heading or explanation that makes the answer understandable. Common approaches are fixed-size chunks with overlap, splits at headings or paragraphs, and semantic chunking, which splits around changes in meaning.

Overlap repeats some text across a boundary so the cut doesn't lose all surrounding context. The code below uses size 800 and overlap 100. Those are settings, not a measured best choice for every document.

The store indexes these embeddings for nearest-neighbor search. Cosine similarity compares how closely vectors point in the same direction. Approximate nearest neighbor search, often using the graph-based HNSW index, finds nearby vectors without comparing the question against every stored vector.

Querying: search, rerank, answer

When a question arrives, embed it too. Retrieve passages with vectors close to the question vector, then use the best passages as context for the answer.

Vector search can connect "car" with "automobile". An exact part number is a different problem. BM25 is a keyword ranking function based on term frequency, so it helps with exact terms that meaning-based search can miss.

Hybrid search runs keyword and vector search and merges their results. Reciprocal rank fusion (RRF) combines result lists using their ranks. A reranker then rescores question-and-passage pairs, often with a cross-encoder model that considers each pair together, to improve precision.

Keyword search finds exact matches, vector search finds related meaning, and hybrid search merges both into one ranked list.
Use exact words and meaning together, then rerank the candidates.
  1. BM25 (exact words)
  2. Vector (meaning)
  3. Hybrid + rerank

Apply metadata filtering to restrict the search to the right client, date range, or permissions. A passage can be relevant to the question and still be outside the reader's access.

Documents pass through chunking and embedding into a vector database; a question drives hybrid search, reranking, and an answer with citations.
Prepare documents during ingestion, then retrieve evidence for each question.

The diagram retrieves a top 20 candidate set and reranks it to a top 5. Keep those two stages separate when debugging: a reranker can reorder a candidate it receives, but it cannot recover a passage that retrieval never returned.

Choose a vector store for the constraints you have

Store What it is Pick it when
pgvector Open-source Postgres extension You already run Postgres and want vectors next to relational data and SQL filters.
Pinecone Fully managed vector database You want no infrastructure to run and easy scaling.
Weaviate Open-source vector DB, self-hosted or cloud You want built-in hybrid search and modules.
Chroma Open-source, developer-friendly store You need a simple Python API for prototypes and local development.

Start with the constraints you can explain. If your data already lives in Postgres, keeping vectors next to relational data may simplify the system. A managed store makes a different tradeoff by taking infrastructure work off your team.

Example: ingest, retrieve, rerank, ground

This sketch uses pgvector in Postgres. The application supplies the chunking, embedding, database, and reranking helpers. It demonstrates vector retrieval; the hybrid search in the diagram also needs a BM25 search and a merge step.

# 1) Ingest: chunk, embed, store (pgvector in Postgres)
chunks = split_into_chunks(doc_text, size=800, overlap=100)
vectors = embed(chunks)                        # one vector per chunk
for text, vec in zip(chunks, vectors):
    db.execute("INSERT INTO chunks (doc_id, text, embedding) VALUES (%s, %s, %s)",
               (doc_id, text, vec))

# 2) Query: embed the question, find nearest chunks by cosine distance
q = embed([question])[0]
rows = db.execute("""
    SELECT text FROM chunks
    ORDER BY embedding <=> %s::vector         -- <=> is pgvector cosine distance
    LIMIT 20""", (q,)).fetchall()

# 3) Rerank 20 candidates, keep the best 5, ground the prompt
top = rerank(question, [r[0] for r in rows])[:5]
prompt = ("Answer only from the context. If it is not there, say so.\n\n"
          + "\n---\n".join(top) + "\n\nQuestion: " + question)

The <=> operator orders by cosine distance. The prompt tells the model to say when the context lacks an answer. Check that behavior in your evaluation set rather than assuming the instruction always works.

Measure retrieval and generation separately

Build a small labeled set of questions with the passages each question should retrieve. For retrieval, ask whether the right passages came back and where they ranked. Use recall@k, precision@k, MRR, or nDCG.

Recall@k is:

(relevant chunks found in the top k) / (all relevant chunks for the question)

If 3 relevant chunks exist and 2 appear in the top 5, recall@5 is 0.67. Precision@k asks how much of that top k is relevant. MRR and nDCG reward useful results near the top of the ranking.

Then inspect generation. Faithfulness, also called groundedness, asks whether the retrieved context supports every claim. Answer relevance asks whether the answer addresses the question. Ragas includes metrics for faithfulness, answer relevancy, and context precision.

Fix the stage that failed. Missing passages call for changes to retrieval. Unsupported claims in an answer with good evidence call for attention to generation and its grounding instructions.

Real-world applications

  • Perplexity answers questions from web search results and cites its sources.
  • Microsoft 365 Copilot grounds answers in a company's email, files, and chats.
  • Glean and Notion AI answer questions over internal company documents.
  • Support bots answer from a help center and link to the articles they used.

How this shows up on the job

You may need to find fields or clauses in long source documents. Choose the approach from the task. RAG fits searching for passages and answering from them. Structured extraction fits sending the right pages and requesting specific fields.

Build a checked set of examples early. A team extracting fields may measure the percentage extracted correctly; a question-answering system also needs checks on retrieval and answer support. Be ready to explain your embedding model, store, and chunking choice against those examples.

How it shows up in interviews

What is your experience with RAG and vector databases?

Define RAG in one line, name a store you've used, and explain why you chose it. Describe how hybrid search and reranking fit between the question and the final context. Keep claims about your experience factual.

Say this term: recall@k, the share of relevant chunks found in the top k.

Your RAG answers are wrong. How do you debug it?

Split retrieval from generation. Check recall@k on a labeled set first. Inspect chunking, hybrid search, reranking, and metadata filters when passages are missing. Once the evidence is right, check faithfulness and tighten the instruction to answer only from context.

Say this term: faithfulness, whether the context supports the answer's claims.

Official docs

To read the entire series, sign up for Premium membership.

ragvector-databasesretrievalai-agents

More Articles are available.

GuidesReady.