A hallucinating chatbot is worse than no chatbot. It answers confidently, cites nothing, and quietly erodes the trust you spent months earning. The fix isn't a better prompt or a bigger model — it's an architecture that grounds every answer in retrieved sources, shows its work, and refuses when it doesn't actually know.

This is a hands-on guide to building that pipeline. It's deliberately framework-light: plain Python, a handful of functions, and code you can paste into a notebook and run. By the end you'll have retrieval-augmented generation (RAG) that cites its sources with [1], [2] markers and says "I don't know" instead of inventing an answer.

We'll keep the model calls behind a thin interface on purpose — as we argued in The Model Isn't the Moat Anymore, you want to swap models freely without touching the pipeline.

What we're building

docs ──▶ chunk ──▶ embed ──▶ store
                                │
query ──▶ embed ──▶ retrieve top-k ──▶ rerank ──▶ generate (grounded + cited)
                                │                        │
                                └── low relevance? ──────┴──▶ refuse

Five moving parts: chunk, embed/store, retrieve, generate with citations, and a refusal gate. Plus a tiny eval at the end so you can prove it works.

Step 1 — Chunk your documents

Retrieval quality is decided here. Chunks that are too big drown the answer in noise; too small and you lose context. Overlapping windows of a few hundred words are a solid default.

def chunk(text, size=800, overlap=150):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]

Split on document structure (headings, paragraphs) when you have it — semantic boundaries beat arbitrary word counts. But the naive version above is a fine starting point.

Step 2 — Embed and store

Turn each chunk into a vector once, up front. For a real corpus you'd use a vector database; to learn the mechanics, an in-memory NumPy array is plenty.

import numpy as np
from openai import OpenAI

client = OpenAI()

def embed(texts):
    resp = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return [d.embedding for d in resp.data]

chunks = chunk(open("handbook.txt").read())
vectors = np.array(embed(chunks))   # shape: (n_chunks, dim)

Step 3 — Retrieve the top matches

Embed the user's query the same way, then rank chunks by cosine similarity. The scores matter later — they're what powers the refusal gate.

def retrieve(query, k=5):
    q = np.array(embed([query])[0])
    sims = vectors @ q / (np.linalg.norm(vectors, axis=1) * np.linalg.norm(q))
    top = sims.argsort()[::-1][:k]
    return [(chunks[i], float(sims[i])) for i in top]

Step 4 — Rerank (the cheap accuracy win teams skip)

Embedding similarity is fast but blunt. A cross-encoder reranker re-scores the top-k by reading the query and chunk together, and it routinely rescues the one relevant passage that ranked 4th into the top slot. It's the highest-leverage 10 lines in the whole pipeline.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query, hits):
    scored = reranker.predict([(query, c) for c, _ in hits])
    ranked = sorted(zip(hits, scored), key=lambda x: x[1], reverse=True)
    return [(c, float(s)) for (c, _), s in ranked]

Step 5 — Generate a grounded, cited answer

This is where hallucinations are won or lost, and it comes down to the system prompt plus the shape of the context. Number the sources, force citations, and forbid outside knowledge.

SYSTEM = (
    "You answer strictly from the numbered sources provided.\n"
    "- Cite the source number in brackets after each claim, e.g. [2].\n"
    "- If the sources do not contain the answer, reply exactly:\n"
    '  "I don\'t know based on the sources I have."\n'
    "- Never use outside knowledge. Never guess."
)

def answer(query, k=5, floor=0.35):
    hits = rerank(query, retrieve(query, k))
    if not hits or hits[0][1] < floor:            # refusal gate (Step 6)
        return "I don't know based on the sources I have.", []
    context = "\n\n".join(f"[{i+1}] {c}" for i, (c, _) in enumerate(hits))
    resp = client.chat.completions.create(
        model="gpt-5.6-mini",                     # swap freely; the pipeline doesn't care
        temperature=0,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"Sources:\n{context}\n\nQuestion: {query}"},
        ],
    )
    return resp.choices[0].message.content, hits

Two details do the heavy lifting: temperature=0 (you want faithfulness, not creativity) and numbered sources in the context so the model has something concrete to cite.

Step 6 — Refuse when you don't know

The refusal gate above is the difference between a tool people trust and one they learn to ignore. It fires on two signals:

  1. Retrieval confidence — if the best reranked score is below a floor, nothing relevant was found, so don't even call the model. Tune the floor on real queries.
  2. Prompt-level refusal — even when chunks pass the floor, they may not contain the answer. The system prompt gives the model an explicit, exact way to say so.

"I don't know" is a feature. A RAG system that occasionally refuses is far more useful than one that always answers and is sometimes confidently wrong.

Step 7 — Prove it with a tiny eval

Don't ship on vibes. A dozen labelled questions — some answerable from your docs, some deliberately not — catch regressions the moment you change a model, a prompt, or a chunk size.

import re

def cited_ids(text):
    return set(map(int, re.findall(r"\[(\d+)\]", text)))

def check(query, should_answer):
    text, hits = answer(query)
    refused = text.startswith("I don't know")
    cites = cited_ids(text)
    valid_cites = bool(cites) and all(1 <= c <= len(hits) for c in cites)
    return (not refused and valid_cites) if should_answer else refused

cases = [
    ("What is the refund window?", True),
    ("Who won the 2019 World Cup?", False),   # not in our docs -> must refuse
]
passed = sum(check(q, a) for q, a in cases)
print(f"{passed}/{len(cases)} passed")

This checks two things that matter: answerable questions get answered and cite real sources, and out-of-scope questions get refused. Expand it to a few dozen cases and it becomes your regression suite for every future change.

Where to go from here

For production you'll swap the NumPy array for a real vector store (pgvector, Qdrant, Weaviate), add caching, and log every retrieval so you can debug bad answers after the fact. But the shape doesn't change: retrieve, ground, cite, and refuse. Get those four right and you have a RAG system people actually trust.

That "survives contact with real users" bar is the whole point — it's the same standard we hold every AI feature to. If you're building on RAG and want it done to that level, see what we build or start a conversation.

Tags:RAGTutorialPythonRetrievalLLM