Stopping LLMs from hallucinating: RAG in production

Every team that ships a large language model feature meets the same wall. The prototype dazzles in the demo. Then it reaches real users, and it confidently invents a refund policy that doesn't exist, cites a document that was never written, or answers a question about last quarter using training data from two years ago. The model isn't broken — it's doing exactly what it was built to do: produce the most plausible continuation, not the most true one.

We build and operate ReplyFlow AI, where a wrong answer isn't a quirky screenshot — it's a customer who was misinformed. So the architecture below isn't theory. It's what we run to keep generated text anchored to facts we can point at.

Why models hallucinate

A language model is a probability engine over tokens. It has no internal database it can query and no concept of "I don't know." When you ask it something its weights don't cover, it doesn't fall silent — it fills the gap with the statistically likeliest words. That gap-filling is the same mechanism that writes fluent prose; you can't switch it off without losing the fluency. The fix is not a better prompt that begs the model to "only use facts." The fix is to change what's in the gap — to put the right facts in front of the model at the moment it generates, so the likeliest continuation is also the correct one.

That is what retrieval-augmented generation (RAG) does.

Grounding: retrieval before generation

The core pattern is simple to state. Before the model answers, you retrieve the relevant source material from a trusted store and inject it into the prompt as context. The model is then instructed to answer only from that context. The flow:

• Ingest — your source of truth (docs, policies, tickets, product data) is split into passages and embedded into a vector index.
• Retrieve — the user's question is embedded and used to pull the top-matching passages.
• Augment — those passages are placed into the prompt alongside the question.
• Generate — the model answers from the supplied context, and cites which passage each claim came from.

The discipline is in the last step. We require the model to attribute every factual claim to a retrieved passage, and we render those citations in the UI. If a claim can't be traced to a source, that's a signal — not a sentence to ship.

The goal of RAG isn't to make the model smarter. It's to make the model accountable to a source you control.

Chunking and embeddings that actually retrieve

Most RAG systems fail at retrieval, not generation. If you fetch the wrong passages, even a perfect model gives a perfectly grounded wrong answer. Two decisions dominate quality here.

Chunk on meaning, not character count

The naive approach splits documents every 500 characters. That slices sentences in half and buries the answer across two chunks, neither of which retrieves well. We chunk on structure — headings, list items, logical sections — so each passage is a self-contained unit of meaning. We also attach metadata (source, section, date) to every chunk, which lets us filter retrieval ("only policies effective this year") before similarity even runs.

Retrieve more, then rerank

Pure vector similarity is fuzzy. We retrieve a generous candidate set, then apply a reranking model that scores each candidate against the query directly. Combining keyword search with vector search — hybrid retrieval — catches the cases where the user's exact term matters (a product SKU, an error code) and embeddings alone would miss it.

In our stack this is typically pgvector on Postgres for the index, a hybrid BM25 + vector query, and a cross-encoder reranker on the shortlist. Postgres keeps the vectors next to the relational data we already filter on, which removes an entire moving part.

Constraining the output

A grounded answer is still free-form text, and free-form text is hard to act on. Wherever the output feeds a downstream system — routing a ticket, populating a form, calling a tool — we constrain generation to a schema. The model returns structured JSON validated against a contract, so a malformed or hallucinated field is rejected before it ever reaches the application.

• Schema-constrained decoding for anything machine-consumed.
• Refusal paths — an explicit "insufficient context" response when retrieval comes back empty, instead of a guess.
• Tool calls over free text when the model needs live data, so the fact comes from the system, not the weights.

Evaluation: the part teams skip

Here's where most RAG projects quietly fall apart. The team ships, eyeballs a few answers, declares victory — and has no idea when a model update, a new document, or a prompt tweak silently degrades quality. You cannot operate what you cannot measure.

We treat the AI system like any other system: it has tests.

• A golden set of real questions with verified answers, run on every change.
• Faithfulness scoring — does each answer stay within its retrieved context? An LLM-as-judge, checked against human labels, flags claims that drift.
• Retrieval metrics — did the right passage make the top results at all? If not, no amount of prompt work helps.
• Regression gates in CI, so a change that drops faithfulness doesn't merge.

Operating it

A RAG system is a living thing. Documents change, users ask questions you never anticipated, and the underlying model gets updated beneath you. We log every query, its retrieved context, and the answer, so we can replay failures and grow the golden set from real misses. We pin model versions and re-run evaluation before adopting a new one. And we keep a human in the loop wherever the cost of being wrong is high — the model drafts, a person confirms.

Done this way, an LLM stops being a party trick and becomes infrastructure: accurate enough to trust, observable enough to operate, and honest enough to say "I don't know."