When I started my AI Engineering project, I made one decision early that shaped everything afterward: I wasn’t going to build a prototype. Plenty of RAG demos can answer a question if you squint and don’t ask too hard. I wanted something that behaved like a system you could actually put in front of employees. One that refuses to make things up, cites where its answers come from, and can prove it works with numbers rather than vibes.
The result is a policy Q&A application for a fictional company, Northwind Technologies. Here’s how I built it, what broke along the way, and what I learned about the gap between “it works in the notebook” and “it’s ready to ship.”
The problem
The premise is simple: an employee asks something like “How many PTO days do I get in my second year?” or “What’s the expense limit for client dinners?”, and the system answers from the company’s actual policy documents — with citations — or tells the user honestly when the answer isn’t in the corpus. That last part matters more than it sounds. A system that confidently invents a PTO policy is worse than no system at all.
The corpus
I generated fifteen synthetic policy documents for Northwind, deliberately spread across four formats: markdown, HTML, PDF, and plain text. The format diversity wasn’t busywork. Real document stores are messy, and an ingestion pipeline that only handles clean markdown is a pipeline that will fail the moment someone uploads a PDF. Building against four formats from day one forced the parsing layer to be honest about that complexity.
Alongside the corpus, I designed a 27-question evaluation set. Each question carries gold document IDs, must-contain substrings, a difficulty rating, and — importantly — a refusal flag for questions that should not be answerable from the corpus. You can’t measure whether a system knows its limits unless you deliberately ask it things it shouldn’t know.
The pipeline
The retrieval and generation stack came together around a few deliberate choices:
- Embeddings:
BAAI/bge-small-en-v1.5, running locally. Small, fast, free, and good enough for a focused policy corpus. No API dependency for the most-called part of the system. - Vector store: Chroma. Lightweight, runs locally, no infrastructure to babysit.
- Retrieval: Maximal Marginal Relevance with k=5. Plain top-k retrieval tends to return five near-duplicate chunks; MMR trades a little raw relevance for diversity, which matters when the answer lives across more than one passage.
- Generation: Groq’s
llama-3.3-70b, chosen for speed and a generous free tier.
A pattern I kept coming back to: build the RAG pipeline before the web app. It’s tempting to scaffold a nice Flask UI early because it feels like progress, but a UI sitting on top of broken retrieval just hides the bugs that matter. Get the core right, then wrap it.
Guardrails, in three layers
This is where “production-grade” stopped being a slogan and started being work. I built three layers of defense:
- Input validation — catch malformed or out-of-scope queries before they ever hit the model.
- Output validation — a hallucination check that looks for the presence of citations and applies a heuristic over digits, dollar amounts, and percentages. If the model states “$75 per meal” but nothing in the retrieved context supports a dollar figure, that’s a red flag.
- Citation filtering — strip citations that don’t actually point at retrieved sources, so the system can’t dress up a guess as a sourced fact.
The interesting bugs all lived here. An early version of the output check used a length-only heuristic to decide whether an answer was “substantive” enough to validate — and it sailed right past short, fact-bearing claims like “The CEO is John Smith, hired in 2020.” Eleven words, two hard facts, zero scrutiny. I rewrote it to look at what the claim asserts rather than how long it is.
Worse was a validation-ordering bug that let answers generated with no retrieved context slip through the guardrails entirely — the exact failure mode the whole system exists to prevent. It’s a humbling reminder that the order you run your checks in is itself a correctness property, not just an implementation detail.
Evaluation: the part that makes it defensible
A RAG system without evaluation is just a vibe. I built a harness measuring five families of metrics:
- Groundedness, via an LLM-as-judge that scores whether each answer is fully supported by its retrieved context.
- Citation precision, recall, and F1 — are the cited sources the right ones, and did we miss any?
- Partial-match against gold answers.
- Refusal rate — does it correctly decline the questions it should?
- Latency percentiles — p50 and p95, because average latency lies.
On top of that, the harness runs a seven-variant ablation: swap the retrieval k, the chunk size, the prompt format, and see what actually moves the numbers. This is the difference between “I chose k=5” and “I chose k=5 because here’s what k=3 and k=8 did to groundedness and latency.” The rubric I was working against rewards the second kind of answer, and so does anyone who has to maintain the system after me.
What I’d tell my past self
A few things crystallized over the build:
Defer the documentation until the code works. I wrote it early once and then rewrote it three times as the design shifted. Now I treat docs as something you write about a verified system, not toward an imagined one.
Decouple your types from your framework. One of my core data structures was tied directly to LangChain, which quietly made the whole pipeline impossible to unit-test in isolation. Pulling it out into its own module fixed the testing problem and, as a bonus, made the dependency on LangChain something I could reason about instead of something baked into everything.
Trust standalone verification scripts. A small, dumb script that just checks “is everything where it’s supposed to be?” caught a manifest file I’d misplaced inside the corpus directory — exactly the kind of error that’s invisible in code review and catastrophic at ingestion time.
Where it landed
The finished system does what I set out to build: it answers policy questions from a real (if synthetic) document corpus, cites its sources, refuses gracefully when it doesn’t know, and backs all of that with measured groundedness, citation accuracy, and latency. The Flask app exposes a chat interface, a /chat API that returns answers with citations and snippets, and a /health endpoint.
The biggest lesson was that “production-ready” mostly comes down to honesty: The system being honest about what it knows, and me being honest about whether it actually works. Everything else is plumbing.
I will be sharing the repository once the project is evaluated and graded at Quantic.
Leave a Reply