This is Part 4 of a 5-part series documenting my MSSE Capstone project at Quantic: a deployed, agentic RAG system that answers natural-language questions about European electricity markets. Part 3 covered building a trustworthy corpus — this one covers turning the system into something that can go beyond it, and the bugs I had to out-think along the way.
Retrieval isn’t enough when the question is “right now”
By the end of Part 3, the system could search a corpus and answer questions grounded in it well. But a corpus is always slightly behind the present, and some questions genuinely need a live number — current load, current wind and solar forecast, the actual price right now — not the closest thing retrieval can find in yesterday’s documents. That gap is what Sprint 3 was for: giving the system a way to reach past its own static corpus mid-conversation, the way a human analyst pulls up a live dashboard instead of relying only on what’s already written up.
The architecture: a tool loop, not a framework’s agent
I built this with LangChain — ChatGroq and ChatGoogleGenerativeAI, bind_tools(), .with_fallbacks() — but deliberately skipped AgentExecutor and LangGraph in favor of a hand-wired tool-call loop. That was a real choice, not a shortcut: for a single tool with a well-defined contract, a framework’s agent executor adds a layer of abstraction and failure modes I’d have to debug through, for a problem that a straightforward loop solves directly. I’d rather own the fifteen lines that decide “call the tool, read the result, decide whether to call again” than debug someone else’s agent loop when it does something surprising at 2am.
The one tool that matters here is get_entsoe_numeric. It calls the ENTSO-E client directly for price, generation, load, and wind/solar forecast data — and deliberately bypasses the frozen ingestion path (CORPUS_FROZEN) entirely. That split is intentional: the RAG corpus stays a frozen, reproducible snapshot so evaluation results mean the same thing every time I run them, while the live tool reaches past that snapshot for anything that needs to be current. Groq (openai/gpt-oss-120b) is the primary model, with Gemini (gemini-2.5-flash) as a fallback — both pinned to temperature=0, because tool selection needs to be deterministic, not creative.
The bugs, in the order they taught me something
None of the above worked on the first try. Here’s what actually broke, roughly in the order I found it.
A retry storm that looked like a slow tool. Early testing showed the live-data tool path taking 53 to 79 seconds — unusable. The cause was tenacity retrying on NoMatchingDataError, a case where retrying can never help, because the data genuinely doesn’t exist for that query. The fix was retry_if_not_exception_type, telling the retry logic to give up immediately on errors retrying can’t fix. That dropped the path to about 8.5 seconds. The lesson: a slow system isn’t always a system doing too little work — sometimes it’s doing the wrong work, repeatedly, with confidence.
A model deprecation I found the hard way. llama-3.3-70b-versatile got deprecated out from under the project, which meant re-pinning to openai/gpt-oss-120b — a reminder that “primary model” is a moving target on a hosted API, not a one-time decision.
A guardrail that couldn’t see the zone the user picked. This is the bug I mentioned at the end of Part 3. passes_agent_relevance_gate never actually received the resolved zone parameter — so when a user picked a zone from the dropdown, the relevance gate evaluating whether a document was worth using stayed blind to that choice. It didn’t crash anything; it just quietly made zone-scoped answers worse in a way that was easy to miss in casual testing and only showed up clearly once the eval harness caught the drop in refusal_correct. Fixing it took refusal_correct from 0.94 to 0.97. This is the case for having an objective eval set: a bug like this doesn’t announce itself, it just erodes a number you’re not staring at every day.
Two guardrail regressions in the other direction. Later, a zone-bypass rule meant to loosen the gate for a specific case ended up too broad — and then, once narrowed, got applied to the wrong band entirely. Both were caught the same way: run the eval, see the number move, go find out why. Neither would have been obvious from spot-checking answers by hand.
A crash that hid behind a stale reload. At one point agent.py simply stopped importing, thanks to a stray python literal left over at module level from a paste artifact. What made this one sting was that a stale uvicorn --reload process kept serving the old, working version underneath me for a while, making it look like the code was fine when it wasn’t. Lesson: when a fix doesn’t seem to be landing, check whether you’re actually looking at the fix.
Citations that vanished for no visible reason. The citation-matching regex only recognized ASCII brackets, so on the occasions the model emitted full-width 【】 brackets instead, the citation extraction silently found nothing. An easy fix once spotted, and a reminder that LLM output formatting is a moving target you have to defend against, not assume.
The one that isn’t a bug at all: quota exhaustion pretending to be a regression. Groq’s daily token quota is 200K. Hit it, and the system silently falls over to the Gemini fallback — which does retrieval but not tool-calling. The result looks exactly like a regression: tool_selection and plausibility_pass both collapse to 0.00, latency shifts. The first time I saw this, I went looking for a code bug that didn’t exist. Now it’s a pattern I check for first, before touching any code, whenever those two numbers move together.
Why the eval harness is the real hero of this part
Almost every bug above has the same shape: invisible in a quick manual test, visible the moment an objective, repeatable eval run measures it. That’s not an accident — it’s the reason the eval harness exists. A tool-calling agent has more ways to quietly misbehave than a plain RAG pipeline does, and “it seemed to work when I tried it” stops being good enough the moment the system can choose, on its own, whether to reach for a live number or not.
What’s ahead
Last part: getting this out of my dev environment and onto a real URL, what the evaluation numbers actually say once everything above was fixed, and the very last complication — adding a fourth bidding zone late enough to expose every place I’d quietly assumed there would only ever be three.
Leave a Reply