July 2026 · 8 min read

I built a self-correcting RAG pipeline. It scored worse than the baseline.

SHARP-RAG added a critic agent that verifies retrieved evidence before answering. Exact match dropped from 25% to 15% and latency tripled. The post-mortem taught me more than a positive result would have.

Self-correction is one of the most seductive ideas in retrieval-augmented generation. Naive RAG retrieves once and never checks itself: if the retrieved passages don’t actually support an answer, the model answers anyway, confidently, from noise. The obvious fix is to add a verifier — an agent that inspects the evidence, decides whether it’s sufficient, and sends the pipeline back for another retrieval pass when it isn’t. Every diagram of this architecture looks like it should work.

I built that system. It’s called SHARP-RAG, and on my multi-hop QA evaluation it scored worse than the naive baseline it was designed to beat — not marginally worse, but ten points of exact match worse, at three times the latency. This post is the post-mortem: what the architecture was, what the numbers said, and why I now believe the interesting variable in self-correcting RAG isn’t the architecture at all.

The architecture

SHARP-RAG is a four-agent pipeline built as a cyclic, stateful LangGraph graph. A Planner decomposes the question into retrieval steps. A Retriever executes them against a ChromaDB index. A Critic inspects the retrieved evidence and emits a structured JSON verdict: is this context sufficient to answer the question? The verdict gates everything downstream — only when the Critic approves does the Synthesizer generate an answer; on rejection, the graph loops back and the Retriever tries again with a refined query informed by the Critic’s objection.

The design target was multi-hop question answering — questions where the answer requires chaining evidence across documents (“What award did the director of X win in year Y?”). Multi-hop is exactly where single-shot retrieval fails most, so it’s where a verification loop should help most. That was the theory.

The numbers

Two honest caveats before the analysis. First, twenty questions is a small evaluation — the error bars are wide, and I wouldn’t publish these numbers as a benchmark claim. But the direction and the size of the gap were consistent across runs, and a 10-point EM drop with a 3× latency increase is not the kind of result that flips sign with a bigger sample. Second, the planning baseline matching naive RAG is itself informative: decomposing the question didn’t help either. The damage came specifically from the loop.

Why the critic made things worse

The failure mode is calibration. The Critic’s job is a binary classification task — “is this evidence sufficient?” — and like any classifier it has a precision and a recall. I had implicitly assumed it would be roughly as good at judging answers as the Synthesizer is at producing them. It wasn’t. The Critic rejected evidence that was actually sufficient, and each false rejection did compounding damage: the loop burned another retrieval pass, the refined query drifted away from the original question, and the context window filled with passages that were individually plausible and collectively diluting. By the time the Critic finally approved, the Synthesizer was often answering from a worse evidence set than the one rejected on the first pass.

There’s a simple way to think about when this architecture can pay for itself. A verification loop only adds value when the verifier’s judgment is better calibrated than the generator is accurate. If your baseline answers 25% of questions correctly, the critic needs to catch a meaningful share of the 75% failures (recall) while almost never vetoing the 25% successes (precision) — otherwise every intervention is more likely to break a correct trajectory than to fix a broken one. My critic was below that threshold, so the loop subtracted. The architecture was fine. The judge was miscalibrated, and the architecture faithfully amplified its miscalibration.

Critique model calibration, not architecture, determines whether self-correction helps or hurts.

That sentence is the paper’s core finding, and I’d argue it generalizes beyond my system. Most of the self-correction literature treats the critic as a free win — a component you add, not a model you have to validate. But the critic is a model making predictions, and nobody would deploy a classifier that gates a production pipeline without ever measuring its precision and recall. That’s effectively what wiring an unvalidated LLM judge into a retrieval loop does.

What I’d do differently

• Measure the critic in isolation first. Before wiring the loop, label a set of (question, evidence) pairs as sufficient/insufficient and score the critic against them. If its precision on “reject” verdicts doesn’t clear the baseline error rate, the loop cannot help.
• Make intervention asymmetric. Only trigger re-retrieval on high-confidence rejections, and cap the loop at one corrective pass. An intervention that fires rarely and confidently beats one that fires often and noisily.
• Keep the original context. My refined queries replaced evidence instead of augmenting it, which is how correct first-pass retrievals got destroyed. Re-retrieval should be additive, with the Synthesizer seeing both passes.
• Evaluate on enough questions that the calibration measurement itself is trustworthy — the twenty-question evaluation was the biggest single weakness of the study.

Why publish a negative result

It would have been easy to tune the system until it eked out a point of EM over the baseline and write that paper instead. But the negative result is the more useful artifact: it names the variable that actually governs whether a whole class of popular architectures works, and it’s reproducible by anyone who wires an unvalidated critic into a loop. I learned more engineering from these twenty questions failing than from any project that worked on the first try — and I’d rather show you how I reason about failure than curate a highlight reel.

The full write-up, including the pipeline implementation, is on Zenodo and GitHub — links in the Research section of this site.

Elia Ghazal · Get in touch