I built a retrieval-backed assistant that kept lying to me with a straight face. Not gibberish. Clean, fluent answers that were wrong. My first move was the one almost everyone makes. I made the context window bigger.
It got worse. Not better. More coherent, still wrong. That failure taught me the thing I want to write down before I forget it under the next deadline.
Why does making the context window bigger make wrong answers worse, not better?
A bigger context window does not fix a wrong answer when the cause is bad retrieval. It hides it. If the correct passage was never fetched, adding more tokens only gives the model more plausible-but-irrelevant material to write a confident, coherent, wrong answer from. You do not fix retrieval by feeding the model more of the wrong stuff.
Here is what I actually did, in order, because the order is the embarrassing part. Answer was wrong, so I raised the top-k. Still wrong, so I stuffed the whole source document in. Still wrong, so I reached for a model with a longer window and told myself the problem was capacity. Three moves. Every one of them added tokens. Not one of them looked at what came back from the retriever.
What was actually broken?
The correct passage was not in the top results at all. On some queries it was not retrieved at all. No context window fixes a chunk that never gets fetched. The failure lived one layer upstream of where I was working, in the retrieval step, and every fix I tried was aimed at the generation step.
The fix was to stop adding and start measuring. I took a single failing query and looked at exactly what the retriever returned for it. Not the final answer. The raw retrieved set. The passage that held the real answer sat outside the top results. On a couple of queries it did not appear anywhere in what was fetched. The model was doing its job perfectly. It was faithfully summarizing garbage, because garbage was what I handed it.
Once I saw the retrieved set, the whole thing reframed. I had been debugging the wrong layer for a day. The model was never the problem. My embeddings, my chunk boundaries, and my ranking were the problem. The context window was a red herring I kept sprinting toward because it had a knob I could turn.
Doesn’t a longer context window mean the model reads everything I give it?
No. A longer window is not the same as the model using all of it. Models retrieve information best when it sits at the very start or the very end of the context and degrade sharply when the relevant passage is buried in the middle. So even if your correct chunk does get fetched, dumping it into position 40 of a giant stuffed prompt can bury it exactly where the model is weakest.
This is not a hunch. It is documented in the “Lost in the Middle” study on how models use long contexts, which showed performance sagging when the needed information sat in the middle of a long input. Read that finding next to what I was doing and the mistake is obvious. I was stuffing more context in, pushing the one useful chunk deeper into the pile, and then blaming the model for not finding it. I built the middle. Then I complained about it.
How do you actually measure retrieval in isolation?
Measure retrieval separately from generation by taking a failing query, running only the retrieval step, and reading the raw passages that come back before the model touches them. Ask one question: is the passage that contains the correct answer in this set, and how high is it ranked? If the answer is no, no prompt engineering downstream will save you.
Concretely, this is what I do now, and it takes a spreadsheet, not a framework:
- Collect ten to twenty queries the system gets wrong.
- For each one, log the raw retrieved chunks and their rank. Not the final answer. The retrieved set.
- Mark by hand whether the chunk holding the true answer is present, and where.
- Split the failures into two buckets: not retrieved at all, versus retrieved but ranked low or buried mid-context.
Those two buckets have completely different fixes. “Not retrieved at all” is usually a chunking or embedding problem. Your chunks split the answer across a boundary, or your query and your document use different words for the same thing and the embeddings never bridge them. “Retrieved but buried” is a ranking and ordering problem. Add a reranker. Put the highest-scoring chunk at the top or the bottom of what you send, not the middle. Different disease, different medicine. You cannot tell which one you have until you read the retrieved set.
Why do smart builders keep blaming the model instead?
Builders blame the model because the model is where the output is, and the output is what hurts. The retrieved set is invisible unless you go looking for it, so the failure feels like it lives in generation. It almost never does. When people say RAG is broken or that they need a bigger window, they have usually measured the wrong layer and blamed the layer they can see.
This is the same mistake I keep writing about from different angles, which is starting to feel less like coincidence and more like my one recurring bug. It is the same reason I wrote about why I read transcripts instead of trusting eval scores: a green number tells you the aggregate passed, not what actually came back on the case that failed. It is the same reason I wrote about measuring against the goal instead of exit code 0: the process succeeded, the outcome was still wrong. Every time, the summary metric was fine and the reality underneath it was broken. Retrieval is one more layer where the summary lies and the raw trace tells the truth.
I want to be honest about the scope of this claim. I am generalizing from my own builds and from watching the same wrong turn happen in other people’s systems, not from a controlled benchmark. Your retrieval might genuinely need a bigger window for a task that is legitimately long. But that is a conclusion you earn by reading the retrieved set first, not the reflex you reach for because there is a knob to turn.
What I do differently now
I measure retrieval before I touch generation. Every time. When an answer is wrong, the first artifact I open is the retrieved set for that exact query, not the prompt and not the model card. If the right chunk is not in there, I fix chunking and embeddings. If it is in there but buried, I fix ranking and ordering. I only think about the context window last, and usually I do not need to.
The context window is not the bottleneck. It is the most tempting place to look because it has a setting. Your retrieval is the bottleneck, and it does not have a setting, it has a trace you have to read. This is most of the work in the retrieval-backed assistants I build, and it is almost never the part people expect to pay for. They expect a better model. What they need is someone willing to read what the retriever actually returned.
Stop adding context. Start reading what came back. The answer is decided before the model ever sees it.