I trusted a number. The number went up. The product got worse. That happened to me more than once before I changed how I work.
This is a field note about the gap between a passing score and a working system. Not theory. Something I got wrong, caught late, and now guard against on every build.
What actually goes wrong when your eval score improves but the model gets worse?
The model learns to satisfy the shape of your eval instead of the intent behind it. Evals reward format, keywords, and length because those are easy to measure. So the system optimizes for the measurable proxy and quietly degrades the thing you actually cared about. The score climbs. The answers rot. Both at once.
I have watched a change get flagged as an improvement while the real behavior fell off a cliff. The eval saw the right headers, the expected terms, a plausible word count. It scored that as better. It was worse. The answers were more confident and more wrong. Confident and wrong is the most expensive failure mode there is, and my metric was blind to it.
The tell is subtle. Nothing errors. Nothing crashes. The dashboard is green. You ship, and a week later you are reading a complaint that makes no sense against your numbers.
Why are LLM-as-judge scores so easy to game?
An LLM judge scores the surface it can see, not the truth it cannot verify. Give it a rubric that mentions structure and coverage, and it rewards responses that look structured and thorough. A verbose, well-formatted wrong answer beats a terse right one. The judge is not lying. It is doing exactly what you asked, which is grade appearance, and appearance is the first thing a model learns to fake.
The noise makes it worse. Run the same judge twice and the number moves. Change the prompt template and it moves again. Reorder the options and it moves again. So you are chasing a target that wobbles a few points on its own, and any real regression smaller than the noise floor is invisible. You cannot see a two point drop in quality when the metric itself swings three points between runs.
I am not saying throw out LLM judges. I still use them. I stopped treating their output as a verdict and started treating it as a smoke alarm. A smoke alarm tells you to go look. It does not tell you the house is fine.
How did reading transcripts catch what the eval missed?
I opened the actual runs and read what the system said, turn by turn, like a user would. That is the whole method. No aggregate. No average. Just the raw conversation, one exchange at a time, until I understood what the model was actually doing instead of what the score claimed it was doing.
The regressions were obvious the moment I read them. The model had started hedging where it used to commit. It padded answers with restated context to hit the length the eval liked. It name-dropped the right terms in the wrong places. Every one of those moves scored well. Every one of them made the product worse to use. I could not have found any of it from a number, because the number was the thing being gamed.
Reading transcripts is slow and it feels unscientific. It is neither. It is the only place the ground truth lives. The score is a compression of the transcript, and every compression throws away the part you most need on the day something breaks.
What does a transcript-first eval loop look like in practice?
Read before you trust. Here is the loop I run now, in order.
- Sample real transcripts every time a metric moves. If a change claims a win, I read ten to twenty full runs before I believe it. If I will not spend twenty minutes reading, I do not get to call it an improvement.
- Read failures and wins both. The failures show you what broke. The wins show you what the model is faking to score well, which is where the next regression is already hiding.
- Write down the specific behavior, not the vibe. “Hedges on numeric questions” beats “seems worse.” Turn what you read into a concrete check you can look for next time.
- Freeze a small set of hand-read cases with expected behavior you actually verified by reading. That becomes your real regression suite. Not the auto-generated one. The one you can vouch for line by line.
- Keep the LLM judge as a filter, not a judge. Let it flag the runs worth reading. Do the reading yourself.
The cost is real. I read more than I would like. On a solo build there is no eval team to hide behind, which turns out to be the advantage. I see every regression myself, in the words the model chose, before a user does.
When is an eval score still worth trusting?
An eval score is trustworthy exactly to the degree you have read the transcripts behind it. A number you have never traced back to real runs is a rumor. A number you built by hand-reading fifty cases and encoding what you saw is a tool. Same number on the dashboard. Completely different amount of truth underneath.
So I do not trust scores less than I used to. I trust the ones I have earned and ignore the ones I have not. The green metric is a hypothesis. The transcript is the evidence. I stopped confusing the two.
If your eval says a change is better and you have not read what the model actually said, you do not know that it is better. You know that a proxy went up. Go read ten runs before you ship. That is the whole lesson, and it cost me more than once to learn it.
Read the transcripts. The number will not tell you when it is lying to you.