I ran a scheduled agent every weekday for a month. It logged zero errors. It produced eleven finished outputs. When I finally sat down and checked those eleven against the real requirement, not one of them passed. Zero errors is not the same as zero failures, and the gap between those two numbers is the reliability problem no demo will ever show you.
This was a tool I built for my own use, not a client system, which is the only reason I can tell the story with the numbers attached. It ran on a timer, every weekday morning, no one watching. Each run it scanned a stream of incoming items, scored each one for fit, and for anything above a threshold it assembled a complete, ready-to-use package. Clean, boring, reliable. Or that is what the logs said.
What actually breaks on the hundredth run?
On the hundredth run, nothing crashes. That is the problem. The code still executes, the job still exits clean, the log still turns green. What drifts is the quality of the judgment inside each run, and judgment has no exit code. A demo is run one. You watch it, it works, you sign off. Runs two through ten look identical, so you stop watching. By run fifty the habit of looking is gone. By run one hundred the agent is making the same confident mistake every single day, and the only instrument that could tell you is a person reading the output, which is exactly the step you cut because it felt redundant.
My agent qualified items it should have rejected on the first line of the requirement. Hard disqualifiers it was explicitly told to screen for. It marked them as passes, assembled the full package, and filed it as done. Every day. The score it printed looked plausible. The package looked complete. Nothing in the run said “this is wrong,” because from the code’s point of view, it wasn’t. It had done exactly what it was told. The instructions were the bug, and no error can catch a bug in the instructions.
Why don’t the demos show this?
Demos only ever show the happy path, on the day you are paying attention. The input is clean, the scope is small, and you are standing right there to catch anything odd. That is the worst possible condition for predicting reliability, because every factor that breaks a system over time is absent. I made this exact point about handoffs in the integration that looked done in the demo and broke in week two, and the hundredth-run version is just that failure stretched across a calendar. Week two tests the edge case. Run one hundred tests whether anyone is still looking.
Why does a clean error log lull you to sleep?
A clean error log lulls you because it answers a question you were not really asking. An error log tells you whether the program finished. It says nothing about whether the result was right. Google’s SRE team built their whole monitoring discipline around this gap: in their chapter on monitoring distributed systems, errors are only one of four golden signals, and even all four together measure the health of the service, not the correctness of what it produced. For a deterministic service that distinction is subtle. For an agent whose entire job is judgment, it is the whole game. My four signals were perfect. My output was garbage.
I had been here before with a non-AI job. I once shipped an automation that ran clean for two full weeks while producing wrong output the entire time, which I wrote up in the post on the automation that failed silently. I thought the lesson there was about one specific script. It was not. It was about every unwatched schedule I would ever build. The agent version is more dangerous, because the agent is fluent. It does not just exit clean. It explains its wrong answer confidently, in full sentences, and the confidence is indistinguishable from the real thing.
What was my agent actually missing?
It was missing ground truth. The agent scored every item on a numeric scale and gated on a threshold, and I never once checked that scale against a single real outcome. There was no feedback signal wired back in: nothing told the agent, or me, whether a score of four actually meant a good result. So the number was theater. It looked like measurement. It was the model’s opinion of its own opinion, printed to two decimal places. I argued this general case in why I stopped trusting eval scores and started reading transcripts, and then I went and built a system that trusted its own score with no transcript in sight.
The fix is not a better score. It is a held-out set of cases where I already know the right answer, run through the agent on a cadence, so drift shows up against something real instead of against the model’s own confidence. That is the difference between measuring activity and measuring whether the thing works, which is the exact mistake I spent my first six months making and wrote up in what I measure now that I ignored early on.
Wasn’t it supposed to improve itself?
I had given the agent a weekly self-review pass, and the pass graded the wrong thing, which made it worse than no review at all. Every week it checked its own funnel: items in, outputs produced, schedule healthy. Every week it concluded the same thing. Supply is fine, eleven packages produced, no changes needed. It was auditing whether it had been busy, not whether it had been right. The model that made the mistakes was grading its own homework, and it gave itself an A every time. A self-check that measures throughput instead of correctness is not a safety net. It is a second voice telling you the broken thing is fine.
What do I run differently now?
Now every scheduled agent I build carries a correctness check before I let it run unwatched, and the check never lives inside the agent. That means three things wired in from the start: a small held-out set of known-answer cases run every so often so drift fails loud, an alert that fires on output drift rather than only on errors, and a human spot-check on a cadence I do not get to skip. Anthropic makes the restrained version of this argument in their post on building effective agents: add autonomy only where you can still verify the result. The verification is the feature. The autonomy is just what you are allowed once verification exists.
This is the same boundary I defend when a client asks me to pull the last human review, which I covered in the one request I turn down every time. A month of clean runs is the most convincing argument there is for removing the check, and it is exactly the argument you should distrust most. Knowing which runs to trust and which to keep watching is most of what my AI consulting work actually enforces. Not building the agent. Building the thing that catches the agent when it is confidently wrong on run one hundred.
Zero errors across a month felt like success for about as long as it took to open the outputs. It was not success. It was a month of nobody looking. The hundredth run is not where a system proves it works. It is where it proves whether anyone checked.