I want to tell you about a failure that never announced itself. No crash. No red log. No 3 a.m. alert. Every run said success. The exit code was zero for two straight weeks. And the whole time, the output was wrong.
This is a pattern, not a case study. I am not going to name a client or hand you a fake metric dressed up as a war story. What I am describing is something I have watched happen more than once, across the kind of small automations I build for founders. The details below are one honest, generic illustration of a real failure mode. Treat it as the lesson, not the legend.
How does an automation fail without throwing an error?
An automation fails silently when its exit code stays green but its output goes wrong, usually because something upstream changed and the script kept running anyway. The code did exactly what I told it. The world it was reading from stopped matching what I assumed. No exception fires when reality drifts. Exceptions fire when code breaks. Those are not the same event.
Here is the generic version. I had an automation running that pulled a daily export, reshaped it, and pushed a clean summary somewhere a human would read it. It worked. I watched it work for a while. Then I stopped watching, because that is the entire point of automation. You build the thing so you can look away.
Somewhere in week two of not looking, the upstream export changed. A column got renamed. The date format shifted from one style to another. Nothing dramatic. My script did not error, because it was tolerant in the worst possible way. It found no match, wrote an empty value, and moved on. Green check. Zero exit. Stale, half-blank output landing in the same place, every day, looking exactly as routine as the correct output had.
Why did no error get thrown when the output was clearly broken?
No error fired because I had built the automation to check that it ran, not to check that it was right. Those are two different questions, and most scripts only answer the first one. “Did the process finish” is easy. “Is the thing it produced actually correct” is the one that matters, and it is the one almost nobody wires up.
The forgiving code made it worse. I had wrapped the fragile parts in handling that swallowed problems instead of surfacing them. A missing field became a blank instead of a stop. That felt safe when I wrote it. It was the opposite of safe. A loud failure would have caught this on day one. My quiet failure hid it for fourteen days.
“No errors” is not “still working.” I want that sentence tattooed somewhere I can see it. An automation reports on its own health using the only thing it can measure without help, which is whether it reached the end. It cannot tell you the end was garbage. It has no opinion about the content. You have to give it one.
How did I finally catch the drift?
I caught it by hand, not by system, which is the embarrassing part and the whole lesson. A human downstream mentioned the numbers looked off. I went to check, expecting a one-day blip, and found two weeks of quietly wrong output stacked up behind a wall of green checkmarks.
Think about that gap. If a person had not happened to glance at the result and trust their own gut over my automation, it could have run wrong for a month. Two months. However long until someone noticed or something downstream finally broke hard enough to make noise. The detection was luck. Luck is not a monitoring strategy.
That is the moment the real problem became clear to me. I did not have a slow automation. I did not have a buggy one. I had a blind one. It could not see its own output, so neither could I.
What check makes silent drift fail loud instead?
The fix is a check that inspects the output itself and screams when it looks wrong, separate from the check that the job finished. I now add three cheap layers to anything I leave running unattended: a canary, a reconciliation check, and a scheduled human spot-check. None of them are clever. All of them would have caught this in a day.
The canary is a heartbeat with an opinion. It does not just confirm the run happened. It asserts something about the result. Row count above a floor. No more than a small share of blank fields. The date on the freshest record is actually today. If the assertion fails, the whole run fails, loudly, to a place I cannot ignore. A blank field is now an alarm, not a shrug.
The reconciliation check compares the output against a source of truth it did not generate. Yesterday against today. The count in against the count out. The total here against the total there. Drift shows up as a gap between two numbers that are supposed to match. When they stop matching, something moved, and I want to know before a person downstream does.
The spot-check is me, on a calendar. Once a week I open the actual output and look at it with my own eyes. Not the logs. The thing. It takes a few minutes. It exists because I no longer trust myself to remember that unattended does not mean unwatched. The machine watches for the failures I predicted. I watch for the ones I did not.
What should a founder take from this?
Automations rot, and a green checkmark is not proof of correctness, so anything you leave running needs a check on its output and not just its exit code. The upstream world will change under your automation. A vendor tweaks an export. An API shifts a field. A format quietly updates. Your script will not notice. It was never built to.
If you run automations in your business, ask one question about each of them. If the output went subtly wrong tomorrow, would anything tell you, or would it just keep looking fine? If the honest answer is that it would keep looking fine, you do not have monitoring. You have a countdown.
I build these for a living and I still got caught by this. That is not false modesty. It is the reason I now wire the loud check in before I ship anything, instead of after something embarrasses me. Build the thing that fails loud when the output goes wrong. Then put a recurring reminder on your own calendar to go look anyway. The automation covers the failures you saw coming. You cover the rest.
A passing run tells you the code finished. It tells you nothing about whether the code was right. Build for the second question, because that is the one that costs you.