I'm not writing this one as a case study. No client name, no dollar figure, no logo. Just a bug I almost shipped, three completely different ways of reading the same code, and the rule I now run by default because of what that difference turned up.
What does building a "self-improving" agent framework actually involve?
A self-improving agent framework is a set of agents that read their own outcomes and rewrite their own instructions without me in the loop for every change. That's the pitch. The day-to-day reality is narrower and less glamorous: agents that act on my behalf while I'm asleep, and my actual job becoming making sure they fail loud instead of quiet.
I've spent the last several months building that for my own operation first, then folding what worked into how I run client work. A cron job that scans for opportunities. A weekly job that reads a week of session logs and updates the standing rules I work from. A pipeline that reviews items in a queue, decides approve or reject, and acts on that decision without me checking each one. None of it is exotic. Most of it is bash scripts and a scheduler that either run or they don't.
I run this solo. No SDR, no junior engineer, no second set of eyes by default. That's the whole pitch of the business too, one person doing the actual work, so when I say "self-improving," I mean it has to catch its own mistakes without a team behind it. That constraint is what forced the discipline in this post. If I had five engineers, one of them eventually notices a bad approval by accident. I don't have that luxury, so the review has to be built in on purpose.
What actually broke?
The worst bug I found this year lived in the pipeline that reviews queued items and decides approve or reject without me checking each one. Under the right conditions, it could approve an item a human reviewer had already explicitly rejected. That's not a typo bug. That's a compliance bypass, the exact failure a review gate exists to prevent, sitting inside the review gate itself.
I only caught it because I'd already started running anything consequence-path, publishing, approvals, money, data, through three separate reviews before shipping: my own read of the diff, an adversarial subagent whose only job is to try to break it, and a different model reviewing the same code cold, with none of my framing. Three passes on one piece of code turned up seven real bugs. Zero overlap between them. The bypass was the one the third pass caught. The first two missed it completely.
Why didn't the first review catch it? Why didn't the second?
My own review missed it because I was checking the paths I already expected to be a problem. I wrote the logic, so I tested it against my own mental model of how it would fail. That model didn't include a bypass, because I didn't think to write a bypass in the first place.
The adversarial review, a subagent whose only job is trying to break the code, missed it too, and that surprised me, because "try to break it" should be exactly the job that finds a bypass. It didn't, because it was still reasoning from roughly the same assumptions I'd built the code with. The pass that found it was running on a different model with zero context on what I'd assumed going in. It didn't know what I "meant." It just read what the code did. One review, no matter how careful, only samples one region of the failure space. Three reviewers with different blind spots covered ground none of them would have covered alone.
What else turned up in that same pass?
Two smaller findings from that same week are worth keeping, because they're not really about one bug. They're about a category of bug that autonomous systems produce by default. First: a couple of my own cron-run instructions told an agent to "revert" or "restore" changes with no scope attached. Read literally by a headless agent running with broad edit permissions, that's an open invitation to wipe an entire directory instead of undoing one file. Nobody wrote it that way on purpose. It drifted in as shorthand, and shorthand is exactly what a machine won't fill in the way you meant.
Second: part of the system mines dated log entries with a regex written for one exact format. A different part of the system started writing entries in a slightly different format that same week, no bold, different punctuation. Nothing errored. The miner just silently processed zero new entries and moved on like it had done its job. No crash, no warning. Just quiet data loss. That's the pattern I keep running into. An autonomous system doesn't usually fail loudly. It fails by doing nothing and reporting success.
What would I do differently, building an agent that acts while I'm not watching?
I'd put the review gate in before the feature, not after. The rule exists because a bug already almost shipped, not because I planned ahead of time. Three things are non-negotiable now for anything touching a consequence path:
- Every consequence-path change gets three independent reviews before it ships: self, adversarial, and a genuinely different model reviewing cold. No exceptions for "it's a small change."
- Every destructive verb in an unsupervised skill, revert, restore, delete, reset, names its exact scope in the instruction itself. "Revert" with no object is a loaded instruction handed to something that can't ask what you meant.
- Anything that parses a hand-written log gets a test that fails loud on a format it doesn't recognize, instead of silently treating zero matches as nothing to do.
What's the actual misconception about self-improving AI systems?
The misconception is that "self-improving" means the system gets safer on its own over time. It doesn't. It means the system gets more places to fail quietly, faster, unless a human designs the failure to be loud. Autonomy doesn't remove risk. It removes the moment you'd normally notice risk. The improvement isn't automatic. The review discipline around it is the actual product.
What paid off is boring, which is usually how you know something real happened. Nothing shipped with that bypass in it. Three reviews and a rule I now apply by default caught a bug before a client, or anyone downstream, ever saw it. That's the whole win. Not a growth chart. A thing that didn't happen.
If you're building anything that acts on your behalf while you're not watching, and you're running it solo, budget for the third review before you budget for the next feature. It's not overhead. It's the only thing standing between "self-improving" and "wrong, faster, with nobody around to notice."