Before you buy the next AI automation, check one thing first: does it depend on someone else’s system to keep working. If it does, you are not buying autopilot. You are renting a dependency, and you become the person who babysits it when the dependency changes. That is the answer most of the pitches skip. The demo runs clean because the demo runs once, on a good day, on the vendor’s terms. The grind starts in week two.
I am Jake. I run a one-person shop in Atlanta. I spent about seven years in economics before this, which mostly taught me to distrust a number with no cost attached. I have bought AI automations that failed. I have built AI automations that failed. One ran green for two weeks and produced nothing. One I opened exactly once. I am not writing this to sell you on AI being broken. I use it every day. I am writing it because the generic “put your business on autopilot” pitch is dying, and it should, and you deserve the checklist I wish I had run before I paid.
The real problem owners describe is not that AI does not work. It is the babysitting tax. You get sold hands-off, then you spend your week supervising, cleaning up, and paying hidden costs for something fragile. Adoption is easy. Integration is the grind. So here are the five checks I run now before I trust or buy anything, and I run them in this order.
Does it depend on someone else’s system?
Ask what breaks the automation when a vendor changes something, because the ones that die on you almost always die upstream. An API changes a field name. A platform deprecates an endpoint. A model gets quietly swapped for a cheaper one and the output drifts. You did nothing, and now it is broken, and you are the one on the hook. The first automation I built that actually paid for itself survived because I owned most of its moving parts, which I wrote about in a post on the one automation that paid for itself and why it did not break. The ones that broke all shared a trait: they leaned on a system I did not control and could not see into. Count the external dependencies before you buy. Every one is a future outage with your name on it.
Can you check its output at a glance, not just exit code 0?
An automation is only trustworthy if you can verify its work in a few seconds without reading logs, because “it ran” and “it did the right thing” are different claims. This is the one that cost me the most. I had a job that reported success every single run. Green status. No errors. It ran green for two weeks and produced nothing usable, and I only caught it when a downstream task came up empty. I broke down that whole mess in a post on the automation that failed silently for two weeks. Exit code zero means the code finished. It does not mean the work is correct. Before you trust an automation, decide how you will eyeball its actual output, a sample row, a rendered draft, a count that should match, in one glance. If you cannot, you have not bought automation. You have bought a thing you have to audit by hand, which is the job you were trying to remove.
Does it save net time after a human reviews it?
Measure the time the task takes including your review and cleanup, not just the time the machine takes to run. A tool can run in four seconds and still cost you twenty minutes, because now you are reading, correcting, and re-prompting what it produced. I shipped one that was genuinely fast and still made my week slower, and I traced the whole accounting in a post on the AI tool that ran fast and cost me net time. The honest formula is simple. Old time to do the task by hand, minus new time to run plus review plus fix. If that number is not clearly positive, the automation is a hobby, not a tool. Run it on paper for one real task before you pay for a year.
Will you actually open it a second time?
Be honest about whether this fits a workflow you already return to, because the graveyard is full of tools that worked fine and got used once. I have bought an AI tool, used it on the day it arrived, and never opened it again, and I wrote about exactly why in a post on the AI tool I used once and never opened again. The tool was not bad. It just did not sit anywhere in how I actually work. A novelty gets opened once. A tool gets opened on a Tuesday when you are busy and did not plan to. Before you buy, point to the recurring moment in your week where this lives. If you cannot name the moment, you are buying a demo you enjoyed, not a tool you will use.
What breaks it, and will it fail loud or silent?
Find out how the automation behaves on a bad day before you depend on it on a good one, because the failure mode matters more than the success rate. A thing that crashes loudly is annoying. A thing that keeps running while quietly producing garbage is dangerous, because you trust it right up until it costs you a client. I had an integration pass its demo clean and break in week two, and the breakage itself was the lesson, which I unpacked in a post on the integration that passed the demo and broke in week two. Ask the vendor, or ask yourself if you built it: what is the single most likely thing to break this, and when it breaks, do I get a scream or silence. If the answer is silence, add a check that turns it into a scream before you rely on it.
So what do you buy, then?
Buy the automation that passes four or five of these, and walk from the one that passes none, no matter how good the demo looked. Most of the value is not in the model. It is in the integration, the review step, and the plumbing that makes a failure visible. That is the part the hype pitch leaves out, because it is the part that is real work. The numbers say the same thing. In a Goldman Sachs survey of 1,256 small-business owners, 93 percent of those using AI reported a positive impact, yet only 14 percent had fully integrated it into their core operations, and 73 percent said more training and support would help them implement it. Belief is not the bottleneck. Integration is, and that is exactly what these five checks are about.
That gap is the whole job. I am a solo founder, so when I build an automation for someone I do not get to hand off the babysitting; I have to make the thing hold on its own, which is why I run this checklist on my own work before anyone else sees it. If you want that kind of restraint applied to your stack rather than another autopilot promise, that is the AI and automation work I take on. And if you only take one thing from this: run the five checks before the money leaves your account, not after.
I was wrong twice. Both times I skipped check one and check two. Both times the demo was great. Do not buy the demo. Buy the thing that still works in week two, on a bad day, when nobody is watching it.