Most first AI pilots don’t fail. They just never finish.
Nobody agreed what good looks like. Nobody can switch it off. Six months later it’s still “going quite well”, and nobody can say what it has changed.
Back in September I shared a short checklist for a first AI experiment. It got more replies than anything else I’ve posted, and most of those replies made it better. So here’s the full version, with your additions in it.
To keep it practical, I’ll run one example all the way through. Kieran Gilmurray suggested it in the replies: meeting prep. Clear benefits, manageable risks.
Imagine a small team with a weekly client meeting. Before each one, someone spends time pulling together last week’s notes, open actions and recent emails into a one-page brief. The experiment is simple. AI drafts the brief, and a person checks it before the meeting.
Here’s the checklist.
1. Small enough to stop
If switching it off needs a meeting, it’s too big. A good first experiment is one you can stop on a Friday without anyone noticing.
Meeting prep: one team, one recurring meeting, one brief. Not every meeting in the business.
2. Clear enough to measure
You need something you can count without arguing about it. Minutes. Mistakes. Corrections. Whether the thing actually got used.
Meeting prep: how long the brief takes to produce, and how many things the person checking it had to fix.
3. One owner, with the authority to stop it
“The team owns it” means nobody owns it. Tommaso Maria Ricci put it well: when the experiment belongs to a committee, nobody is allowed to call it a failure, so it drifts for two quarters and gets renamed.
Harish Gupta, DPA added the bit that usually gets watered down. One name means one person who can kill it without a meeting. If stopping needs a committee, it won’t stop.
Meeting prep: the person who runs the meeting owns it. They can pause it on the spot, no sign-off needed.
4. A clear definition of “better”
Faster? Fewer mistakes? Less chasing? Pick one before you start. Otherwise every result looks like it might be progress.
Meeting prep: “better” means the brief is ready the day before, and the meeting lead spends less time on it than they do today, without missing anything that matters.
5. Write the failure down first, and who acts on it
Adam Faik suggested this one. Write down what a wrong answer looks like before you start, so you know when to kill the experiment instead of arguing from vibes.
Haseeb Afsar took it a step further. The same list tells you who has to act when that failure happens. Put a name on the same line as the failure. Teams notice. People act.
Meeting prep: the failures are an action marked as done when it isn’t, or notes pulled in from the wrong client. The meeting lead checks every brief, and if the wrong client shows up even once, they pause it.
6. Measure the human first
Cristobal Santana ’s addition. Make the experiment easy to compare against a human baseline. Otherwise you can’t tell whether AI improved the work or just changed it.
Do the job by hand once or twice. Time it. Write down what went wrong. The AI has to beat the person doing the job today, not an empty page.
Meeting prep: time the next two briefs done by hand, before the AI touches anything. Note the minutes and anything that got missed.
7. Set the stop date and the stop rule up front
Tommaso asked me in September whether I set the stop date up front. I do. I’d rather run a small pilot for 30 days, learn from it, then make a deliberate decision: scale it, change it, or stop it.
Waterreptile19 added the rule that goes with the date: “if it doesn’t save X hours in 30 days, we pause it”, agreed at the start so the decision isn’t emotional later. Put the date in the diary too, so 30 days doesn’t quietly become 90.
Meeting prep: four weekly meetings, then a decision. The rule is agreed on day one. If the brief isn’t saving the meeting lead time by week four, or the wrong-client failure happens, it stops.
When you stop, write down why
Mykyta Pavlenko said it in one line: a stop isn’t progress if you never learn what broke. A stopped pilot with a short note on what went wrong is a result. A stopped pilot with no note is just a wasted month.
The whole thing on one page
Small enough to stop
Clear enough to measure
One owner who can stop it
A clear definition of “better”
The failure written down first, with a name next to it
The human measured first
A stop date and a stop rule agreed up front
None of this needs a big budget or a transformation programme. It needs one task, one person and a date in the diary.
Three of these seven came straight from your replies, and the rest got sharper because of them. So I’m asking again.
If you were switching on your first AI pilot next Monday, what’s the one check you’d add that isn’t on this list?