An automation pilot is the smallest version of an automation you can put in front of real work and learn from. Not a demo. Not a slide. A working system doing one job, in production, for a team that will tell you fast if it breaks. Most first automations fail because they were never scoped as a pilot. They were scoped as a program, with a steering committee, a roadmap, and a launch date nine months out.
This playbook is the opposite of that. It takes one workflow from the day you pick it to the day you decide whether to keep it, kill it, or grow it, in about 90 days. You will learn what an automation pilot actually is, how to choose the right one, a week-by-week plan, how to define success before you build, and the mistakes that quietly kill a pilot before it can prove anything.
What is an automation pilot?
An automation pilot is one workflow, automated end to end, running against real work in production for a fixed window, with a clear rule for what happens next. It has edges. It handles a defined case, a defined volume, and a defined owner. When the window closes, you have data, not opinions, about whether to scale it.
The word that matters there is production. A pilot that runs in a sandbox proves nothing except that the sandbox works. Real work is messier than any test case. It has the malformed inputs, the edge cases, and the one customer who always does the thing nobody planned for. You want to meet all of that early, while the stakes are small and the system is easy to change.
A pilot is also time-boxed. Open-ended pilots do not end. They drift into a permanent "we're still testing it" state where nobody owns the decision to scale or stop. Put a date on it.
Why your first automation should be a pilot, not a project
A pilot lowers the cost of being wrong. The first automation a team builds is the one they understand least, because they have not yet learned how their own work resists automation. Betting a nine-month project on that first guess is how budgets die quietly. A pilot lets you be wrong in three weeks instead of three quarters.
There is a second reason, and it is about trust. Teams do not adopt automation because it is clever. They adopt it because it made someone's Tuesday easier and did not embarrass them in front of a customer. A small pilot that clearly helps one team is worth more than a grand plan nobody believes yet. It earns the right to the next build.
The trap to avoid: treating the pilot as a throwaway. Build it as if it will go to production, because it will. The point of a pilot is not lower quality. It is smaller scope.
How do you choose the right pilot workflow?
Pick a workflow that is high-volume, rule-heavy, low-stakes when it errs, and owned by one team that wants the help. Those four traits are what make a pilot both buildable and safe to learn on. Chase a flashy, high-stakes process first and a single bad output can end your automation program before it starts.
Run every candidate through a short test.
- Is it boring and bleeding? Boring means repetitive and rule-based, the same steps most of the time. Bleeding means it costs the team real hours or real errors today. Boring-and-bleeding is the sweet spot. Nobody misses doing it, and everyone feels the fix.
- Can one team own it? If the workflow crosses four departments, the pilot becomes a negotiation, not a build. Start where a single owner can say yes.
- Is a wrong answer recoverable? A misrouted internal ticket is recoverable. A wrong number on a client's tax filing is not. Save the irreversible processes for after you have earned confidence.
- Is there enough volume to learn from? Five cases a month will not tell you much in 90 days. Fifty a week will.
If you have mapped your operation already, this gets easier. The eight operational objects framework is built to surface exactly these candidates: the hand-made Decision done fifty times a day, the Knowledge that lives in one person's head. Those are your pilots.
And before you commit, run the numbers. A quick ROI model you build before the pilot tells you whether the payback is worth the 90 days. If the honest answer is no, pick a different workflow. A pilot that succeeds technically but saves nothing is still a failure.
The 90-day automation pilot playbook
Here is the plan, in four blocks. The dates are a guide, not a contract. A simple workflow finishes faster. A gnarly one earns another two weeks. What matters is the order.
Weeks 1 to 2: Scope and map
Do not build yet. Map the workflow as it really runs, following one live case from the trigger to the finished outcome. Write down every step, every rule, and every point where a person makes a judgment call. This is where you find the leaks, and the leaks are the reason to automate.
By the end of week two you should have three things: a map of the real process, a written definition of success, and a named owner on the team who will run the pilot day to day. If you cannot get a named owner, stop. An unowned pilot is a hobby.
This scoping work is exactly what a two-week Executive Study produces, if you would rather have it done with you than do it cold.
Weeks 3 to 6: Build small
Build the narrowest version that does the real job. Automate the common path first, the case that happens 80 percent of the time. Do not try to handle every exception in version one. Route the weird cases to a person and log them. Those logs are gold; they tell you which exceptions are worth automating later and which are so rare they never will be.
Decide early where a human stays in the loop. For anything with customer-facing or financial stakes, keep an operator reviewing the agent's work before it goes out. That review step is not a lack of trust in the system. It is how you build the trust.
Weeks 7 to 10: Run in production
Turn it on for real work. Start at low volume, watch it closely, and widen the tap as it earns confidence.
This is the phase that separates a pilot from a demo, because production is where automations tend to die. Inputs change, an upstream system updates, a rule you assumed was fixed turns out to have three exceptions.
Instrument it from day one. You want to see throughput, error rate, how often work falls back to a human, and how long the whole thing takes end to end. If you are not sure which numbers matter, the metrics an ops team should track is a good starting list. Watch the fallback rate especially. A pilot that routes 60 percent of cases to a human is telling you the common path was not as common as you thought.
Weeks 11 to 12: Decide
Close the window and make the call. Three outcomes, and all three are fine:
- Scale it. It works, the team trusts it, the numbers hold. Fold it into the roadmap and pick the next workflow.
- Fix and extend. The core works but the exception handling is thin, or the payback is smaller than hoped. Give it another cycle with a tighter scope.
- Kill it. It cost more to run than it saved, or the work turned out to need judgment a system cannot fake yet. Write down why, and move the effort to a better candidate.
The only bad outcome is no decision. A pilot with no verdict becomes shadow infrastructure that nobody maintains and nobody trusts.
How do you define success before you build?
Write a one-line success rule with a number and a date, agreed by the pilot owner, before any code exists. Something like: "By day 90, this handles 70 percent of routine cases without a human touching them, at the same accuracy the team hits today." Vague goals like "improve efficiency" cannot be passed or failed, which means the pilot can never end.
Tie the number to something the team already feels. Hours given back, errors avoided, a backlog cleared, a response time cut. Pick the one metric that would make the owner say "yes, that was worth it," and make that the headline. Track the rest as supporting detail.
One expert tip: define the failure line too. Decide in advance what result would make you kill the pilot. Writing that down before you are emotionally invested in the build is the single best defense against sunk-cost thinking later.
Common mistakes that kill a pilot
- Scoping a program and calling it a pilot. If it has a steering committee and a six-month timeline, it is not a pilot. It is the thing a pilot is supposed to prevent.
- Never leaving the sandbox. A pilot that only ever runs on test data proves the test data works. Get it in front of real work, safely, early.
- No owner on the team. If the only people who care are the ones who built it, adoption dies the day the builders leave.
- Automating the exceptions first. Teams love to prove the hard cases. Build the common path first; the exceptions are version two.
- No success rule. Without a written line to clear, every result becomes a debate, and the pilot never closes.
- Ignoring the fallback logs. The cases that fall back to a human are the roadmap for the next build. Throwing them away throws away the learning.
Best practices for running an automation pilot
- Keep a human in the loop where it counts. Review the outputs that carry customer or financial risk. Automate the routing and the drudgery; keep judgment supervised until the numbers earn its release.
- Ship to production, not a demo. Real inputs teach you things no test case will.
- Instrument before you launch, not after. You cannot learn from a pilot you did not measure.
- Widen the tap slowly. Low volume first, watch, then open it up. A quiet failure at 5 percent is a lesson. The same failure at full volume is an incident.
- Time-box it and honor the date. The deadline is what forces the decision, and the decision is the whole point.
Teams that follow this get to a real verdict in a quarter instead of arguing about a roadmap for a year. And a workflow that clears a pilot is a far easier sell for the next workflow automation build, because now you have evidence instead of a pitch.
A real-world shape: an exception backlog
The pattern shows up clearly in operations work. A 3PL drowning in delivery exceptions is a textbook pilot candidate. The work is high-volume and rule-heavy, one team owns the queue, and a misrouted exception is recoverable.
The pilot version does not try to close every exception type. It takes the most common one, automates the triage and the routine responses, keeps a coordinator in the loop for the judgment calls, and logs everything else. Run that for a few weeks against the live queue and you learn two things fast: how much of the backlog is genuinely routine, and where the real judgment lives. That is the whole purpose of a pilot. It turns a vague sense that "exceptions take too long" into a specific, costed decision about what to build next.
Frequently asked questions
How long should an automation pilot run?
Long enough to see real variety in the work, usually eight to twelve weeks of live running on top of a couple of weeks to scope and build. Any shorter and you miss the edge cases. Much longer and it stops being a pilot and becomes unowned infrastructure.
What makes a good first automation pilot?
A workflow that is repetitive, rule-heavy, high in volume, owned by a single team, and forgiving when it errs. Boring and bleeding is the phrase to remember: dull enough to be predictable, painful enough that fixing it matters to the people who do it.
How is a pilot different from a proof of concept?
A proof of concept shows the technology can work, often on sample data. A pilot shows the workflow works in production, with real inputs and a real owner, against a success rule you can pass or fail. A pilot is the harder, more useful test.
Should a pilot be fully autonomous?
Usually not at first. Keep a person reviewing anything with customer or financial stakes, and automate the routine steps around them. As the numbers prove the system is reliable, you widen what it does without a human check. Trust is earned in that order, not assumed.
What if the pilot fails?
Then it did its job. A pilot that fails in 90 days cost you a quarter, not a year, and it taught you where your work resists automation. Write down why it failed, move the effort to a better candidate, and count the money you did not spend on scaling the wrong thing.
Who should own an automation pilot?
Someone on the team that does the work, not just the builders. The owner runs it day to day, feels the wins, and flags the problems. Without that person, the pilot has no advocate once the build phase ends, and unowned automations are the first to rot. See why automation dies in production for what that rot looks like.
Final thoughts
A pilot is the cheapest way to be wrong and the fastest way to be right. Pick one boring, bleeding workflow. Give it a named owner and a one-line success rule. Build the common path, run it against real work for a fixed window, and make an honest call at the end. Do that and your automation program grows on evidence instead of hope.
If you want the first pilot scoped, costed, and prioritized before you commit a quarter to it, that is what a two-week Executive Study delivers, and it follows the same STAR method this playbook is built on. Start a conversation and we will help you pick the right one.
