If you lead an organization today, AI proposals are probably arriving faster than you can evaluate them. Some come from vendors, some from enthusiastic teams, some from a board asking what your AI strategy is. Many of them come with an impressive demo.
Demos are easy to love. The hard part comes later, in the space between a convincing prototype and something people rely on every day. That is where many AI initiatives stall: the pilot "worked," but nobody can say what changed, who owns it, or whether it is safe to scale.
I design and build multi-agent AI solutions, and I hold Microsoft's AI Transformation Leader and AI Agent Builder certifications. The most useful lesson from that work is not technical. The initiatives that succeed are the ones where leaders asked a few plain questions before the money was spent.
Here are the five I would ask.
1 What work, or which decision, will be different, and for whom?
Why it matters. "We need to use AI" is not a problem statement. AI is a capability, not an outcome. A pilot needs a specific piece of work or a specific decision it will improve, and a specific group of people whose day will change.
"It will make us more efficient," or "Our competitors are doing it."
"Our analysts spend about two days assembling each quarterly report from five sources. We want a first draft produced in hours, which they then review and finalize."
2 What does "good enough" look like, and who checks it?
Why it matters. AI output is probabilistic. It will sometimes be wrong, and sometimes confidently wrong. That is not automatically a deal-breaker; people make mistakes too. But you need to decide in advance how good the output must be, how errors will be caught, and what an error costs.
"The model is very accurate," with no definition of accurate for this task.
"We have a set of real past cases with known correct answers. We will measure the pilot against them, and every output will be reviewed by a qualified person until the results justify lighter review."
In my own work, this is where most of the design effort goes: deciding what the AI does, what a person checks, and what guardrails keep the output reliable enough to review and trust.
3 What information will it need, and are we allowed to use it?
Why it matters. An AI solution is only as useful as the context it can access. That raises practical questions quickly: where does the information live, is it current, who is allowed to see it, and can it leave your environment? Privacy obligations, such as Quebec's Law 25, as well as client contracts and internal policies, do not pause for a pilot.
"We'll figure out the data once the pilot is approved."
"Here are the sources it needs, here is who owns each one, and our privacy and security teams have confirmed what can be used and where it can be processed."
4 Who will need to work differently, and what will make that possible?
Why it matters. A successful AI pilot is a change initiative, not just a technology project. If the people expected to use the tool do not trust it, understand it, or see how it helps them, they will work around it. The tool will be "deployed" and quietly ignored.
"We'll train everyone at launch."
"The analysts who will use it are helping define the quality bar. Their manager is sponsoring it visibly, and we have agreed how their role shifts from assembling reports to reviewing and improving them."
This is the same lesson change management teaches on any program: organizations do not change; people do, one at a time. AI does not get an exemption.
5 What happens after the pilot?
Why it matters. Many pilots are designed to succeed as pilots, not to become operations. Before you start, it is worth knowing what success would trigger, and what would make you stop.
"If it works, we'll scale it."
"If the pilot meets the quality bar and the analysts are using it after eight weeks, it moves to production with this owner, this budget for running costs, and this support model. If it misses the bar, we stop and keep what we learned."
Ask, too, about the cost of running it, not just building it. Usage-based AI costs, monitoring, updates, and support continue long after the pilot team moves on.
The pattern behind the five questions
None of these questions is about models, prompts, or vendors. They are about judgment: what problem you are solving, how you will know it works, what risks you accept, who has to change, and what you will do with the result.
AI is a powerful force multiplier, but it multiplies whatever it is given.
That includes vague goals, poor information, and unclear ownership. The leaders who get real value from AI are rarely the ones with the most impressive demos. They are the ones who insisted on clear answers before the pilot started.
A one-page check before you say yes
| Question | You are ready when... |
|---|---|
| What will be different, and for whom? | You can name the work, the people, and the change in one sentence |
| What is "good enough," and who checks? | There is a measurable quality bar and a named reviewer |
| What information, and are we allowed? | Sources, owners, and privacy constraints are confirmed |
| Who works differently, and how? | Future users helped shape the pilot and have a visible sponsor |
| What happens after? | Success and stop criteria, owner, and running costs are agreed |
If a proposal cannot answer most of these yet, that does not mean the idea is bad. It means the next step is a sharper problem definition, not a bigger budget.
Let's talk
If you are weighing an AI initiative and want a second set of eyes on the questions above, LinkedIn is the best place to reach me.