The successful pilot that did not pass the investment test
Testing attribution before an attractive pilot becomes an enterprise commitment
PAWorld27 Zurich · 3–4 March 2027 · Zürich
About this session
A pilot team reports stronger output after receiving new AI tools. In this worked scenario, an evaluation reconstructs the baseline and finds that demand, staffing and case allocation also changed. A phased comparison and sensitivity tests reduce confidence in the claimed effect.
The session examines how to design that evaluation, and how to tell insufficient evidence of value from evidence that a tool cannot work, so that the next step is a smaller, better-targeted test rather than a full licence expansion or an outright rejection.
Learning outcomes
- Identify the comparator and confounders behind a before-and-after claim.
- Specify a feasible phased evaluation and its remaining limitations.
- Use a minimum worthwhile effect and an uncertainty assessment to fund, defer or stop.
Strategic questions
How much uncertainty is acceptable for a reversible commitment compared with a costly one?
Context
Business cases often rely on before-and-after changes that coincide with shifts in staffing, demand or allocation. Investment decisions need to account for what would have happened anyway and for how much uncertainty matters to the commitment being made.
