Chapter 8
Evaluating the result and choosing what comes next
Combine task observations, product signals, and operational evidence while keeping outcome claims proportionate to what was measured.
After the pilot, the dashboard shows fewer repeated assignment attempts. The team wants to announce that the feature reduced wasted work. The analyst asks whether the pilot administrators were more experienced and whether the event definition changed during release.
Those questions affect the claim you can make. The decision is what the evidence supports and whether to iterate, expand, observe longer, or retire the change.
Return to the original outcome
We wanted administrators to complete an appropriate assignment and know what happened. Repeated attempts were one possible signal of difficulty. They were never the complete outcome.
Review task completion, routing correctness, uncertain outcomes, support questions, and accessibility alongside repeated attempts. Fewer attempts could mean clearer decisions, but they could also mean that people abandoned the flow or lost access to an action.
Check the instrumentation first. Confirm the event definitions, cohort, time window, and missing data. If the new interface stops sending a particular event, its disappearance does not demonstrate a better experience.
Work through a small example
The following counts are invented for this exercise. They do not describe actual usage, sessions, or research results. The two periods are deliberately too limited to establish a causal effect.
| Signal | Earlier period | Pilot period | Interpretation limit |
|---|---|---|---|
| Submitted assignment attempts | 100 | 100 | Equal counts do not imply comparable cases or administrators |
| Attempts classified as repeats on an unchanged case | 20 | 12 | Classification and coverage need verification |
| Confirmed successful attempts | 76 | 79 | Does not establish correct routing or whether the task was completed overall |
| Attempts whose outcomes remain unconfirmed in the available data | 4 | 6 | Recovery may be incomplete or measurement may be missing |
The rows describe different signals and should not be added together; a repeat attempt may also succeed. The repeated-attempt proportion is 20% in the earlier period and 12% in the pilot. That is an observed difference of eight percentage points in the supplied data. It is not evidence by itself that the feature caused the difference.
The unconfirmed outcomes require attention before expansion. A visible improvement in one measure does not cancel a potentially more consequential problem in another.
Look for an explanation beyond the headline
Ask who was in each cohort, how cases differed, and what else changed. Experienced administrators might handle eligibility more efficiently regardless of the interface. Training, staffing changes, a service incident, or a different case mix can alter the result.
Where feasible, work with an analyst on an appropriately designed comparison, including assignment of cohorts and measurement duration. In a before-and-after review, describe the limitations directly. You can still learn from an imperfect pilot without presenting it as a controlled experiment.
Seek qualitative evidence around the uncertain outcomes. Did administrators understand that they needed to check the record? Could they find the check? Did the service respond? Did they continue in another tool? Task observation and support conversations can identify mechanisms that aggregate counts hide.
Make a decision with the evidence available
For this fictional exercise, the recommendation is to hold expansion while investigating the unconfirmed outcomes and validating event coverage. Keep the pilot within its agreed support conditions if no stop condition has been met. If investigation reveals a critical defect, use the release plan to pause or roll back the interface.
Record what would enable a different decision: reliable classification of outcomes, verified recovery behavior, acceptable routing quality, and evidence from a broader range of administrators. Do not simply extend the pilot indefinitely. Name the question, owner, and next review point.
If the investigation shows that the new flow adds complexity without enough benefit, recommend narrowing or retiring it. Evaluation includes the cost of maintaining the solution.
Practice: write an outcome review
Use your actual evidence or the supplied fictional table, keeping their origins explicit. Define the measures and denominators, examine missing data, and separate description from explanation. Include a user-quality guardrail and a plausible alternative explanation.
Write a recommendation that identifies the decision, evidence, uncertainty, next action, and owner. Explain what would change your recommendation. Ask someone who favors expansion to critique the reasoning.
Review your work
Your claim should survive replacing “the feature caused” with an honest description of what you observed. The recommendation should address conflicting signals and the strength of the evidence. If no production release occurred, describe a proposed evaluation or a fictional-data exercise rather than reporting business impact.
The next chapter revisits the organizational dependencies that shape what the team can do next.
Found an error? Report it.