Measuring AI-Enabled Work
The quarterly AI report shows usage up, drafts up, hours saved and satisfaction holding. Someone wants a decision from it: expand the pilot or end it. Each number is accurate. The question is what each one shows.
What a results chain is
A results chain is the sequence of levels from what AI-enabled work did to what it was worth, each a separate claim that needs its own evidence.
A number is evidence only for its own level. A thousand summaries show that a thousand summaries were made. They do not show that anyone read one.
Moving up a level takes evidence that connects the two: drafts written to drafts accepted, say, or quicker service to customers who stay.
The levels form a chain of evidence, not a ladder of maturity. The term comes from program evaluation and is adapted here.
Seven levels
Take an insurer's claims team, constructed for this page. AI summarizes each new claim. A handler uses the summary to prepare a file, and a senior adjuster reviews the file and decides the claim.
| Level | Question | Insurer example |
|---|---|---|
| Activity | What did the AI and the people do? | Summaries generated, tokens used, hours the handlers report saving |
| Output | What units of work came out? | Files prepared for review |
| Accepted output | Which outputs passed the checks that define acceptable work? | Files that pass the completeness check |
| Delivered output | Which reached the recipient? | Files that passed the check and reached the adjusters' queue |
| Recipient outcome | Did the output help the recipient? | Adjusters decide without chasing missing documents |
| Customer outcome | Did the customer's situation improve? | Claims settled sooner, with no second request for the same document |
| Business outcome | What did the work do for the business, at what cost? | Cost per settled claim, and customers who renew |
Where a number sits depends on the work item being tracked, here the claim file. In another workflow, a summary could be the output. Tokens and reported hours saved are activity, because they measure effort, not work produced.
The adjuster is the file's recipient, and the claimant is the customer. The workflow test covers the output levels: was the file acceptable and within authorized work, and did it reach the adjuster? The value-delivery test starts at the recipient outcome: did the file help the adjuster decide? A better decision is a real result, but not yet a result for the claimant, so the test carries on to the customer outcome.
Failures have levels too. Sending a file back to the handler is a correction. An incomplete file is a defect: an output that failed. Asking a claimant twice for the same document is harm. The rework and the lost renewal that follow are the cost of failure. A report that counts corrections is counting activity.
Five ways a report misleads
Reading one level as the next
The handlers say the summaries save them time, and the report totals it in hours. Hours saved are activity, and even an accurate saving is at most the time it freed (Where Freed Capacity Goes). Self-reports can get the direction wrong. In a randomized study, experienced open-source developers took 19% longer on tasks where AI tools were allowed. But afterwards, they estimated that AI had made them about 20% faster (METR, 2025).
Move up a level only with evidence that connects the two.
Counting in the producer's unit
A file used to mean one claim. Now each submission counts as a file, so a complex claim that is submitted in two parts counts twice. The file count rises; the claim count does not. Nobody decided to redefine "file." Nothing in the report shows the drift, because the team that produces the files also defines what counts as a file.
Ask who defines the unit, and whether its definition is written down and versioned.
Counting repairs instead of harm
The adjusters used to send incomplete files back to the handlers. Now the adjusters chase the missing documents themselves. Returned files drop to almost none. Incomplete files do not. An adjuster who completes a file may spare the claimant any harm. But the defect still happened, and the returned-file count no longer shows it.
In DORA's change fail rate, a deployment counts as failed if it needs immediate intervention after release, whichever repair it gets: a rollback or a hotfix.
Count the defects themselves, whatever repair each one got, and count the harm separately, because a count of one kind of repair shows neither.
Dividing cost at the wrong level
Cost per summary falls every quarter. That figure says nothing about the cost per settled claim. That cost can rise while summaries get cheaper, if files still go back to the handlers. And the token bill may include work for other teams.
Separate the cost lines first, then divide by a unit at the level you want to price.
Combining windows or populations
The report shows counts of summaries for the busiest day, files for the month and settled claims for the year. Those figures cover different claims. A chain can still cross windows, because a claim may settle a month after its file is prepared. A figure's population is the cases or people it counts. Klarna said in February 2024 that in its first month its AI assistant did work equivalent to 700 full-time agents. Its chief executive said its customer service agents work for outside providers and are not Klarna employees. Klarna's 2025 filing counts about 5,527 full-time employees at the end of 2022 and 3,422 at the end of 2024. Those employees are not the outsourced agents whose work the 700 figure describes.
State the window and population of every figure. Combine or compare figures only when they follow the same cases, allowing for the time the work takes to reach the next level.
One published account, read level by level
Spawn, a ten-person company, lets people make multiplayer games by talking to an in-product AI, and agents do most of the company's own work. In October 2026 its founder, Jacob Sansbury, published detailed figures in an article and a follow-up post. Every figure here is his own and unaudited.
On the chain:
- Activity: $3.1 million on tokens in the month before the article, including free model use given to creators.
- Output: 77,000 "substantial changes" in 30 days, a producer's unit he defines by example: engine code or game logic, not small changes.
- Accepted output: no acceptance count; review by agents is described.
- Delivered output: deployments about hourly, a frequency rather than a count of what reached anyone.
- Recipient outcome: the changes reach players and creators directly, so here recipient and customer are the same people.
- Customer outcome: three measures from the follow-up post, all chosen by the founder: mobile crashes down from 95% of plays to 20% in six weeks, creators' ratings of the in-product AI, and retention for the best games.
- Business outcome: not reported; Spawn sells nothing yet.
Over the year before the article, 1,055 of 123,548 changes were reverted, about 0.9%. The founder later said the 123,548 was his commit count when he wrote the article. Broken changes, he says, are usually fixed by a new change on top, not reverted. So the 0.9% measures one kind of repair, and the kind he usually uses never shows up in it. But one mobile play in five still crashed. The two figures cover different windows and are not combined here. A revert count cannot tell you how often plays crash.
This reading tests the structure of the claims, not their truth, and is no verdict on Spawn.
Boundaries and distinctions
The lower levels are not worthless. File counts arrive this week; renewals can take months. Labeled as what they are, measures at those levels can be honest leading indicators. Say when you will look for the outcome.
The chain is not a return-on-investment method. Dividing by the right unit keeps a cost figure honest, but the full cost of useful AI work is a separate question.
A number that moved does not show what moved it. Settlement times can fall because the storm season ended, not because the files improved. Telling the two apart takes a baseline and a comparison, as Where Freed Capacity Goes sets out for redirected time and Reviewing AI Rules for a changed rule.
No detected difference is not proof of no effect. If settlement times have not moved, the effect may need more time to show, or a measure sensitive enough to detect it.
Applied test
- Take a report your organization relies on, about AI-enabled work.
- Place each number on the chain.
- Find the highest result the report claims, and check whether it was measured directly. If it was inferred from lower levels, name the evidence for each link and mark each unsupported link.
- Find one failure count and ask: does it count defects and harm, or only one kind of repair?
- Check that every cost figure is divided by a unit at the level of the result claimed, and that combined figures follow the same cases.
- Choose the missing measure most likely to change the decision the report supports, such as expanding or ending the pilot. Name who would be responsible for collecting it. If you cannot get the measure collected, take the gap to someone who can.
Sources and lineage
Measuring AI-Enabled Work is part of Applied Enterprise AI, a body of work developed by Curtis Hibbs and Joshua Barnes. The results chain is an established term in program evaluation (OECD), close kin to the Kellogg Foundation's logic model and Kirkpatrick's four levels of training evaluation. The OECD and Kellogg sequences start at inputs. This page starts at activity, splits outputs into three levels, and splits outcomes into the recipient's and the customer's. The line between output and outcome follows Seiden, and Kersten traces outcome gains back to what was delivered. DORA counts a failed deployment whichever repair it needed.
- OECD, Glossary of Key Terms in Evaluation and Results Based Management for Sustainable Development, second edition (2023).
- W. K. Kellogg Foundation, Logic Model Development Guide (2004).
- Kirkpatrick, "Techniques for Evaluating Training Programs", four articles from 1959, and Evaluating Training Programs: The Four Levels (1994).
- Josh Seiden, Outcomes Over Output (2019).
- Mik Kersten, Output to Outcome (2026).
- DORA, software delivery performance metrics guide (updated January 5, 2026).
- Becker, Rush, Barnes and Rein, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", METR, July 2025.
- Klarna, press release, February 27, 2024; Klarna Group plc, Form F-1, March 2025; CBS News interview with Klarna's chief executive, March 2024.
- Jacob Sansbury, "How I Built the Most AI-Pilled Company in the World", X article, October 28, 2026, and follow-up post on X, October 30, 2026.