Lesson 6 of 6
Measuring throughput and quality of AI-assisted remediation
8 min
Listen to this lesson
"The agent fixed 600 findings in a day" is a throughput claim. It says nothing about whether those fixes were right, how long review took or how many came back as defects in testing. Before Grace funds AI Units for the whole custom code backlog, she wants numbers that cover both speed and quality. This lesson defines them and shows how Contoso ran a fair pilot.
The measures
| Measure | Definition | Why it matters |
|---|---|---|
| Findings closed per developer day | Priority 1 and 2 findings resolved and rechecked, divided by developer days including review | Throughput that counts review, not just generation |
| Auto-applied share | Agent fixes applied directly / all agent outputs | How much the agent really does on your code |
| Comment share | Low-confidence suggestions left as comments / all agent outputs | Hidden manual work that the headline number omits |
| Rework rate | Applied fixes changed or reverted during review / applied fixes | Quality of what the agent applied |
| Escaped defects | Defects in CMQ or mock testing traced to an agent fix, per 100 fixes | The quality number the business feels |
| Review hours per 100 fixes | Reviewer time recorded on pull requests or the review log | The true bottleneck once generation is fast |
| AI Units per closed finding | Consumption from the AI fix tools / findings closed | Cost, for the business case |
Measures for AI-assisted custom code remediation.
Running a fair pilot
Compare like with like. Contoso picked two packages from the sandbox cycle with similar size, usage and clean core profile: ZCMC_SD_EXT (sales extensions) for the agent, and ZCMC_MM_EXT (procurement extensions) for the team's usual method of quick fixes plus manual work with Joule explanations. Same developers, same review rules, same regression evidence. The pilot ran for two two-week sprints.
- Sprint 1, week 1BaselineCount findings by priority; capture ECC outputs for critical programs in both packages
- Sprint 1, week 2RemediateAgent run on ZCMC_SD_EXT; usual method on ZCMC_MM_EXT; reviews logged
- Sprint 2, week 1TestBefore-and-after comparisons and characterization tests; defects traced to fixes
- Sprint 2, week 2DecideCompare measures; Grace's steering group decides on scaling
| Measure (illustrative pilot figures) | Agent: ZCMC_SD_EXT | Usual method: ZCMC_MM_EXT |
|---|---|---|
| Priority 1 and 2 findings | 480 | 455 |
| Findings closed per developer day | 41 | 19 |
| Auto-applied share of agent output | 71% | n/a |
| Left as comments | 29% | n/a |
| Rework rate in review | 9% | 4% |
| Escaped defects per 100 fixes | 0.8 | 0.7 |
| Review hours per 100 fixes | 6.5 | 3.0 |
Contoso pilot results; figures are illustrative for this course.
Reading the results
The agent roughly doubled throughput, and escaped defects were about the same, which is the result Grace hoped for. Two numbers temper it. Rework was higher, and review hours per fix doubled, so reviewers, not generation, became the constraint. Scaling the agent to all 7,500 objects without adding reviewers would have pushed review quality down, and with it the defect rate. Contoso's decision: use the agent for level A and B packages with moderate usage, keep level D and high-criticality finance and order-to-cash objects with their owners, and add a second reviewer from Priya's team for agent pull requests during the CMQ cycle.
Keep measuring after the pilot
A pilot answers whether to scale; ongoing measures answer whether scaling still works. Contoso tracks the same measures per sprint through the CMQ conversion and the mocks, with one addition: defects found in Mock 1 and Mock 2 testing are traced back to the fix that caused them, by type (quick fix, AI fix applied, AI suggestion implemented by hand, manual fix). If AI-applied fixes start to cause more escaped defects than the other types, the scope narrows again. Grace's steering group sees the measures alongside the custom code burn-down, so a fast burn-down can't hide a quality problem.
Two cautions about the numbers themselves. First, AI Unit consumption depends on how SAP prices the capabilities under your contract, which can change; record consumption in units, not just currency, so comparisons over time stay valid. Second, measures can be gamed: a team rewarded for findings closed per day will prefer easy findings. Pair throughput with escaped defects and review hours in every report, and never show one without the others.
What would you do?
After the pilot, a vendor presentation claims "90% automatic remediation". Grace asks whether Contoso should expect that. What is the best answer?
- Measure throughput including review, auto-applied and comment shares, rework, escaped defects, review hours and AI Units.
- Pilot on comparable packages with the same people, review rules and evidence.
- Faster generation moves the bottleneck to review; plan reviewer capacity before scaling.
- Scope scaling by clean core level and criticality; keep high-risk code with its owners.
Check your understanding · not graded
Lesson check
1. Why include review time in "findings closed per developer day"?
2. What does the comment share tell you about an agent run?
3. In Contoso's pilot, which number argued against scaling without changes?