Now accepting guest editorial from SAP, Oracle, Workday and ServiceNow practitioners.Pitch a guide
The Sign-Off Table

Lesson 5 of 5

Progress metrics

10 min

Listen to this lesson

0:00 / 7:49

Modernization is easy to fake. Lines of code rewritten, number of RAP apps built and percentage of "modern syntax" all go up while upgrade risk and defect rates stay the same. Grace needs metrics that show whether the estate is genuinely getting safer to change and cheaper to upgrade, and whether the program is breaking anything along the way.

Outcome metrics

MetricWhat it showsContoso baseline → year 1 target
Extensions by clean core level (A/B/C/D)Shift of the estate toward upgrade-stable levels18/64/121/37 of 240 → fewer than 20 at level D by end of year 2
Level D in critical processesThe riskiest items where it hurts most11 → 0 in order-to-cash and finance by end of year 1
Extensions retiredEstate shrinking; less to test and upgrade0 → 47 confirmed retirements done
Upgrade adjustment effortSPAU and SPDD items, custom code fixes, regression defects per upgradeBaseline from the conversion → measured at the next upgrade
Critical extensions with automated testsHow safely the estate can change4 of 60 → 40 of 60 with characterization or unit tests

Outcome metrics for Contoso's modernization. Targets are the program's own, set by Grace's board.

Safety metrics

Outcome metrics can improve while the program quietly causes incidents. Track safety alongside: escaped defects traced to modernization slices (found in UAT or production), slices rolled back, and reconciliation differences found in parallel runs. A rising level A count with rising escaped defects means the team is moving too fast or testing too little.

Outcome metrics only count as progress when the safety metrics show the program is not breaking things along the way.
The Sign-Off Table · thesignofftable.com/academy/sap-abp-401
Board viewLevel trend, level D in critical processes, upgrade effort, escaped defects
Program viewSlices planned, in flight, finished; open switches; retired extensions; test coverage of critical extensions
Team viewATC findings trend, characterization suites added, unit test failures, flaky tests
Data sourcesATC clean core runs, usage data, SAP Cloud ALM features and test results, defect records
Three audiences, one set of sources. Every number on the board view traces back to ATC, Cloud ALM or defect data.
The Sign-Off Table · thesignofftable.com/academy/sap-abp-401

Collect these numbers the same way every time. Clean core levels come from a scheduled ATC run against the pinned classification file; retirements and slices from features in SAP Cloud ALM; escaped defects from the defect records Priya's team keeps, tagged with the slice that caused them; upgrade effort from the adjustment lists and the regression results of the upgrade project. If a number cannot be traced to one of these sources, leave it off the board view.

Every number on Grace’s board view traces back to one agreed source; anything that cannot is left off.
The Sign-Off Table · thesignofftable.com/academy/sap-abp-401

Leading and lagging indicators

Upgrade effort is the metric that matters most and the one you see least often: once a year at best. Use leading indicators to steer in between. Characterization coverage of critical extensions predicts how safely you can refactor them. Level D in critical processes predicts upgrade pain. Open switches and half-finished slices predict maintenance cost. When the yearly upgrade comes, compare its adjustment effort and regression defects with the baseline to confirm that the leading indicators were telling the truth.

  1. MonthlyLeading indicatorsLevel shifts, retirements, test coverage of critical extensions, open slices and switches
  2. Per releaseSafetyEscaped defects and rollbacks from modernization slices; reconciliation results
  3. Per upgradeLagging outcomeAdjustment items, custom code fixes and regression defects against the conversion baseline
When each kind of metric is reviewed.
The Sign-Off Table · thesignofftable.com/academy/sap-abp-401

Vanity metrics to avoid

  • Lines of code rewritten: rewards churn, not risk reduction.
  • Number of new RAP apps: rewards building, not retiring or simplifying.
  • Overall code coverage percentage: invites tests without real assertions; track critical extensions with meaningful tests instead.
  • ATC finding count alone: deleting unused code lowers it as much as real remediation; show both, separately.

What would you do?

At the year-one review, the program reports 22 new RAP apps and 140,000 lines of legacy code rewritten. Level D in critical processes went from 11 to 9, and two modernization slices caused production incidents. What is the honest assessment Kenji should give?

Key points
  • Measure outcomes: level shift, level D in critical processes, retirements, upgrade effort, tested critical extensions.
  • Measure safety: escaped defects, rollbacks and reconciliation differences from slices.
  • Steer with leading indicators monthly; confirm with upgrade effort against the baseline.
  • Avoid vanity metrics such as lines rewritten or apps built; trace every number to its source.

Check your understanding · not graded

Lesson check

  1. 1. Which is an outcome metric for modernization?

  2. 2. Why track escaped defects from modernization slices?

  3. 3. Which is a leading indicator of upgrade pain?

  4. 4. Why is ATC finding count alone a misleading progress metric?