Lesson 5 of 6
Value metrics
13 min
A TCoE that cannot show its value will be cut in the next budget round. Counting test cases written or hours logged does not help, because those measure activity, not value. Value metrics answer three questions the business cares about: are we protected, are we fast, and are we efficient?
A balanced set
| Question | Metric | How to measure | Contoso target |
|---|---|---|---|
| Protected? | Escaped defects | Defects found in production within an agreed period after a release, by severity | Zero severity 1; downward trend overall |
| Protected? | Defect detection percentage | Defects found before release ÷ (before + escaped) | Above 90% per release |
| Protected? | Regression coverage of critical processes | Share of critical end-to-end processes with maintained regression tests | 100% of critical, 80% of all |
| Fast? | Regression cycle time | Elapsed days from build ready to regression results | From 15 to 5 working days |
| Fast? | Data and environment lead time | Days from request to test-ready data or environment | Under 3 days |
| Efficient? | Cost per regression cycle | Internal and partner effort plus tool cost, per full cycle | Down 40% over two years |
| Efficient? | Automation share and stability | Share of regression executed automatically; share of runs failing for script, not system, reasons | 60% automated; under 5% false failures |
A balanced TCoE scorecard. Choose a few metrics, define them precisely and track trends.
Escaped defects: the metric that matters most
An escaped defect is one that reached production even though the change went through testing. Track them by release and severity, and do a short root-cause review on each serious one: was the scenario missing from regression, was the data unrealistic, was the test skipped under pressure, or was the defect untestable before production? The answers feed directly into the regression pack and the standards. Watch for a trap: if production incidents are not linked back to the release that introduced them, escaped defects will look artificially low. Agree with the support team how incidents are tagged to releases.
Cost per regression cycle
This metric makes automation and test data investments visible in money. Measure the full cost of one complete regression cycle: TCoE and business tester hours, partner fees, environment and tool costs. Contoso's first post-go-live regression cost around 1,400 hours, much of it key users. A year later, with 60% automation and data recipes for the main processes, a comparable cycle took about 700 hours, and business testers focused on judgment, not keystrokes.
- Does the metric reflect an outcome the business cares about?Yes → keep · No → demote to an internal activity metric
- Can it be measured consistently from existing data?Yes → automate collection · No → define the data source first
- Could it be gamed by changing behavior, not quality?Yes → pair it with a balancing metric · No → publish
- Does someone act on it?Yes → include in the quality council pack · No → drop it
Reporting rhythm
Report to two audiences on two rhythms. Per release, give the release manager and process owners a short quality report: scope tested, results, open risks, escaped defects from the previous release. Quarterly, give the quality council the scorecard trends, roadmap progress and the investment case for the next step. Keep the per-release report factual and fast; keep the quarterly report about trends and decisions. Use the same definitions every time, so a quarter-on-quarter change means a real change, not a new way of counting.
What would you do?
After a release, Contoso's TCoE dashboard shows a 99% regression pass rate and a 4-day cycle time, but three severity 2 incidents in order to cash were reported in the first week of production. What should the TCoE lead report to the quality council?
- Report value (protected, fast, efficient), not activity.
- Escaped defects by release and severity are the core protection metric; root-cause each serious one.
- Cost per regression cycle makes automation and data investments visible in money.
- Pair metrics so they balance each other, and never use them to rank individuals.
Check your understanding · not graded
Lesson check
1. Which metric best indicates a TCoE is protecting the business?
2. How is defect detection percentage calculated?
3. Why might escaped defects look artificially low?
4. Which metric makes automation investment visible in money?