From cycle metrics to TCoE metrics

A test lead on a project measures a cycle: how many tests ran, how many passed, how the defect curve is bending, whether exit criteria will be met by Friday. That is the subject of sap-tst-307, and it ends at a go/no-go decision. A TCoE has a different question to answer. It exists permanently and costs money every year, so its sponsor asks: is this capability worth what it costs, and is it getting better? No single cycle can answer that. It takes measures across releases, quarters and years, tied to the business case that funded the TCoE.

What changes when you measure a capability

The two sets are linked. A TCoE metric such as escaped defects per release is built from the same defect records that cycle reports use, plus production incidents linked back to releases. What changes is the aggregation and the decision. The overview course (sap-tst-401) introduced a balanced scorecard with three questions: are we protected, are we fast, are we efficient? This course turns those into a KPI system with definitions, data sources, owners, targets, governance and a value story finance will accept.

Start from goals, not from the tool's reports

The easiest way to pick TCoE metrics is to open the analytics in your test tool and report whatever it shows. That produces activity counts. A better way is the Goal-Question-Metric approach, which ISTQB's Advanced Test Management syllabus (CTAL-TM v3.0) recommends for analytical test process improvement: state the goal, ask the questions that tell you whether it is being met, then choose metrics that answer them, and only then find the data.

Normalize, or the trend lies

Capability measures compare periods that differ in size. A quarter with a feature pack stack update and 40 enhancements will produce more defects than a quiet quarter, whatever the TCoE does. Normalize: escaped defects per release event and per 100 changes, cost per regression cycle rather than total spend, coverage as a share of critical processes. At Contoso, a "change" is one approved change record that reaches production, whatever the number of transports it needed.

Lagging measures such as escaped defects prove value, but only after the damage. Leading measures such as coverage of critical processes or automation stability move first and give the TCoE time to act. A good KPI set has both. TMMi treats this as a process area of its own at level 4, Test Measurement, which is a reminder that measuring a test capability well is itself a sign of maturity.