Lesson 5 of 5
Delivery metrics
14 min
How do you know whether the move to trains, gates and gCTS is working? Not by counting meetings saved. The DORA research program, which studies software delivery performance, defines a small set of metrics that balance speed against stability. They apply to SAP with some translation.
The metrics
| Metric | DORA definition (summarized) | Group |
|---|---|---|
| Change lead time | Time from a change being committed to version control to being deployed in production | Throughput |
| Deployment frequency | Number of deployments over a period, or time between deployments | Throughput |
| Failed deployment recovery time | Time to recover from a deployment that fails and needs immediate intervention | Throughput |
| Change fail rate | Share of deployments that need immediate intervention afterwards | Instability |
| Deployment rework rate | Share of deployments that are unplanned and happen because of a production incident | Instability |
The DORA delivery metrics as currently published on dora.dev.
The point is balance. Throughput alone rewards reckless speed; stability alone rewards never changing anything. Looking at both shows whether a team is getting faster without getting worse, or safer without grinding to a halt. DORA's guidance also warns against turning the metrics into targets, because teams will game them.
Translating to SAP
| Metric | How Contoso measures it | Data source |
|---|---|---|
| Change lead time | From transport release in CMD (or commit with gCTS) to import into CMP | Transport logs or Git history plus Feature Traceability |
| Deployment frequency | Production deployments per month: trains plus urgent changes | Cloud ALM releases and deployed features |
| Failed deployment recovery time | From detection of a failed deployment to restored service | Incident records linked to features |
| Change fail rate | Production deployments followed by an urgent fix or fallback within 14 days | Urgent features linked to the train they fixed |
| Deployment rework rate | Share of production deployments that are urgent changes caused by incidents | Urgent-tagged features divided by all deployments |
SAP-specific definitions. Write yours down so the numbers mean the same thing every month.
- Requirement approvedNot part of lead time in DORA’s definition, but worth tracking as wait time
- Transport released or commit pushedLead time starts
- Gates and testATC, ABAP Unit, feature tests, regression
- Wait for trainOften the largest share of lead time
- Production importLead time ends
Reading the numbers
Contoso's first measurement, three months after moving to monthly trains: median lead time 19 days (down from 55 with quarterly releases), 1.3 production deployments a week including urgent ones, change fail rate 9% and falling, recovery time median six hours. Most of the lead time was waiting for the train, not testing. That is the expected shape for a train model, and the lever is cadence, which in turn depends on regression automation.
Collecting the data
Most of the data already exists. Cloud ALM knows when features were deployed and which release they belonged to; transport logs and Git history know when changes were released or committed; urgent features are tagged; incidents are linked to features in your service management process. The work is in joining them consistently. Start with a monthly extract maintained by one person, with written definitions, before building a dashboard. A dashboard of numbers nobody trusts is worse than a spreadsheet everybody understands.
- Month 0BaselineMeasure the last two quarterly releases with the new definitions
- Month 1–3First trainsMonthly extract; trend reviewed at each post-train review
- Month 4Constraint foundWaiting for the train dominates lead time; regression automation prioritized
- Month 6Cadence changeTwo-week train for low-risk tracks once regression runs in one day
Pitfalls
- Comparing teams with the metrics: DORA warns the aim is to improve over time, not to compete.
- Counting transports as deployments: one train with 200 transports is one deployment.
- Ignoring urgent changes: they are deployments too, and they drive rework rate.
- Measuring only ABAP: BTP and integration changes are part of the same delivery.
- Using averages: one stuck change can distort an average; use medians and look at outliers.
What would you do?
The CIO wants to set a target: "lead time under 7 days for every team by next quarter". What should Luis advise?
- DORA metrics: change lead time, deployment frequency, failed deployment recovery time, change fail rate, deployment rework rate.
- Balance throughput and stability; neither alone tells the story.
- Define SAP-specific measurement points and data sources once and keep them stable.
- Use metrics as trends to find constraints, not as targets or team rankings.
Check your understanding · not graded
Lesson check
1. Which metric indicates how often releases cause production problems?
2. How does DORA define change lead time?
3. Why does DORA warn against setting the metrics as goals?
4. How should a train with 200 transports be counted for deployment frequency?