Lesson 5 of 5
Honest value
9 min
Listen to this lesson
Automation business cases are often written backwards: count the hours a manual run takes, multiply by the number of runs, call the difference savings. That ignores what automation really costs over its life. An honest view includes building the tests, maintaining them through every change, running them, and investigating every failure, including the ones that turn out to be nothing.
The full cost picture
The ISTQB syllabus lists the disadvantages plainly: additional cost, an initial investment, time to develop and maintain, rigidity when the system under test changes, and the possibility of defects in the automation itself. In SAP, maintenance dominates because the UI and processes keep changing: a Feature Pack Stack that rearranges a Fiori app, or a release upgrade that replaces a transaction, can break dozens of scripts at once.
Flakiness is a cost, not an annoyance
A flaky test is one that sometimes passes and sometimes fails without any change to the system. Each flaky failure costs triage time, and worse, it teaches people to ignore red results. Once the team says "just rerun it", a real defect can hide among the noise. Treat flakiness as a defect in the test suite: track it, fix the cause (usually data, timing or identification) or remove the test until it is fixed.
A simple, honest calculation
| Item | Contoso order-to-cash batch (illustrative) |
|---|---|
| Manual effort per run | 3 person-days for 40 tests |
| Runs per year | 26 (every second week) plus 2 upgrade cycles = 28 |
| Manual cost per year | 84 person-days |
| Build effort | 30 person-days, once |
| Maintenance per year | 20 person-days (UI changes, two feature packs, data fixes) |
| Triage per year | 0.25 person-days per run = 7 person-days |
| Automated cost, year 1 | 30 + 20 + 7 = 57 person-days |
| Automated cost, later years | 27 person-days |
An illustrative calculation with made-up numbers. Use your own measured values.
In this example automation pays back in year one, but only because the batch runs often. If the same tests ran twice a year, the build and maintenance cost would never be recovered. That is why frequency is the first selection criterion. The calculation also shows where to look when value disappoints: if maintenance or triage grows, the suite is too fragile or too high in the pyramid.
- Is maintenance effort growing release after release?Yes → move checks down the pyramid; centralize locators; retire low-value tests · No → next question
- Are more than a few percent of failures flaky?Yes → stop adding tests and fix data, waits and identification first · No → next question
- Do the automated tests cover the top business risks?No → re-score candidates; value is measured by risk covered, not test count · Yes → keep expanding in small batches
Measuring value honestly
- Risk coverage: share of high-risk processes with automated regression checks.
- Cycle time: time from a transport batch to regression evidence.
- Defects found: real defects caught by automated runs before production.
- Maintenance ratio: effort spent fixing tests versus building new ones.
- Flaky rate: failures with no system cause, as a share of all failures.
What would you do?
Twelve months in, Contoso's automation suite has 300 UI scripts. Maintenance now takes more time than building, and a third of failures are reruns that pass. Grace asks Priya what to do. What is the best response?
- Lifetime cost includes build, maintenance, runs, triage and skills.
- In SAP, maintenance after UI changes and upgrades usually dominates.
- Flaky tests cost triage time and erode trust; treat them as defects.
- Measure value by risk coverage, cycle time and defects caught, not by test count.
Check your understanding · not graded
Lesson check
1. Which cost is most often left out of SAP automation business cases?
2. What is a flaky test?
3. Why does run frequency matter so much in the value calculation?
4. Which metric best shows whether automation reduces business risk?