Lesson 2 of 5
Flakiness
18 min
Listen to this lesson
A flaky test is one that passes and fails on the same code, data and environment. It is the most corrosive problem in automation, because it trains people to ignore red results. Once the team says "it is probably just flaky", a real defect can sail through. The ISTQB Test Automation Engineering syllabus is direct about it: tests that do not give a reliable result should be moved out of the active suite and analyzed separately, otherwise time is spent repeatedly analyzing the same failures.
Measure it
You cannot reduce what you do not count. Track, per test, how often its outcome changed between runs when nothing relevant changed. A simple measure is the flip rate: the share of consecutive runs where the result flipped between pass and fail without a code, transport or data change. Another is the rerun pass rate: how often a failed test passes on immediate rerun. Both need run history, so keep results per test, not just per suite.
| Signal | What it tells you | Watch for |
|---|---|---|
| Flip rate per test | How unstable the test is over time | Tests above your threshold, for example 5% over 30 runs |
| Rerun pass rate | How much of the red is noise | A high rate means reruns are hiding problems |
| Failure signature spread | Whether failures share one cause | The same timeout message across many tests points to a shared helper or environment |
| Time-of-day pattern | Whether load or jobs interfere | Failures clustered when background jobs or other runs are active |
Common causes in SAP
- Synchronization. The script acts before the screen or service is ready. In Fiori, the app is still busy loading data; in SAP GUI, a dialog box appears later than expected.
- Asynchronous processing. SAP often saves a document and completes database updates in an update task after the commit. A test that reads the document from another session at once may not find it yet. Outputs, workflows, IDoc processing and background jobs are asynchronous too.
- Shared data. Parallel runs or people change the same records. See sap-tau-302 for pools and reservation.
- Order dependency. A test passes only if another test ran first and left data behind.
- Environment noise. Load from other teams, background jobs, locks held by other sessions, or a system restart during the run.
Fix synchronization properly
The ISTQB syllabus describes three wait mechanisms. Hard-coded waits, such as "wait five seconds", are a root cause of many automation defects because response times vary. Dynamic waiting by polling for a condition is better: the test waits only as long as needed, with a timeout so it cannot hang. Best of all is subscribing to the system's own events or readiness signals, when the tool and the application support it. UI5's own test frameworks follow the polling idea: OPA5 actions wait until the app is not busy before acting.
- Hard-coded waitSleep a fixed time. Too short fails under load; too long wastes hours per run.
- Poll for a conditionCheck for the expected state every short interval, with a timeout.
- Wait for a readiness signalUse the framework's own idle or busy state, or the application's event, with a timeout.
- Assert the business stateRead the document back through a service until it exists or the timeout expires.
Quarantine, then fix
- Has the test flipped more than the threshold in recent runs?Yes → quarantine: run it, but report it separately · No → keep in active suite
- Does the failure share a signature with other tests?Yes → fix the shared helper or environment first · No → analyze this test alone
- Is the cause timing, data, order or environment?Timing → polling or readiness · Data → pool or self-contained setup · Order → remove dependency · Environment → raise with Basis
- Has it passed consistently after the fix?Yes → return to the active suite · No → repeat analysis or redesign the test
Quarantine is not deletion. The quarantined test keeps running so you collect evidence, but its result does not block a release or fill the defect queue. Set a time limit: a test that stays in quarantine for more than a release or two either gets fixed or goes to the pruning review in lesson 5.
What would you do?
Priya notices that 22 tests fail intermittently, all with the same "element not found" message on the Fiori launchpad home page, mostly during the morning run. What should she do first?
- A flaky test passes and fails with no relevant change; it destroys trust in automation.
- Measure flip rate and rerun pass rate per test from run history.
- In SAP, look for synchronization, asynchronous updates, shared data, order dependency and environment noise.
- Replace fixed waits with polling or readiness signals, always with a timeout.
- Quarantine flaky tests with a time limit; count and report reruns.
Check your understanding · not graded
Lesson check
1. A test passes and fails intermittently with no code, transport or data change. What is it?
2. Which wait mechanism does the ISTQB syllabus identify as a frequent root cause of automation defects?
3. Why might a test fail to find a document it just saved in SAP?
4. What is the correct status for a quarantined test?