Lesson 4 of 6
Criteria, metrics and reporting
13 min
Listen to this lesson
Entry and exit criteria are where a strategy earns its keep. They turn "are we ready?" from an opinion into a check. Metrics are the evidence that the criteria are met, and reporting is how that evidence reaches the people who decide.
Entry and exit criteria
Each level needs entry criteria (what must be true to start) and exit criteria (what must be true to finish). Write them so they can be checked from data. "Environment ready" is not checkable; "transport baseline for wave 3 imported to QAS client 100 and smoke test passed" is.
| Level | Entry criteria (examples) | Exit criteria (examples) |
|---|---|---|
| SIT | Unit and string tests passed for in-scope processes; test cases Prepared and linked in Cloud ALM; data load cycle 2 reconciled; interfaces connected; testers have roles | 100% of planned test cases executed; 95% passed; no open S1; S2 only with approved workaround; all closed defects retested |
| UAT | SIT exit met or exceptions accepted; key users trained; UAT test cases reviewed by process owners | All UAT test cases executed; process owners accept results; open defects listed with owners and dates |
| Data migration | Mapping signed off; load programs unit-tested | Reconciliation within agreed tolerance; business data owners sign |
| Regression (Run) | Upgrade or change applied in test system; regression plan Prepared | Regression pack executed; no new S1 or S2; differences explained |
Example criteria for Contoso. Thresholds are program decisions, not universal rules.
Criteria and quality gates
Link each level's exit criteria to the SAP Activate quality gate that relies on it. SIT and UAT exit feed the end-of-Realize gate; cutover rehearsal checks and OAT feed the go-live decision in Deploy. In SAP Cloud ALM, put the criteria into the quality gate checklists so each gate asks the same questions and stores the answers.
- SIT exitCriteria checked from Cloud ALM data
- UAT exitProcess owners accept their processes
- End-of-Realize gateUses SIT and UAT exit results
- Rehearsal checks and OATCutover runs on time; solution is operable
- Go-live decisionTaken in Deploy on the evidence above
- Were all planned test cases executed or formally descoped?Yes → check results · No → not ready; list the gap and its owner
- Are there open S1 defects?No → check S2 · Yes → not ready unless the decision owner accepts the risk in writing
- Do all S2 defects have an approved workaround and target date?Yes → check critical processes · No → not ready
- Did every named critical process pass end to end?Yes → recommend exit · No → not ready for that process
Metrics that evidence the criteria
| Metric | What it shows | Source at Contoso |
|---|---|---|
| Execution progress | Executed versus planned, by plan and process | Cloud ALM Test Execution Analysis |
| Pass rate by process | Where quality is good or poor | Test Execution Analysis by scope |
| Open defects by severity and age | Risk and fixing speed | Defect reporting |
| Retest closure | Defects closed only after a passed retest | Defects linked to passed runs |
| Requirement coverage | Approved requirements with passed tests | Requirement Traceability |
| Process readiness | Processes with requirements confirmed and tests passed | Solution Process Traceability |
Metrics mapped to the criteria they evidence.
Choose a small set. Every metric should evidence a criterion or flag a risk early. Defect age is a good early warning: S2 defects older than five days usually mean the fix capacity is short, long before the pass rate shows it.
Reporting cadence and audiences
| Audience | Cadence | Content |
|---|---|---|
| Test team and build leads | Daily during cycles | Blocked tests, new S1 and S2 defects, retests due |
| Program leadership (Grace) | Weekly | Progress against plan, risks, decisions needed |
| Steering committee | Monthly and at gates | Readiness by process, criteria status, recommendation |
| Process owners (Dan, Maria) | At UAT start and exit | Results for their processes, open defects and workarounds |
Reporting plan.
The test status report and test completion report in ISO/IEC/IEEE 29119-3 correspond to the weekly report and the cycle exit report. A completion report should state the criteria, the actual results, the deviations and the recommendation, and it should be the document the gate decision refers to.
What would you do?
At Contoso's SIT 2 exit, the pass rate is 93% against a 95% threshold. The failures are three S3 defects in output forms and one S2 defect in fleet pricing with a manual workaround. The program director wants to "round up". What should Priya recommend?
- Write checkable entry and exit criteria per level and link them to the quality gates they feed.
- Pair percentage thresholds with severity rules and named critical processes.
- Use a small set of metrics that evidence criteria, from Cloud ALM analytics where possible.
- Set reporting cadence by audience; agree criteria before each cycle, and record any exception as a decision.
Check your understanding · not graded
Lesson check
1. Which is a checkable entry criterion?
2. Why pair a pass-rate threshold with severity rules?
3. Which metric gives early warning of short fix capacity?
4. When should entry and exit criteria be agreed?