Lesson 7 of 7
Governing automation as a shared service
16 min
Listen to this lesson
This course discusses ACCELQ by name. Matt Angerer, who runs The Sign-Off Table, works at ACCELQ. Disclosure
A tool and a portfolio do not keep themselves healthy. Without governance, automation suites decay into a nightly run of red results that nobody believes. This lesson sets up automation as a TCoE service with an owner, a definition of done, pipeline gates, health metrics, and rules for the tool and the vendor across its lifecycle.
Ownership
| Asset or decision | Accountable | Responsible | Consulted |
|---|---|---|---|
| Test design (what is checked) | Test architect for the process | Test analysts | Key users |
| Automated implementation | Automation lead (Ravi) | Automation engineers, partner capacity | Test architect |
| Shared Actions, environments and agents | Automation lead | Platform administrator | Basis (Sam), security (Nadia) |
| Automation backlog order | TCoE lead (Priya) | Automation lead | Process owners |
| Tool standard and vendor relationship | CIO via the quality council | TCoE lead | Procurement, security |
Automation ownership at Contoso. Test design stays with the process test architect, whatever tool implements it.
Definition of done for an automated test
- Named to the TCoE convention and linked to its manual test case in SAP Cloud ALM.
- Data-driven, using a data recipe that does not depend on a previous run's leftovers.
- Environment-independent: system URLs and users come from environment settings, passwords are encrypted.
- Built from shared Actions where they exist; new shared Actions reviewed by the automation lead.
- Passes five consecutive unchanged runs before joining the regression suite.
- Reviewed by a key user, who confirms in writing that it checks what the manual test checks.
- Has a named owner and appears in the health dashboard.
Pipelines and gates
Automation earns its keep when it runs without anyone remembering to start it. Contoso wires three triggers: an import of a transport into CMQ starts a smoke suite of about 30 critical tests; a feature pack stack update or release bundle starts the full suite in CMR after its quarterly refresh; and a nightly run covers the processes changed that week. Results feed the readiness assessment and SAP Cloud ALM. sap-tau-304 covers the CI/CD mechanics; the governance point is that each trigger has an owner who reads the results the next morning.
Health metrics
| Metric | Target | Why it matters |
|---|---|---|
| False-failure rate | Under 5% of failures | Above it, people stop believing red results |
| Maintenance hours per change event | Trending down; compare with the TCO assumption | Tests the core assumption of the business case |
| Self-heal events reviewed | 100% reviewed within a week | An unreviewed self-heal can hide a real change |
| Quarantined tests | Under 3% of the suite, none older than one release | Quarantine is a hospital, not a graveyard |
| Critical-process automated coverage | Rising toward the portfolio target | Shows the portfolio is being delivered |
| Cycle time from build to results | Within the regression service level | Connects automation to the TCoE's promise |
Automation health metrics, reviewed monthly by the automation lead and quarterly at the quality council.
The tool across its lifecycle
CTAL-TM v3.0 describes four stages in a tool's life: acquisition, support and maintenance, evolution, and retirement, with a named tool owner and administrator throughout. For a platform, governance adds three habits: a quarterly vendor review against the health metrics and roadmap, an annual check of the TCO assumptions against actuals, and an exit plan kept current. The exit plan is simple if the TCoE standard has been followed: test design lives in SAP Cloud ALM, results and evidence are exported and retained, and the platform holds only the implementation.
- AcquisitionSelection, PoC, pilotOwner and administrator named; standards written
- SupportSteady stateHealth metrics monthly; vendor review quarterly
- EvolutionSAP or vendor changeRe-baseline maintenance; update standards
- RetirementMoving on pays offTest design kept in Cloud ALM; history retained
What would you do?
Three months after go-live, the nightly suite shows 18 red results every morning, 15 of them script problems from a UI5 update. Key users have stopped looking at the dashboard. Luis asks whether automated results can still be used in Friday's readiness assessment. What should Ravi and Priya do?
- Test design stays with the process test architect; implementation, shared assets and agents with the automation lead.
- Use a definition of done: naming, data recipe, environment independence, five green runs, key-user review, owner.
- Wire triggers to transports, release events and nightly runs, and give each an owner who triages every morning.
- Track false failures, maintenance hours, self-heal reviews, quarantine and coverage; review the vendor quarterly and keep an exit plan.
Check your understanding · not graded
Lesson check
1. Who stays accountable for what an automated test checks?
2. How should a quarantined test appear in a readiness assessment?
3. Why review every self-heal event?
4. What makes the exit plan for a platform simple?