Lesson 7 of 7
Decision and rollout: pilot, metrics and sign-off
16 min
Listen to this lesson
This course discusses ACCELQ by name. Matt Angerer, who runs The Sign-Off Table, works at ACCELQ. Disclosure
Opinions about AI and test platforms are cheap. A pilot on your own scope, in your own regression environment, through a real change event, is what settles the question. Contoso runs a post-go-live regression pilot in November 2026, three months after go-live, so the chosen approach is in place before the first feature pack stack or release upgrade. This lesson gives Contoso's pilot design, the metrics that matter, a decision path and the checklist Priya uses before the steering group signs off on an approach.
Pilot design
- Same scope on both tracks. 24 scenarios: 12 Fiori order-to-cash and procure-to-pay flows, 8 SAP GUI transactions (VA01, VL02N, MIGO, FB60) and 4 OData checks. Track A uses Claude Code and Playwright (plus a GUI Scripting approach for the GUI tests); track B uses the platform, with Claude Code allowed as an assistant.
- Same environment and data. CMR client 100, the masked pre-production copy of CMP where release regression runs after go-live; the same test users per business role, the same data plan.
- A real change event. Run both suites before and after a support package or a release's transports into CMR, so maintenance is measured, not estimated.
- Three two-week sprints, the cadence Contoso kept from Realize: build, run through the change, then evidence and decision.
- Independent measurement. Grace's PMO logs hours from timesheets; the teams don't grade themselves.
Success metrics
| Metric | How to measure it | Why it matters |
|---|---|---|
| Build hours per scenario | Timesheets per track, including review | Entry cost of new coverage |
| Repair hours per change event | Hours to green after the support package or transports | The biggest driver of 3–5-year cost |
| Flaky rate | Failures that don't reproduce on rerun, per 100 runs | Trust in red results |
| Time to sign-off-ready evidence | Hours from run end to an evidence pack Dan and Maria accept | Cost of every test cycle |
| Key-user review coverage | Share of scenarios a key user could read and approve without an engineer | Who can own and challenge tests |
| SAP GUI coverage | Share of the 8 GUI transactions automated and stable | Reach across Contoso's real scope |
| Maintainers | Number of people who could fix a failing test | Key-person risk |
| Security findings | Findings from Nadia's review of data flow, secrets and access | Whether the approach can pass GRC |
Pilot metrics, measured on both tracks the same way.
Decision path
Rollout
- Nov–Dec 2026PilotBoth tracks on 24 scenarios in CMR through a real change event.
- Next quarterFoundationShared Actions or page objects for log-on, navigation and common steps; environments for sandbox, CMQ and CMR; guardrails in settings and the pipeline.
- Before the first feature pack stackScaleRegression suites by priority, CI Jobs on every transport import to CMQ, a regression window in CMR, evidence linked to SAP Cloud ALM.
- Every release afterOperateThe TCoE owns the suite through each feature pack and upgrade, with maintenance hours tracked per change event.
- Scope covers Fiori, WebGUI if used, SAP GUI for Windows and APIs in the real regression set.
- Every test traces to an approved test case and requirement in SAP Cloud ALM.
- Results record system, client, user, data and time, and changes to tests are versioned.
- AI-assisted assets are labeled and have a named human reviewer.
- No production data or credentials were sent to a model; masked data only.
- Prompt injection risks are addressed by least privilege and isolation, not by prompt wording.
- The 3–5-year cost model uses measured pilot ratios.
What would you do?
The pilot ends. Track A (code-first) built faster but needed 31 repair hours after the support package and only engineers could review it; track B (platform) took longer to set up, needed 9 repair hours, and Maria and Dan reviewed its evidence. One engineer argues track A "won on speed". What should Priya recommend to Grace?
- Pilot both approaches on the same scope, environment and a real change event.
- Measure repair hours, flaky rate, evidence time and key-user review coverage, not just build speed.
- Decide on GUI reach, reviewability, evidence acceptance and measured five-year cost.
- Roll out through a TCoE that owns the suite and tracks maintenance per change event.
Check your understanding · not graded
Lesson check
1. Why does the pilot run both tracks through a real change event?
2. Which pilot metric best shows whether process owners can own and challenge the tests?
3. Track A built faster but needed three times the repair hours after the support package. What does that suggest for a five-year view?
4. Who should log the hours for both pilot tracks, and why?