Lesson 5 of 7
The automation portfolio
9 min
Listen to this lesson
This course discusses ACCELQ by name. Matt Angerer, who runs The Sign-Off Table, works at ACCELQ. Disclosure
A tool decides how you automate; the portfolio decides what. sap-tst-401 introduced the rule of thumb (automate stable, frequent, high-risk processes first). This lesson turns it into a scoring model, a layered portfolio and a target for Contoso's regression pack.
Scoring each regression test
| Factor | Question | Score 1 | Score 5 | Weight |
|---|---|---|---|---|
| Business risk | What does failure cost? | Cosmetic or internal | Revenue, close or compliance | 30% |
| Run frequency | How often would it run? | Once a year | Every release and nightly | 25% |
| Stability | How often do the screens and flow change? | Every release | Rarely | 20% |
| Data effort | How hard is data to set up and reuse? | Manual, consumed each run | Recipe exists, repeatable | 15% |
| Automatability | Can the tool drive it reliably? | Unproven technology | Proven in the PoC | 10% |
Contoso's scoring model. Tests above 3.5 enter the automation backlog; below 2.5 stay manual.
Stability and data effort are the factors teams underweight. A high-risk process that changes every release costs more to keep automated than to test by hand, and a test that consumes its data cannot run nightly. CT-TAS makes the same point from the economics side: if a project or support period is shorter than the point where automation turns positive, it is not worth automating.
Here is the model applied to four tests from Contoso's pack. The weighted score is risk × 0.30 + frequency × 0.25 + stability × 0.20 + data × 0.15 + automatability × 0.10. Two tests with the same business risk land on opposite sides of the line because of stability and data.
| Test | Risk | Freq. | Stab. | Data | Auto. | Score | Decision |
|---|---|---|---|---|---|---|---|
| O2C-010 Order to billing, CMC-ALT-100, 1710/10/00 | 5 | 5 | 5 | 4 | 5 | 4.85 | Automate first (end-to-end and UI) |
| P2P-020 Goods receipt for Lakeside PO, SAP GUI MIGO | 4 | 5 | 5 | 4 | 4 | 4.45 | Automate (SAP GUI) |
| O2C-310 Order to billing with CFDI, company code 1910 | 5 | 5 | 1 | 2 | 3 | 3.55 | Borderline: automate API checks only; rescore after Wave 2 settles |
| R2R-090 Year-end balance carryforward | 5 | 1 | 4 | 2 | 3 | 3.15 | Manual with a checklist |
Worked scores for four Contoso regression tests (illustrative course data). Tests above 3.5 enter the backlog; the borderline case is a judgment call the architect records.
Layers: test at the lowest level that proves the point
An SAP end-to-end UI test is slow and has many reasons to fail. Many checks can be proven lower down. Contoso's portfolio has three layers: API and service checks (OData reads, IDoc processing results, pricing via API) that are fast and stable; process UI tests for a single app or transaction; and a smaller set of end-to-end flows across apps, systems and BTP that prove the business process as a whole. Code-level tests (ABAP Unit, CAP and UI5 unit tests) sit beneath all three and belong to development.
Contoso's target
Contoso's risk-ranked regression pack has about 400 test cases. Scoring put 240 (60%) into the automation backlog, matching the automation share in the TCoE scorecard: about 150 web tests (Fiori apps, WebGUI transactions and the BTP extension), 60 SAP GUI for Windows tests and 30 API checks. The remaining 160 stay manual: exploratory checks, rarely run year-end steps, and processes still changing after Wave 2. The backlog is built in order of score, one process at a time, so each process becomes fully automated and starts paying back before the next one starts.
What would you do?
Ravi's backlog puts the Wave 2 order-to-cash flow for company code 1910 near the top, because it is high-risk and runs every release. Lucía, the Monterrey O2C key user, says CFDI output and pricing for 1910 are still changing every sprint. How should the 1910 flow be scored and handled?
- Score each regression test on risk, frequency, stability, data effort and automatability; automate from the top.
- Stability and data effort are the factors teams most often underweight.
- Push each check to the lowest layer that proves it: API and service, then process UI, then end-to-end.
- Contoso targets 240 of about 400 regression tests (60%), automated process by process.
Check your understanding · not graded
Lesson check
1. Which two scoring factors do teams most often underweight?
2. Why push a check to the lowest layer that proves it?
3. How many of Contoso's roughly 400 regression tests are in the automation backlog?
4. A high-risk process runs every release but its screens change every sprint. What does the scoring model suggest?