Planning the regression window

Three weeks sounds like plenty until you subtract the first day (system checks and smoke tests), the last few days (fix, transport, retest and sign-off) and the people who are also closing the month. A good plan front-loads automation, saves human time for what only humans can judge, and leaves a buffer for defects. Build it before the window opens, not on day one.

Before the window opens

Put the test and production upgrade dates in the annual test calendar as soon as the schedule is published.
Run RASD and the What's New Viewer for the release; list changed apps and features in your active scope.
Update the regression pack: which processes run as automates, which as manual tests, which as key-user acceptance.
Check test users and data: virtual test users with current passwords and roles, data variants and test data containers that still match your master data.
Book key users, freeze non-urgent configuration changes in the window, and agree the defect triage rhythm.

Day 1Smoke and automation launchLog in, check critical apps, start full automated test plans
Days 2–4Triage automation resultsSeparate script maintenance from real defects; rerun after fixes
Days 3–8Manual and exploratory testsCustomized processes, new features, output and workflow
Days 6–11Integration and end-to-end testsInterfaces and connected systems, period-end steps
Days 9–13Fix, transport, retestCorrections imported into Test and regression rerun
Day 14 onwardReadiness report and sign-offLeaving buffer before the production weekend

A sample working-day plan for the Contoso what-if. Adjust the days to your window, which comes from SAP's schedule.

The Sign-Off Table · thesignofftable.com/academy/sap-tst-305

Automation first, people second

On day one, launch every automated test plan you have, including post-upgrade tests if you gave consent. Automates run overnight and give you a map of where to look. While they run, testers do the quick human checks: can key users log in, do the launchpad and the most-used apps open, do the critical outputs print? By day two you know whether failures are clustered in one area, which tells you where to put people.

Human time is the scarce resource. Spend it on what automates cannot judge well: new or changed screens, workflow and approval behavior, output layout, analytical apps and plausibility of numbers, and processes you have not automated yet. For Contoso, Maria would spend her days on the customized order-to-cash flow for Great Lakes Fleet Services and Dan's team on supplier invoice approvals and the journal entry custom field.

WatchLow criticality, high change: one manual pass on changed screens

Test deeplyHigh criticality, high change: manual plus automated, key-user acceptance

Automate and sampleLow criticality, low change: automated run only

Automate and confirmHigh criticality, low change: automated run plus a key-user spot check

Prioritizing regression in the window by criticality and by how much the release changed the process.

The Sign-Off Table · thesignofftable.com/academy/sap-tst-305

Defect triage in a short window

Triage at least daily. Every failure goes into one of four buckets: test asset (the automate, data or test user needs maintenance), configuration or extension (yours to fix and transport into Test), suspected SAP issue (report to SAP through the support channel with evidence), or expected change (the release changed behavior on purpose; update the test and inform users). Record defects in SAP Cloud ALM against the failed test step so the evidence and the retest stay together.

Every failure in the window lands in one of four buckets, and each bucket ends with a retest recorded against the failed step in SAP Cloud ALM.

The Sign-Off Table · thesignofftable.com/academy/sap-tst-305

Each late fix needs a full transport and retest loop, which is why the window must leave its last days free.

The Sign-Off Table · thesignofftable.com/academy/sap-tst-305

What would you do?

On day two of the what-if window, 14 of Contoso's 60 automated tests fail. Twelve fail on the same step in sales order entry, where a field moved. What should Priya do?

Hands-on lab · ACCELQ

Launch day-one regression across two App Environments

An App Environment lets one Test Suite run against different systems by swapping Global Property values, with no change to the tests. Here you practice this lesson's day-one move: launch the whole automated pack against the upgraded system, then read the failures as a map of where to send people.

Practice on SAPUI5 Shopping Cart demo app (public, no login), standing in for both your pre-upgrade and your upgraded test system

If you haven't set up ACCELQ yet, do the setup lab first.
Make sure you have at least three Scenarios on the Shopping Cart, for example browsing a category, searching for a product and adding an item to the cart. Record any that are missing with the Web Recorder, picking any category or product you see.
In Project Administration (click the project name at top right, then the Settings icon), open Application Environments. Use two existing environments, or add two if you are a Project Admin, and treat them as "Pre-upgrade" and "Upgraded".
In Resources, create a Global Property named CartURL and enter the Shopping Cart URL as its value for both App Environments. In a real window the two values would be your test system before and after SAP's upgrade.
In each Scenario, click the URL on the Initiate Browser Session step and point it to the CartURL Global Property, so no Action holds a URL.
Click "+" > Test Suite, choose Static, and add the Scenarios in the order you'd want them to run on day one.
Click Run, pick the suite, set App Environment to "Upgraded", choose your Local Agent and leave Self-Healing on under Smart Reconcile. Then run it again with "Pre-upgrade" as the baseline.
Open both Test Reports. Put every failure into one of this lesson's four buckets: test asset, your configuration or extension, suspected SAP issue, or expected change.

Done when The same suite ran twice with only the App Environment changed, and every failure carries a triage bucket. Keep both Test Report links as your day-one record.

Learn more in ACCELQ’s docs: Setting up Application Environments · Opening browser with dynamic URL in ACCELQ · Running Automation Test

Start the free ACCELQ trial Matt Angerer, who runs The Sign-Off Table, works at ACCELQ. Disclosure

Key points

Prepare before the window: dates, impact analysis, regression pack, users, data and people.
Launch automation on day one and use the results to point people where they are needed.
Spend human time on changed, customized, critical and unautomated processes.
Triage daily into test asset, your fix, suspected SAP issue or expected change; protect the last days for fix and retest.

Before the window opens

Automation first, people second

Defect triage in a short window

Lesson check