Assessing the library

An inherited automation library is an asset only if it gives trustworthy answers at a reasonable cost. Many do not: scripts fail for reasons nobody investigates, cover processes that no longer exist, or need more maintenance than the manual tests they replaced. Assess the library with numbers, not impressions, and share the numbers with the people who will decide its future.

Four measures

Measure	How to get it	Healthy sign	Warning sign
Pass stability	Run each configuration three times on an unchanged system; compare logs	Same result every time	Results flip with no change: data, timing or environment problems
Business coverage	Map configurations to processes in the target process scope	Critical processes covered end to end	Many scripts on low-value or obsolete transactions
Maintenance cost	Hours spent repairing scripts per release over the last year	Small, predictable effort	Repairs exceed the effort of manual testing
Evidence quality	Read the logs: are business results checked?	CHEGUI/CHEVAR-style checks on documents and values	Scripts only replay steps and check that nothing crashed

Measuring an inherited automation library.

The Sign-Off Table · thesignofftable.com/academy/sap-tau-205

Reading the drivers

The driver mix predicts fragility. TCD scripts replay classic screens without controls and need no GUI; they are robust against GUI changes but break when the screen sequence of a transaction changes. SAPGUI scripts record control interactions and need a GUI for playback, so they are sensitive to controls, layouts and GUI versions. FUN scripts call function modules and are the most stable, but only as long as the function module's interface does not change. Scripts using external tool interfaces depend on a third-party tool that may no longer be licensed or supported.

Check the execution settings too. eCATT's start options include an execution error behavior, ranging from terminating the start process to No termination: continue with next script command. A library that runs with the most tolerant setting can report runs that carried on past failed steps, so read the logs rather than trusting a summary. The same applies to variants: a configuration may define dozens of variants while the regression plan only ever executed the default one, so the coverage on paper is far larger than the coverage actually tested.

With the most tolerant error behavior, a run carries on past a failed step, so only the log tells the truth.

The Sign-Off Table · thesignofftable.com/academy/sap-tau-205

The S/4HANA effect

For a library built on ECC, the conversion to SAP S/4HANA changes the picture. Some ECC transactions are replaced or redirected: customer and supplier master maintenance moves to the business partner (transaction BP), so TCD or SAPGUI scripts recorded on the old customer and vendor master transactions will no longer match. Many users will work in Fiori apps rather than the transactions the scripts recorded, which means even a green script may test a path nobody uses. And data model changes can alter screens and checks. Check every script against the simplification list items relevant to your scope.

Scripts recorded on ECC customer and vendor master transactions lose their target, because S/4HANA maintains both as business partners.

The Sign-Off Table · thesignofftable.com/academy/sap-tau-205

Low value, cheapKeep only if it runs unattended; otherwise retire.

Low value, expensiveRetire now. Repair effort buys nothing.

High value, cheapKeep temporarily and run; candidate for rebuild later.

High value, expensiveRebuild in the target approach as a priority.

Placing each script by value and maintenance cost.

The Sign-Off Table · thesignofftable.com/academy/sap-tau-205

Running the assessment

Export the list of test configurations with owner, last change, last execution and last result.
Run a stability sample: the 50 most important configurations, three times each, against an unchanged system.
Map each configuration to a process in the S/4HANA process scope; mark those with no target process.
Collect repair hours from the last four releases from the team's time records or tickets.
Read a sample of logs for real business checks, not just "transaction completed".
Place every configuration in the value and cost matrix and review it with process owners.

What would you do?

Kenji argues that because 300 eCATT configurations "still exist", they represent a large investment that must be preserved through the conversion. How should Priya respond?

Hands-on lab · ACCELQ

Measure stability and check strength on one Scenario

The lesson's assessment measures apply to any automation, including the replacements you'll build. Practice two of them, pass stability and evidence quality, on an ACCELQ Scenario so you know what healthy looks like before you judge an inherited eCATT library.

Practice on The pre-configured sample project in your ACCELQ trial (any sample Scenario that passes)

If you haven't set up ACCELQ yet, do the setup lab first. Pick one sample Scenario. Open each of its Actions and count the statements that verify something against those that only act (click, enter, select). This is the ACCELQ version of looking for CHEGUI or CHEVAR-style checks in an eCATT log.
Open Run and make sure Auto Re-Run under Smart Reconcile is not switched on for this exercise. An automatic rerun can hide a flaky first result.
Run the Scenario three times on your Local Agent with nothing changed between runs.
Compare the three Test Reports. The same result every time means the Scenario is stable. A flip with no change points to data, timing or environment, exactly as it would for an eCATT configuration.
Check each report for Self-Healing. A healed element passed, but it is still a maintenance signal: the designed identification no longer matched.
Open Scenario Insights for the Scenario to see its pass rate and flakiness over a 90-day window, the long-run version of your three-run sample.
Record runs, results, the verification count and any self-heals, then place the Scenario in the lesson's value and cost matrix.

Done when You have three Test Reports, a stability note and a verification count. A script with no business checks scores a warning on evidence quality, whatever tool it was built in.

Learn more in ACCELQ’s docs: Scenario Insights · Self-Healing · Setting up Auto Re-Run

Start the free ACCELQ trial Matt Angerer, who runs The Sign-Off Table, works at ACCELQ. Disclosure

Key points

Assess with four measures: stability, business coverage, maintenance cost and evidence quality.
The driver mix predicts fragility: FUN most stable, SAPGUI most sensitive to UI changes, external tools depend on third parties.
S/4HANA replaces some ECC paths (such as customer and vendor master to business partner), so many ECC scripts lose their target.
Place each script in a value and cost matrix and review it with process owners.
Decide on current value, not on how much the library cost to build.

Four measures

Reading the drivers

The S/4HANA effect

Running the assessment

Lesson check