Now accepting guest editorial from SAP, Oracle, Workday and ServiceNow practitioners.Pitch a guide
The Sign-Off Table

Lesson 5 of 6

Control and regression evidence for AI-made fixes

9 min

Listen to this lesson

0:00 / 11:58

An ATC run with no findings proves that the code no longer uses simplified objects in ways the checks can see. It doesn't prove that the code does what it did before. For AI-made fixes, that gap is the whole question. This lesson sets the controls around an agent run and the regression evidence Priya needs before she signs custom code as ready.

Controls around a run

  • Where: the agent runs in the sandbox or CMD, never in CMQ or later systems. Fixes travel by transport and are applied in each conversion cycle.
  • What: one package per run, one dedicated request, ADT MCP server access restricted to that package.
  • Who: the developer who starts the run owns its output; the package owner reviews high-risk objects.
  • Diff: the unified diff of the request (the abap_transport_get_unified_diff tool, or a git pull request as in sap-aid-204) is the review artifact.
  • Gate: no request is released until the review is complete, ATC is clean for priority 1 and 2, and the regression evidence below exists for the object's risk class.

Risk classes keep the evidence affordable. Contoso uses three. Class 1 (frequent use, high criticality: order-to-cash, finance close, interfaces) needs before-and-after comparison, characterization tests for central classes, and the owner's review. Class 2 (moderate use or criticality) needs a functional test of the main path in the converted system and normal review. Class 3 (rare use, low criticality) needs the ATC recheck, a smoke test and a reviewer who has read the diff. AI-made fixes never drop below class 2 if they change data access, whatever the object's usage.

  1. Baseline in ECCBefore the run, capture outputs of key programs in a copy of EP1 with fixed test data
  2. Agent runATC, quick fixes, AI fixes in the sandbox or CMD, one request
  3. ReviewUnified diff, comments resolved, ATC recheck, activation
  4. CompareRun the same programs with the same data in the converted system; compare results
  5. Explain differencesEvery difference is expected (documented S/4 change) or a defect
  6. Sign offEvidence attached to the request; Priya signs by risk class
Before-and-after evidence: capture, fix, compare, explain, sign.
The Sign-Off Table · thesignofftable.com/academy/sap-aid-301

Before-and-after comparison

The most convincing evidence for a remediated report or interface is simple: same input, same output. Run the program in a system that still has ECC behavior (a copy of EP1, or ED1 with representative data) and in the converted system with the same selection, then compare. For Contoso's warranty accrual program, that means the same posting period, the same company code 1710 and the same claims, with totals compared to the cent. Differences aren't automatically defects: S/4HANA changed some data on purpose, for example status fields that moved from VBUK to VBAK. Each difference must be explained, and the explanation reviewed.

Before-and-after comparison is the core evidence that a remediated object still behaves the same.
The Sign-Off Table · thesignofftable.com/academy/sap-aid-301

ABAP Unit as a safety net

Most legacy Z code has no unit tests, and writing them for 7,500 objects isn't realistic. For the high-risk box of the matrix, it is. Before the agent touches a critical class or function module, write characterization tests: tests that record what the code currently returns for chosen inputs, right or wrong. After the fix, they must still pass, or the difference must be explained. Joule for Developers can generate ABAP Unit tests, which speeds this up; the same review rules apply as for any generated test: read the assertions, and break the code to prove a test fails.

What static checks miss

GapExampleHow to cover it
Dynamic codeTable names built at runtime, dynamic method callsCode review plus targeted tests; search for dynamic statements in remediated objects
Runtime behaviorEmpty table in FOR ALL ENTRIES, missing ORDER BYRuntime Check Monitor (SRTCM) in the new system, as SAP's guide recommends
PerformanceA fix that turns a fast read into a slow join on ACDOCASQL Monitor (SQLM) and timing in the comparison runs
Data differencesStatus or amounts read from new placesBefore-and-after comparison with explained differences
InterfacesField lengths in IDocs or files after MATNR extensionInterface tests with partners or stubs; see Contoso's interface regression pack

Where AI-made fixes need evidence beyond a clean ATC run.

What would you do?

The comparison for the warranty accrual program shows totals 0.4% lower in the converted sandbox. The agent's fix replaced a KONV read. The developer says "rounding, probably" and wants to release. What do you do?

Key points
  • A clean ATC run isn't proof of unchanged behavior; regression evidence is.
  • Run agents only in sandbox or CMD, one package and one request at a time, with MCP access restricted to the package.
  • Before-and-after comparisons with explained differences are the core evidence for used, critical objects.
  • Characterization tests protect critical code; SRTCM and SQLM cover runtime and performance gaps.

Check your understanding · not graded

Lesson check

  1. 1. What does a clean ATC run after an AI fix prove?

  2. 2. What is a characterization test?

  3. 3. Which SAP tool does the guide recommend for runtime checks after go-live?

  4. 4. A before-and-after comparison shows a difference. When is it acceptable?