Reporting AI-change quality for sign-off

At release sign-off, Luis and Priya need to know two things about AI-assisted changes: did they pass the same gates as everything else, and is AI assistance making quality better or worse over time? Neither question needs a separate process, just a few fields recorded consistently and a small set of measures.

Record AI assistance where the change lives

Mark AI assistance in the place reviewers already look. For ABAP, Contoso adds a line to the transport request description ("AI-assisted: Joule for Developers" or "AI-assisted: coding agent") and the same tag on the pull request for BTP and integration content. Record which tool, not which prompt; the point is to compare populations of changes, not to audit every keystroke.

Measure	How it is computed	What it tells you
AI-assisted share	AI-tagged changes ÷ all changes in the release	How much of the release the measures cover
First-pass gate rate	Changes passing all independent gates on first submission	Whether in-loop checks work; low means the loop is leaking
Findings per change	ATC findings by priority, lint errors, failed tests at the release gate	Quality arriving at the gate, compared with non-AI changes
Suppressions and exemptions	New pragmas, pseudo-comments, skips and exemption requests per change	Whether gates are being bypassed rather than met
Test changes	Tests added, modified or deleted per change	Whether coverage grows honestly
Review effort	Reviewer time and rework rounds per change	Where the cost moved
Escaped defects	Defects found in CMQ or production traced to a change	The outcome that matters, by population

Seven measures for AI-assisted changes, each compared with the non-AI baseline.

Compare AI-assisted and other changes on the same measures. If AI-assisted changes pass gates more often but cause more escaped defects, the gates are being satisfied without the code being right, and review needs to focus on intent and test meaning. If both look the same, AI assistance is neutral on quality and the question becomes cost.

Every changeGates and tagsAgent self-checks, independent gates, AI tag on transport or pull request
Every sprintMeasures refreshedKenji and Theo review first-pass rate, suppressions and test changes
Release readinessEvidence pack sectionAI-assisted share, gate results, open exemptions and review notes for the release
Go/no-goSign-offLuis confirms every change passed the same gates; Priya reports escaped defects by population
After releaseEscapes tracedDefects in production linked back to changes and their AI tag

When AI-change quality data is produced and used across a release.

The Sign-Off Table · thesignofftable.com/academy/sap-aiq-205

AI-change evidence flows from tags, gate results, reviews and traced defects into the release pack.

The Sign-Off Table · thesignofftable.com/academy/sap-aiq-205

Keep the evidence where it already lives. ATC results stay with the transport, CAP and UI5 gate results in the pipeline's run history, mapping simulation outputs and payload regression reports with the integration package, and regression test reports in the test tool. The evidence pack links to them, so nobody copies results into slides that drift from the source.

What would you do?

At release readiness, the sprint measures show AI-assisted ABAP changes with a 92% first-pass gate rate against 78% for other changes, but also 11 new pseudo-comments that silence ATC findings, all in AI-assisted changes, and none mentioned in review notes. Luis asks whether the release can go. What should Priya recommend?

Hands-on lab · ACCELQ

Export regression evidence for an AI-change release pack

Release evidence should link to the original results, not copies. Here you rerun a regression Scenario after a change, export the results, and file the export and Test Report link the way the release pack expects. It uses Run, the Test Report and result export.

Practice on Your payload regression Scenario from lesson 4, against the TripPin OData V4 reference service (public, read-only, no login)

If you haven't set up ACCELQ yet, do the setup lab first.
Treat your Library Action as the "AI-assisted change": edit one statement, for example add a verification of a fourth field, and note the change as you would on a transport (what changed, AI-assisted or not).
Click Run, set the Purpose to something a reviewer can read, such as Release R12 regression after AI-assisted change, and run the Scenario on your Local Agent.
Open the Test Report. Check that the information sidebar shows the purpose, environment and agent, and that each Test Case shows its verifications.
From Results, export the run's results (the label may differ slightly in your version) and save the file with the release name.
Write a three-line evidence entry: the change note, the Test Report link and the exported file name, plus whether any verification was changed or removed in the change.

Done when An evidence entry that links the change note, the Test Report and the export, with an explicit statement that no verification was removed. Keep all three together for your release pack.

Learn more in ACCELQ’s docs: Exporting Test Results · Overview of Test Report

Start the free ACCELQ trial Matt Angerer, who runs The Sign-Off Table, works at ACCELQ. Disclosure

Key points

Tag AI-assisted changes on the transport or pull request; record the tool, not the prompt.
Measure first-pass gate rate, findings, suppressions, test changes, review effort and escaped defects, compared with non-AI changes.
Gates passed but more escaped defects means checks are being satisfied without correct code; refocus review.
Link evidence where it lives; the go/no-go confirms AI-assisted changes passed the same gates as everything else.

Record AI assistance where the change lives

Lesson check