Now accepting guest editorial from SAP, Oracle, Workday and ServiceNow practitioners.Pitch a guide
The Sign-Off Table

Lesson 6 of 6

Measuring whether AI actually saves delivery time

8 min

Listen to this lesson

0:00 / 7:19

SAP says requirement generation saves 50% of requirement-writing time and 20% of user story time. Those numbers may be true for SAP's reference projects; they say nothing yet about yours. AI Units cost money, reviews cost key-user time, and rework from a bad draft can cost more than writing from scratch. The only way to know is to measure, the same way you'd measure any process change.

Measure the whole cycle, not the drafting

The commonest mistake is timing only the generate button. A requirement that drafts in 2 minutes but takes 40 minutes to review and two rework rounds is not a 90% saving. Measure from input to approved artifact, and keep counting into the next stage, because defects that slip through show up later as test failures and change requests.

MetricHow to measureWhere the data comes from
Cycle time to approved requirementWorkshop end to Approved, per requirementSAP Cloud ALM requirement history
Review effortReviewer minutes per artifactShort time log by reviewers during the pilot
Rework rateShare of drafts edited substantially or rejectedApproval history; reviewer notes
Downstream defectsTest failures or change requests traced to requirement errorsDefect root-cause field in SAP Cloud ALM
CoverageException paths per process covered by test casesTest plan coverage information
CostAI Units consumed per approved artifactSAP for Me consumption view

What to measure in an AI delivery pilot.

Design a fair pilot

  1. BaselineMeasure two comparable processes done the usual way in Wave 2 sprint 1.
  2. PilotUse AI on two comparable processes in sprint 2, same team, same review rules.
  3. CompareCycle time, review effort, rework, defects and AI Units, per approved artifact.
  4. DecideScale, adjust (inputs, batch size, reviewers) or stop, and record why.
A two-sprint pilot that compares like with like.
The Sign-Off Table · thesignofftable.com/academy/sap-ai-206

Keep the comparison fair: similar process complexity, the same reviewers, and the same review checklist in both arms. If the AI arm uses a better transcript or more experienced reviewers, you are measuring those, not the AI. Record what was different and say so in the result.

Contoso's two-sprint pilot plan for Wave 2, with a steering decision at the end.
The Sign-Off Table · thesignofftable.com/academy/sap-ai-206

What Contoso found

Contoso's Wave 2 pilot (illustrative figures, not SAP data) compared two order-to-cash processes drafted manually with two drafted by requirement generation. Drafting time per requirement fell from about 55 minutes to about 10. Review time rose from about 15 to about 25 minutes, because reviewers checked every statement against the transcript. Net cycle time fell by roughly 45%, close to SAP's claim, but rework was higher in the first week until the facilitator started cleaning transcripts and using the process, object, rule title format. Downstream, two requirement errors reached testing in the AI arm against one in the manual arm, both caught in system integration test.

Put a price on each approved artifact

Time is only half the business case. Divide the AI Units consumed in the pilot by the number of approved artifacts, convert that to money at your contract rate, and add the reviewer time at an internal cost rate. Compare the result with the manual cost per artifact from the baseline sprint. This "cost per approved artifact" is easy for a steering group to understand and hard to argue with, and it makes the trade-off visible when review effort rises. If AI drafting saves consultant hours but consumes more key-user hours, that is a real cost shift to the business, and the business should agree to it before you scale.

Low saving, low riskShort, simple requirements: AI adds little; write them directly.
High saving, low riskLong workshops with clear decisions and clean transcripts: the sweet spot.
Low saving, high riskContentious topics with no clear decision: AI drafts confuse more than help.
High saving, high riskLarge batches of test cases: big gains only with strict review and sampling.
Where AI drafting pays off, and where it needs the strongest controls.
The Sign-Off Table · thesignofftable.com/academy/sap-ai-206

What would you do?

After the pilot, a vendor slide claims "80% faster test design with AI." Contoso measured 30% faster with higher review effort. Grace asks which number goes in the business case for Wave 3. What should Priya say?

Key points
  • Treat vendor time-saving figures as claims to test, not as plan inputs.
  • Measure the whole cycle: drafting, review, rework, downstream defects and AI Unit cost.
  • Run a fair pilot: comparable processes, same reviewers and the same review rules in both arms.
  • Report measured results with their conditions, and decide to scale, adjust or stop.

Check your understanding · not graded

Lesson check

  1. 1. Why is timing only the "generate" step misleading?

  2. 2. What makes an AI pilot comparison fair?

  3. 3. Which source shows AI Units consumed per feature?