Lesson 6 of 6
Evaluating a pilot and deciding to scale
9 min
Listen to this lesson
A pilot is an experiment with a decision at the end. Dan's question in eight weeks will be: scale, adjust or stop? This lesson sets up the measurement so that the answer comes from evidence. SAP's own guidance sets the tone: agents' "results are not always the same for the same input", and they "need to be tested and validated carefully".
- Weeks -4 to 0Baseline and setupMeasure a representative month; provision the assistant in SAP for Me; assign roles; confirm identity stage 2
- Weeks 1 to 2Evaluation in CMQRun the evaluation set three times; test every control; classify and fix errors
- Week 3Go decision for production pilotDan and Priya review results; Grace confirms the AI Unit budget
- Weeks 3 to 10Production pilotFour AR accountants, fleet customers only; weekly sampling and consumption review
- Week 11Scale decisionScorecard, evidence pack in SAP Cloud ALM, decision recorded: scale, adjust or stop
Baseline first
Measure today's process before the agent arrives. For Contoso's AR pilot that means, for a representative month: how many dispute cases were created, how long each took from customer complaint to case, how many were later corrected or cancelled, and how long an account review for a fleet customer took. Without a baseline, "the agent saves time" is an opinion.
The evaluation set
Before the pilot starts in production, Priya's team builds an evaluation set: realistic cases with the expected outcome, run in CMQ. For the Dispute Creation Agent, each case is a customer complaint (written in the customer's style), the right billing document, and the expected dispute: customer, amount, reason. The set includes easy cases, hard cases (partial deliveries, several invoices, ambiguous wording) and cases that must trigger escalation. Run the whole set several times; agents vary, so one run isn't the result.
| Measure | How | Pilot target (Contoso, illustrative) |
|---|---|---|
| Outcome accuracy | Share of evaluation cases where the proposal matched the expected result | 95% or better before production; tracked weekly by sampling in production |
| Control compliance | Confirmation asked for every data change; threshold and escalation rules obeyed | 100%; any miss is a defect that stops the pilot |
| Time per case | Complaint to dispute case, versus the baseline month | At least 40% faster |
| Correction rate | Disputes changed or cancelled later | Not worse than baseline |
| Cost per case | Agent actions per case × AI Unit rate | Known and below the value of time saved |
| User acceptance | Short survey and rejection reasons | Accountants would keep it |
Pilot scorecard for Contoso's AR agents. Targets are Contoso's illustrative choices.
Error analysis
When a case fails, classify why. The class decides the fix:
- Retrieval: the agent read the wrong customer or document. Check input quality and tool scope.
- Reasoning: the data was right but the conclusion wrong. Report to SAP with the case; consider tighter instructions for custom agents.
- Control: confirmation, threshold or escalation didn't behave. Treat as a severity-1 defect.
- Data quality: the agent was right about bad master data (for example a wrong dunning procedure). Fix the data.
- Expectation: the evaluation case's "expected" answer was wrong. Fix the set.
Watching cost
Agents are premium AI, charged per agent action. One request can involve several actions. The AI Estimator in the SAP Discovery Center listed the AR agents at 0.02 AI Units per agent action when this course was reviewed (17 October 2026); check the current rate and your contract. During the CMQ runs, record the actions per case, multiply by the rate, and you have a cost per case to compare with the time saved. In production, watch consumption per premium feature in SAP for Me each week.
Observability and governance
SAP AI Agent Hub, delivered in SAP LeanIX and announced as generally available in October 2026, represents AI agents, MCP servers and AI models as fact sheets linked to business capabilities, organizations and owners. Registering the pilot's agents there, with Dan as business owner, gives Contoso's inventory a place for every agent it runs. SAP's 8 October 2026 article also names SAP Signavio Agent Mining among the new capabilities that "measure the outcomes every agent delivers". Agent mining, first presented in November 2025, is meant to trace agent decisions, measure their effect on process time, accuracy and compliance, and monitor model costs. SAP's own reference architecture says the agent mining model is planned for agents built on the new Joule Studio, which wasn't generally available yet, so check its status before planning on it; until then, Priya's evaluation set and SAP for Me are the measuring tools.
What would you do?
After eight weeks, accuracy in sampling is 96 percent, control compliance is 100 percent, time per case is 35 percent faster than baseline (target 40 percent), and cost per case is well below the value of time saved. What should Dan decide?
- Measure a baseline month before the agent starts.
- Build an evaluation set with easy, hard and escalation cases; run it several times.
- Classify errors (retrieval, reasoning, control, data quality, expectation); controls failures stop the pilot.
- Track cost per case from agent actions and AI Unit rates; decide scale, adjust or stop on the scorecard.
Check your understanding · not graded
Lesson check
1. Why must a pilot measure a baseline month first?
2. Which error class should stop the pilot until fixed?
3. How do you estimate cost per case for an agent?
4. What does SAP AI Agent Hub represent as fact sheets?