Lesson 2 of 6
Evaluation sets
17 min
Listen to this lesson
An evaluation set is the AI equivalent of a regression suite: a fixed, versioned collection of cases, each with an input, the system state it needs, the behaviors expected and a rule for scoring them. You run the whole set, many times, and the results become the evidence for sign-off. SAP uses the same idea. Its benchmark evaluation for Joule custom knowledge scores each question in a golden dataset, and SAP AI Core's evaluations run a registered test dataset against prompt templates and models with chosen metrics.
Anatomy of a case
| Field | What goes in it | Order Status Agent example |
|---|---|---|
| ID and category | A stable ID and the skill or risk area | OSA-017, order status |
| Prompt (and turns) | The user input, or a short multi-turn script | "Has order 1000457 shipped yet?" |
| Data state | The documents and their status in the test client | Sales order 1000457 delivered on 80001234 and billed on 90001877 in CMQ client 100 |
| Must | Behaviors required in every run | Calls the sales order and delivery APIs; states it shipped; quotes delivery 80001234 |
| Must not | Behaviors that fail the case outright | Invents a tracking number; calls the credit memo tool; mentions another customer |
| Should | Quality expectations scored on a scale | Gives the billing document and date; under 80 words |
| Scoring rule | How each behavior is judged | Trace check for tools; exact match for document numbers; rubric 1–5 for completeness |
| Risk | How bad a failure is | Medium: wrong status misleads a customer but changes no data |
One evaluation case, field by field.
Where good cases come from
- The agent's own examples. SAP's Joule Studio documentation says the example utterances in an agent's skills are exactly what Joule uses for routing and recommends using them verbatim for testing. They are your first cases, not your only ones.
- Real requests, masked. Maria's team logs the questions customers actually ask. Rewrite them with test data; never copy customer data from production into a case.
- Paraphrases. Five ways to ask the same thing, including short, rude, misspelled and mixed-language versions, to test routing and robustness.
- Edge and negative cases. An order that doesn't exist, a partly delivered order, an order for another customer, a canceled order, a question with no order number.
- Out-of-scope requests. "Change the price on 1000457" should be refused or redirected, not attempted.
- Multi-turn cases. Follow-ups such as "and the invoice?" that depend on context from the previous turn.
Size the set by risk, not by habit. For the Order Status Agent, Priya plans 120 cases: 40 order status, 25 invoice explanation, 20 credit memo drafting, 15 out-of-scope and 20 multi-turn. The adversarial cases in lesson 4 form a separate suite owned by Nadia. Stratify so each skill and risk level has enough cases for its own rate; a single headline score can hide a skill that fails half the time.
Scoring: three kinds of judge
SAP AI Core's evaluations show the menu. Its system-defined metrics include computed ones such as Exact Match, JSON Schema Match, BLEU, ROUGE and BERT Score (several need a reference answer), and LLM-as-a-judge ones such as Pointwise Correctness, Pointwise Instruction Following and Pointwise RAG Groundedness, scored on small scales; SAP marks some as experimental, and you can register custom metrics. These evaluations target prompt templates, models and orchestration configurations on the generative AI hub, so they fit the model and prompt parts of a solution. For a whole agent you still need case-level checks on the trace.
Repetitions and data state
Run every case several times; five is a practical start. Record each run, not just the best one. A case that passes 5 of 5 is stable; 3 of 5 is a variance problem even if the average looks fine. Reset the data state before each cycle: the cases refer to real documents in CMQ, so a client refresh or another tester's posting can break them. Keep the case data in a seed list Wei's test data team can recreate, and version the set itself, because every result only means something against a known version.
What would you do?
The BTP team proposes an evaluation set of 40 cases, all copied from the Order Status Agent's own skill examples, each run once, scored by an LLM judge nobody has calibrated. Priya has a week before the first evaluation cycle. What should she change first?
- An evaluation set is a versioned regression suite for AI: prompt, data state, must, must-not, should, scoring rule and risk per case.
- Source cases from the agent's own examples, masked real requests, paraphrases, edge, out-of-scope and multi-turn cases.
- Score with deterministic checks first, graded rubrics or LLM-as-a-judge second, and people for calibration and hard calls.
- Run each case several times, record every run and reset the data state before each cycle.
Check your understanding · not graded
Lesson check
1. What does SAP recommend as the most reliable inputs for testing a Joule Studio agent in the sandbox?
2. Why should an LLM judge be calibrated against a key user before it scores a whole evaluation set?
3. A case passes 3 of 5 runs while the category average looks healthy. How should Priya read it?
4. Which SAP AI Core metric checks whether output matches a predefined structure without needing a reference answer?