Lesson 1 of 6
Testing something that varies
10 min
Listen to this lesson
Every test you have written for SAP so far rests on one assumption: the same input, in the same system state, gives the same output. Post a goods issue for 50 pieces of CMC-ALT-100 and you get a material document, a quantity of 50 and an accounting document for 7,500.00 USD, every time. That assumption is what makes an expected result possible. AI features break it. Ask a Joule agent "Where is order 1000457?" twice and you can get two differently worded answers, and on a bad day a different tool call or a wrong date. This course is about how to test and sign off that kind of behavior. It builds on sap-aiq-201, which used AI to help you test; here the AI is the thing under test.
What you are actually testing
"Testing Joule" is too broad to plan. Break the feature into the parts that can fail separately. SAP's own documentation for agents built in Joule Studio does the same: when a user writes to Joule, Joule compares the message with the description and examples of every registered agent to decide which one to invoke, and the agent then calls tools and composes a response. Each step is a place where behavior can vary and where a test can look.
Contoso's running example for this course is a custom agent its BTP team is building in Joule Studio: the CMC Order Status Agent. It serves Maria's order-to-cash team. It answers order status questions by reading the sales order, outbound delivery and billing document through published APIs on CMQ client 100 (through the CMC_S4HANA destination from the cmc-ext-test subaccount), explains an invoice such as 90001877 for Great Lakes Fleet Services, and can draft a credit memo request for returned units. The last skill changes data, so it is where the stakes are highest.
Where the variation comes from
| Source | What varies | Example in the Order Status Agent |
|---|---|---|
| Sampling | The model chooses among likely next words, so wording, order and sometimes facts change between runs | One run says "shipped on delivery 80001234"; another leaves out the delivery number |
| Routing | Whether Joule picks your agent at all depends on how close the prompt is to the agent's description and examples | "Has GLFS's stuff gone out?" may not route to the agent |
| Tool selection and parameters | Which API the agent calls, in what order, with which filter | It reads the billing document but never checks the delivery |
| Data state | CMQ data changes with every test cycle and client refresh | Order 1000457 is fully billed in one cycle and partly delivered in the next |
| Model and platform versions | SAP and model providers update models, Joule and the agent runtime | A model update makes answers longer and drops the order date |
| Conversation state | Earlier turns shape later ones | After a question about another customer, the agent reuses the wrong sold-to party |
Six sources of variation; only the first is "randomness" in the everyday sense.
SAP says this plainly in its guidance for building Joule agents: results are not always the same for the same input, and behavior can vary with the model and configuration, so agents need careful testing and validation over several iterations. The ISTQB AI testing syllabus (CT-AI v2.0) frames the same problem as the probabilistic and non-deterministic behavior of AI-based systems. In testing terms it is a test oracle problem: for many prompts there is no single correct output to compare against, only a range of acceptable ones.
From expected results to measured behavior
The fix is not to give up on expected results. It is to state them as behaviors that can be checked, and to measure how often the feature shows them across many runs. Some behaviors are deterministic even when the wording is not: the agent must call the sales order API with 1000457, must not call the credit memo tool for a status question, and must return the delivery number that exists in CMQ. Others need a graded judgment: is the explanation of invoice 90001877 correct and complete? A few need a person.
| Question | Classic SAP test case | Test of an AI feature |
|---|---|---|
| What is expected? | One exact result: document created, values, message | Behaviors: must do, must not do, facts that must appear, tone and format rules |
| How is it checked? | Exact comparison | Deterministic checks where possible, graded scoring (rubric or LLM-as-a-judge), human review for the rest |
| How many runs? | One run per data variant | Several runs of every case, because one pass proves little |
| What is the verdict? | Pass or fail | A rate per behavior, compared with a threshold agreed with the business |
| When is it retested? | When the code or configuration changes | Also when the model, prompt, grounding content, data or platform changes |
Who does what changes too. SAP tests the agents and AI features it delivers, but your configuration, your data, your roles and your process are yours to test, the same as with any SAP standard function. For agents you build yourself in Joule Studio, everything above the platform is yours. Priya, Contoso's test lead, owns the evaluation approach; Maria owns what "correct" means for order-to-cash; Nadia, the security and GRC lead, owns the safety cases; and Dan signs anything that touches credit memos and therefore finance.
What would you do?
The BTP team shows the Order Status Agent to Grace's steering group. It answers five order questions perfectly, and Grace asks Priya whether it can go live with the next release. What should Priya say?
- Break an AI feature into routing, reasoning, tool calls, response and confirmation; each can vary and each can be checked.
- Variation comes from sampling, routing, tools, data, versions and conversation state, not only randomness.
- State expected results as behaviors, check them deterministically where you can, and measure rates across repeated runs.
- SAP tests what it delivers; your configuration, data, roles and custom agents are yours to test.
Check your understanding · not graded
Lesson check
1. SAP's guidance for Joule agents says results are not always the same for the same input. What does that change for the tester?
2. Which of these is a deterministic check you can make on the Order Status Agent even though its wording varies?
3. Why is a successful steering-committee demo weak evidence for go-live?
4. Which source of variation can make Joule skip your agent entirely?