Regression for prompts and models

A classic SAP regression suite reruns when something you transport changes. An AI feature can change behavior when nothing in your transports changed at all. The model behind it can be updated or retired, SAP can update Joule and the agent runtime, grounding documents can be edited, and CMQ data shifts every cycle. Your evaluation set is your regression suite, and the plan has to cover all of these triggers.

What triggers a rerun

Pin what you can. Where the platform lets you choose a model version rather than "latest", choose one and record it with every evaluation run, so a change is a decision you can test for. SAP publishes model availability and retirement dates for the generative AI hub; put the dates for your models in Luis's release calendar the same way you track SAP maintenance dates.

Baselines and comparison

Every signed-off evaluation run becomes the baseline for its agent version: the set version, model version, data seed, per-case results and per-skill rates. A regression run is compared with that baseline, not with the threshold alone. The questions are: did any zero-tolerance case fail; did any skill rate fall below threshold; and which individual cases got worse? A case that drops from 5 of 5 to 2 of 5 is a regression even if the skill rate stays above threshold.

Expect some movement. Repeated runs of a non-deterministic system never give identical numbers, so agree a tolerance band with Maria, such as a skill rate within two points of baseline, and investigate anything outside it. To tell variance from a real regression, rerun the cases that moved with more repetitions. Noise averages out; a regression doesn't.

What the platform tests for you, and what it doesn't

SAP's new Joule Studio has a Testing step that runs generated tests for agent solutions in three categories: Structure Tests (expected files and modules), Server Tests (the server starts, the agent-card endpoint responds, a basic invoke returns a valid response) and Agent Tests (tool invocations, response formatting and error handling, typically one per tool plus an end-to-end conversation). Its Score is simply passed tests divided by total tests. Joule Studio also runs an AI-powered evaluation of intent alignment, requirements coverage and specification adherence against the generated requirements and specification.

Use all of that; it catches broken builds early and cheaply. But notice what it measures: whether the solution matches what was generated and specified, with tests the platform wrote. It doesn't know Contoso's thresholds, CMQ data, roles or injection risks. A 100% Score is an entry criterion for your evaluation, not a substitute for it. For prompt and model choices built on the generative AI hub, SAP AI Core's evaluations can compare runs across models and prompt templates with the same dataset, which is useful when a model must be replaced.