Test SAP AI Features After a Model or Joule Update

To test an SAP AI feature after a model or Joule update, capture a baseline before the change, re-run the same business prompts and grounded answers in a test system that matches production configuration, and review every difference against the expected business result. Your regular regression pack will not catch most of this. It checks that transactions complete. It does not check whether a Joule answer about a supplier invoice is still correct, or whether a model swap changed how a summary reads.

Why AI features change without a transport

In ECC, the change you tested was the change you transported. With SAP Business AI, some of the change arrives on SAP's schedule. SAP's Joule Preview Landscape exists for that reason. According to SAP's integration guide, customers will be able to test and validate content and framework updates in this environment before they are rolled out to production. The same guide describes the window: the Joule Framework will initially be released to the Joule Preview Landscape and remain in the Preview Landscape for 2 weeks, and content follows the same path before it reaches production. Check the current guide for which data centers have the preview landscape, because the guide states the preview landscape is available on EU10 & EU30 data centers, with plans to expand.

Generative AI Hub adds a second source of change: the model itself. Model versions have deprecation dates, and a deployment that pins a version stops working on that date. SAP's launchpad guide says that when you choose the latest version instead, existing generative AI deployments will automatically use the latest version of the given model once SAP AI Core supports it. A pinned version, by contrast, will be used in your deployments irrespective of updates to the models supported by SAP AI Core. Both options have a cost. Auto upgrade can change output between two test runs. A pinned version can stop working on a date you did not plan for.

Feature activation is the third source. On SAP S/4HANA Cloud Private Edition, the new AI capabilities in 2025 FPS02 are new capabilities in FPS02 and require activation. On Public Edition, the base AI features work differently. SAP's learning content says that you can start with a test system and a small group of users, validate that everything works, and then extend the role assignment to production and your broader user base. That sequence is your test plan. Put the activation step into the plan so it is not a surprise at go-live.

Step 1: Define the triggers that force a re-validation

Write the triggers down before the first cycle. If the trigger list lives only in people's heads, someone will skip it during a busy month. Use this as a starting point and adjust it to your scope:

Trigger	Where you see it	What to re-run	Who decides
Joule framework update in the preview landscape	SAP BTP Joule landscape and SAP release communications	Full baseline set, plus role-based access checks	Test lead with the Joule product owner
Joule content update	Same landscape, content release path	Baseline set for the affected LoB only	Functional lead for that process
Model version change or deprecation notice	Generative AI Hub deployment, SAP Note 3437766	Full baseline set on the model in use, plus latency and token checks	Architect and test lead
New AI feature activated in a test tenant	Feature list, activation request or Central Business Configuration	Feature test scripts and the affected role set	Business process owner
Grounding document or knowledge source changed	Document grounding index or source repository	Retrieval checks against known answers	Functional lead and data owner
Role or business catalog change	PFCG roles, business catalogs	Access tests for each role, including negative cases	Security role owner

For model versions, the SAP Note is the authority, not a blog post. SAP's supported-models documentation says you can find information about available models, including token conversion rates, rate limits, and deprecation dates, in SAP Note 3437766. Check it at the start of each cycle and record the date you checked.

Step 2: Build the baseline before anything changes

A baseline is a frozen set of prompts, the expected business answer for each, and the source record that proves it. Build it from the processes you signed off in fit-to-standard. If you cannot name the document number, status or approver that a correct answer must show, the test is not ready.

A workable baseline has four kinds of case:

• Happy-path questions with a known answer, such as the status of a named supplier invoice or the approver of a specific document.
• Action requests where the AI should hand off to a transaction or app, and the result is visible in the database or on screen.
• Negative cases: a user without the right role asks for the same data, a question about another company code, or a request the feature should refuse.
• Edge cases from your data: blocked items, partial deliveries, reversed documents, and anything your business has already had a complaint about.

Role coverage matters because AI features depend on authorizations, not just on being switched on. SAP's documentation for the Joule ILM data destruction capability notes that you must have the following business catalog assigned to your user: SAP_BASIS_BC_ILM. Test that dependency the way you test any other catalog. Our Joule agent guide covers this in more depth: Joule Agent Authorization Testing: A Practitioner Guide.

If you are building a question set from scratch, the free Testing Joule Agents and AI Features in SAP: Evaluation Sets, Non-Determinism and Sign-Off course in the SAP Academy covers how to write evaluation sets for answers that vary from run to run.

Step 3: Run the delta, not just the pass count

Run the baseline on the old and new configuration and compare them side by side. The process is the same whether the trigger was a framework update, a model change or a new role:

1. Confirm the triggerRecord the release, model version or activation request, with its date.
2. Match the systemConfirm the test system has the same configuration, roles and data shape as production.
3. Re-run the baselineUse the same prompts, users and inputs. Log the answer, the source and the time taken.
4. Score the differencesMark each case as same, acceptable change, or defect. A change in wording is not a defect if the business fact is correct.
5. Decide and signThe business owner accepts or rejects each defect, and the decision goes into the log.

The scoring step is where most teams go wrong. Non-deterministic output means a case can be right in wording and wrong in substance, or wrong once in three runs. Run each high-risk case more than once. Mark a case as failed if the business fact is wrong in any run, even if the next run is correct.

Step 4: Check the model deployment before you test the answers

Some failures are not AI failures. They are deployment failures, and they show up as errors rather than wrong answers. A model version that has been retired or a deployment that points to the wrong resource group will stop the feature completely. Check the deployment first. Our guide to the most common resource group and orchestration error is here: SAP AI Core Resource Group & Orchestration 404 Fix.

Then check the retirement dates in your own deployment. The SAP AI SDK release notes show how quickly models leave the catalog. In one release, SAP removed gpt-4.1 (retirement date: 2026-10-14), gpt-4.1-mini (retirement date: 2026-10-14), o3 (retirement date: 2026-10-16) and o4-mini (retirement date: 2026-10-16). Those dates have already passed as of today. If a test system still points to one of those models, the test results from that system are not evidence for production.

What changed from ECC

The change control model is different. In ECC, a custom report or enhancement moved through a transport, and the transport log told you what changed. With AI features, part of the change arrives from SAP, part from a model provider's version schedule, and part from your own activation decision. The transport log will not show all of it. Your change log has to do that job.

The scope has also grown. SAP's release notes for S/4HANA Cloud Private Edition 2025 describe Joule for Developers and ABAP AI capabilities as now available in SAP S/4HANA Private Cloud Edition 2025. That means developers, not only business users, now receive AI-generated output. Test any AI-generated ABAP the same way you test any other custom code. Run ATC and unit tests, and review the change. Our Joule in S/4HANA Private Edition guide maps the capabilities by line of business and is a good scoping reference: Joule in S/4HANA Private Edition: What It Does by LOB.

Who signs off, and what they need to see

The test lead should not be the only signer. The business process owner signs the answers, the security owner signs the role tests, and the architect signs the deployment and model version. The SAP program lead signs the overall release only when all three have signed.

• The trigger record with the date and the release or model version.
• The baseline set, version-controlled, with the source record for each expected answer.
• The re-run results, with each run logged for high-risk cases.
• The list of changed answers, with the owner's decision on each.
• Role test results for every role in scope, including negative cases.
• The deployment check: model version, retirement date and resource group.
• A rollback or disable plan: which feature to switch off, who can do it, and what users see.

If you need a starting structure, the SAP Joule & Embedded AI Feature Acceptance Checklist covers the acceptance side, and the Generative AI Hub in SAP AI Core free course explains model, orchestration and prompt registry settings you will need to record in the log.

Frequently asked questions

Do we still need to test Joule if SAP manages the updates?

Yes. SAP tests the framework and content it ships, but it does not test your roles, your data, your processes or the answers your users will rely on. The preview window gives you a place to test before production, and you need to use it. Schedule the baseline run inside that window.

How often should we re-run the baseline?

Run it on every trigger in your register, and keep a fixed cadence as a backstop, such as once per release cycle. Do not wait for a defect to tell you the feature changed. The baseline should take hours to run for a focused scope, not weeks.

Can the regression suite cover AI features?

Only partly. A regression suite confirms that transactions still post and screens still open. It does not tell you whether a generated answer, summary or recommendation is still correct. Keep AI answer checks in a separate baseline, and link each case to the process it supports.

What if the model answer is different but correct?

Mark it as an acceptable change and record why. The rule is the business fact, not the wording. If the answer is correct in substance but the source record is missing or the user would not know where it came from, treat that as a defect and raise it with the business owner.

Sources

1. SAP Business AI: Release Highlights Q4 2025, SAP News Center, January 2026
2. Joule Preview Landscape (Integration Guide), SAP Help Portal
3. Create a Deployment for a Generative AI Model, SAP AI Launchpad User Guide, SAP Help Portal
4. Supported Models, SAP AI Core documentation, SAP Help Portal
5. Prompt LLMs in the generative AI hub in SAP AI Core & Launchpad, SAP Developers
6. SAP AI SDK for JavaScript Release Notes, SAP, 2026 entries
7. Release Navigator for SAP Cloud ERP Private: What's New in 2025 FPS02, SAP Community
8. Activating and Configuring Base AI-Assisted Features in SAP S/4HANA Cloud Public Edition – PET 2026, SAP Learning
9. What's New in SAP S/4HANA, 2025 FPS01, SAP Help Portal
10. SAP Joule for Developers Expands to Private Cloud, SAP Community
11. SAP UX Update for SAP S/4HANA Cloud Private Edition 2025 FPS02, SAP Community