Lesson 3 of 7
The orchestration service
8 min
Listen to this lesson
The orchestration service is the part of the generative AI hub that makes it more than a model proxy. SAP describes it as a harmonized API that combines several modules "into a pipeline that can be executed with a single API call", where "the response from one module is used as the input for the next module." For Contoso's IDoc Explainer, that one call retrieves the right runbook passages, fills the prompt template, masks partner data, checks content, calls the model and checks the output.
The modules
| Module (V2 config key) | Purpose | Required? |
|---|---|---|
Templating (prompt_templating) | The prompt with placeholders filled at run time, plus the model configuration (name, version, parameters) | Yes |
Grounding (grounding) | Retrieves relevant passages from your document repositories into a placeholder | Optional |
Data masking (masking) | Anonymizes or pseudonymizes personal data before it reaches the model | Optional |
Content filtering (filtering) | Screens input and output for harmful content and prompt attacks | Optional |
Translation (translation) | Translates input before and output after the model | Optional |
Orchestration modules in the V2 API. SAP's documentation: templating and model configuration are mandatory; all others are optional.
SAP's documentation says the execution order of the pipeline "is defined centrally in orchestration"; you configure each module and leave out the optional ones. Conceptually, the flow looks like this: context is retrieved, the template is filled, input is translated, masked and filtered, the model answers, the output is filtered and translated, and pseudonymized values are restored.
The V2 request
In the V2 API, the application sends a POST to {orchestration deployment URL}/v2/completion with three headers (Authorization: Bearer token, content-type: application/json and AI-Resource-Group) and a body with two parts: config.modules, holding each module's configuration, and placeholder_values, holding the run-time values for the template's placeholders. Placeholders are written as {{?name}} in the template. Teams migrating from version 1 must move to the /v2/completion endpoint and adjust their payloads.
prompt_templating.prompt.template: the messages with roles (system, user) and placeholders such as{{?idoc_status}}and{{?runbook_context}}.prompt_templating.model: the model name, version and parameters such as maximum completion tokens and temperature.grounding: the grounding service type, repository IDs and search settings; its result fills a placeholder.masking: the provider (SAP Data Privacy Integration), the method (anonymization or pseudonymization) and the entity types.filtering: separate input and output filter lists.
Fallbacks
Orchestration V2 can take a list of module configurations in order of preference. SAP's documentation: if the preferred model isn't supported in the deployed region, orchestration uses the next configuration; for non-streaming requests it also falls back on 429 Too Many Requests, 408 Request Timeout and any 5xx server error. Fallback configurations may differ, for example a cheaper model or a different filter set, so test each one: a fallback is a second production path.
| Need | Direct model deployment | Orchestration |
|---|---|---|
| Switch from one model provider to another | New deployment and code change | Change the model name in the config |
| Mask partner names | Build it yourself | Masking module |
| Ground on Contoso runbooks | Build retrieval yourself | Grounding module |
| Block prompt attacks | Build or buy separately | Content filtering with prompt attack detection |
| Survive a model outage | Custom retry logic | Fallback configurations |
Tools and agents
Orchestration also supports tool definitions for models that call functions. In September 2026 SAP added the option to define tools through model parameters at run time ("dynamic tool assembly") instead of only in a prompt template, and a field that returns provider-specific information. Agent frameworks can sit on top: SAP's documentation names LangChain support and an SDK path to LiteLLM. For Joule agents built in Joule Studio, SAP manages this layer for you.
What would you do?
Theo's first prototype calls a Claude model through a foundation-models deployment and masks partner names with his own regular expressions. Grace asks whether that's acceptable for production. What should Theo propose?
- Orchestration runs templating (mandatory, with model config) plus optional grounding, masking, filtering and translation in one call.
- V2: POST /v2/completion with Authorization, AI-Resource-Group, config.modules and placeholder_values.
- Fallback configurations switch on region-unsupported models and, for non-streaming calls, on 429, 408 and 5xx errors.
- Use orchestration when you need SAP's modules or want to switch models by configuration.
Check your understanding · not graded
Lesson check
1. Which orchestration modules are mandatory?
2. What is the V2 orchestration endpoint path?
3. When does orchestration switch to a fallback configuration for a non-streaming request?
4. How are placeholders written in an orchestration prompt template?