Lesson 7 of 7
Cost and limits
8 min
Listen to this lesson
The generative AI hub is billed by use, and use grows quietly: more users, longer prompts, more grounding, a second content filter. Grace's question to Theo before production is simple: "What will this cost per month, and how will we know if it's drifting?" This lesson shows how SAP meters the hub and how to estimate, cap and monitor it.
How SAP meters generative AI
SAP's metering page explains the chain. Models process text in tokens; input tokens (your prompt, including grounded context) and output tokens (the answer) are counted separately. Usage is converted into GenAI tokens, "a virtual unit used to represent usage across different models", using a conversion rate per model for input and for output. GenAI tokens are then converted into capacity units, the billing metric for SAP BTP. SAP notes that "output tokens tend to be slightly more expensive than input tokens", and that more capable models typically cost more per token. The conversion rates per model are in SAP Note 3437766.
- TokensInput tokens (prompt, grounded context) and output tokens (answer) per request
- GenAI tokensEach count × the model's conversion rate per 1,000 tokens (SAP Note 3437766)
- Capacity unitsGenAI tokens converted to SAP BTP capacity units
- Other modulesGrounding storage (gigabyte-days), retrieval, masking and filtering calls, inference observability (data volume)
- BillCapacity units consumed under the extended plan
A worked estimate
SAP's documentation uses a fictitious example that is a good template. A grounded application handles 25,000 requests per month, each with 3,500 input tokens (mostly retrieved context) and 300 output tokens. With fictitious conversion rates of 0.00112 GenAI tokens per 1,000 input tokens and 0.00320 per 1,000 output tokens, each request costs (3,500 / 1,000 × 0.00112) + (300 / 1,000 × 0.00320) = 0.00392 + 0.00096 = 0.00488 GenAI tokens. For the month that is 25,000 × 0.00488 = 122 GenAI tokens for the model calls, before converting to capacity units and before adding grounding storage and retrieval, content filtering, data masking and inference observability, each with its own metric.
| Driver | Effect on cost | Lever |
|---|---|---|
| Grounded context size | Usually the largest share of input tokens | Retrieve fewer, better chunks; trim templates |
| Output length | Output tokens cost more per token | Set maximum completion tokens; ask for concise formats |
| Model choice | More capable models cost more per token | Use the smallest model that passes the regression set |
| Content filters | Each filter type is a separate billed request | Use the filters the risk needs, not every filter |
| Prompt optimization | Many requests per run, minutes to hours | Run deliberately, on a bounded dataset |
| Retries and fallbacks | Failed calls and fallbacks consume too | Fix root causes; monitor 429s |
Main cost drivers for a generative AI hub application.
Limits you will meet
- Deployment quotas per tenant (deployments and replicas); requests to raise them go to support component CA-ML-AIC.
- Resource groups: 50 per tenant by default.
- Tenant-wide generic secrets: five per tenant.
- Model rate limits: listed per model in SAP Note 3437766; exceeding them returns 429, which orchestration fallbacks can absorb for non-streaming calls.
- Context length: per model; SAP-ABAP-1, for example, has a 128,000-token context window.
Measure, don't guess
The best estimate comes from your own test runs: log input and output tokens per request in cmc-ext-test, multiply by the conversion rates for the chosen model, and scale by expected volume. Contoso's IDoc Explainer averaged about 2,900 input and 250 output tokens per request in testing, and failed IDocs that need explaining run at roughly 300 per month after go-live, so the model cost is small; grounding storage and the two content filters turned out to matter as much as the model calls. Without measuring, the team would have optimized the wrong thing.
What would you do?
A month after go-live, the IDoc Explainer's consumption is three times the estimate, with the same number of requests. What should Theo check first?
- Tokens → GenAI tokens (per-model rates in SAP Note 3437766) → capacity units; output tokens cost more.
- Grounding, filtering, masking and observability add their own metrics.
- Limits: deployment quotas, 50 resource groups, five tenant-wide secrets, per-model rate limits and context windows.
- Estimate from measured tokens per request in test, set a budget, monitor weekly and tune what drives cost.
Check your understanding · not graded
Lesson check
1. Into which unit is generative AI usage converted before billing as capacity units?
2. Which tokens does SAP say tend to be slightly more expensive?
3. In SAP's fictitious example, how many GenAI tokens does one request with 3,500 input and 300 output tokens use?
4. Consumption tripled with the same number of requests. What is the most likely driver?