How LLMs answer SAP questions

Ask a general-purpose large language model (LLM) such as Claude, GPT or Gemini how to create a customer in SAP S/4HANA and you may get a confident answer that starts with transaction XD01. In SAP S/4HANA, customer master data is maintained through the business partner, transaction BP, and the old customer transactions redirect there. The model isn't lying. It's predicting the most likely text from what it learned, and a lot of what it learned describes SAP ECC. This lesson explains why that happens and what you can do about it, before we write a single prompt.

Prediction, not lookup

An LLM generates text one piece (a token) at a time, choosing what is most likely to come next given everything before it. It does not look facts up in a database unless you give it a way to. Three consequences matter for SAP work:

• Training is a snapshot. A model knows what was in its training data up to a cutoff. Release 2025 features, the October 2026 Joule for Developers licensing change or a new SAP Note may simply not be in it.
• Old SAP outnumbers new SAP. Twenty years of ECC forum posts, blogs and books dominate the training text. Ask a vague question and you get the ECC answer.
• Plausible beats true. When the model lacks a fact, it can still produce text that looks like one: a transaction code, a table name, a message number, an app ID. These are the most dangerous errors because they look right.

Context windows and grounding

The model only "knows" two things during a conversation: what it learned in training and what is in its context window, meaning your prompt, the documents you attach and the conversation so far. Putting the right material into that window is called grounding. Grounding is the single most effective way to get correct SAP answers, and it's what SAP builds into its own tools:

• SAP Joule for Consultants answers from SAP's own knowledge base (SAP says more than 18 million pages of documentation and training material) and shows the sources it used.
• Joule in SAP applications works on your business data with your authorizations, and agents show a Source and AI Reasoning section.
• The generative AI hub in SAP AI Core has a grounding module that retrieves relevant passages from your own document repositories at the moment of the question (course sap-ai-203).

Why the same prompt gives different answers

Run the same prompt twice and you will often get two different wordings, and sometimes two different answers. Models sample from a range of likely next tokens, and settings such as temperature (Google's guide lists it alongside top-K, top-P and maximum output length) control how adventurous that sampling is. For SAP work, variation in wording is harmless; variation in facts is a warning. If a transaction code or a configuration path changes between runs, the model doesn't know it, and you need a source.

Most 2026 models also reason before they answer: OpenAI's reasoning models, Gemini's thinking models and Claude with extended or adaptive thinking work through a problem internally first. That helps with multi-step questions such as tracing why a billing document didn't post, but it doesn't add facts the model never had. With thinking switched on, some providers fix or ignore sampling settings such as temperature, so check your tool's documentation before you rely on them. Reasoning makes a model better at using the context you give it; it doesn't replace that context.

What models are genuinely good at in SAP work

None of this makes LLMs useless. They are excellent at the language work around SAP: turning a messy workshop note into a structured requirement, turning a process description into test steps, explaining what an error message means in plain words, summarizing a long list into the five things a release manager must know, and drafting a defect report that a developer can act on. In every one of those tasks, you supply the facts and the model supplies the structure and the words. That division of labor is the theme of this course.