Lesson 1 of 6
Why RAG: grounding answers in your own SAP documents
10 min
Listen to this lesson
A large language model knows a lot about SAP in general and nothing about your program in particular. Ask it how Contoso Motor Co. handles the credit block on sales orders, and it will answer fluently from what it saw in training: generic SAP behavior, maybe a transaction from an older release, maybe a field that your configuration doesn't use. That fluent but unsupported answer is what people call a hallucination. In a project setting it is worse than no answer, because it sounds like someone checked.
Retrieval-augmented generation (RAG) fixes the source of the problem. Before the model answers, a retrieval step finds the passages in your own documents that relate to the question and puts them into the prompt. The model then writes its answer from those passages. SAP calls this grounding: the generative AI hub's grounding module connects a model to external, domain-specific data at run time so that answers reflect your content rather than only the model's training data.
The pipeline in five steps
Every RAG system, whether you build it yourself or use SAP's managed service, has the same two halves. Indexing happens ahead of time; retrieval and generation happen on every question.
- CollectGather the documents: functional specs, runbooks, configuration guides, test evidence.
- ChunkSplit each document into passages small enough to retrieve precisely.
- EmbedTurn each chunk into a vector (an embedding) that captures its meaning.
- RetrieveEmbed the question, then find the chunks whose vectors are most similar.
- GenerateGive the model the question plus the retrieved chunks and ask it to answer only from them.
An embedding is a list of numbers, often 768 or more of them, produced by an embedding model. Texts with similar meaning land close together in that space, so "release strategy for purchase requisitions" and "PR approval over $10,000" end up near each other even though they share few words. That is why vector search finds relevant passages that keyword search misses.
RAG, fine-tuning or a longer prompt?
Teams often ask whether they should fine-tune a model on their documents instead. For SAP program knowledge the answer is almost always no. Fine-tuning changes how a model writes and reasons; it is a poor way to teach facts that change every sprint, and it can't tell you which document a statement came from. RAG keeps the facts in documents you already control, updates when the documents change, and lets every answer cite its source.
| Approach | Good for | Weak at | Fit for SAP program knowledge |
|---|---|---|---|
| RAG (grounding) | Facts in your documents that change often; answers with citations | Questions whose answer isn't in any document; poor retrieval | Best default: specs, runbooks, test evidence, decisions |
| Fine-tuning | Style, format and narrow task behavior | Fresh facts, traceability, deleting a fact later | Rarely the right first step |
| Long prompt with whole documents | One or two short documents, one-off questions | Cost, latency, hundreds of documents, access control | Fine for a single spec review, not for a library |
| Model alone | General SAP concepts | Anything specific to your system or program | Never for program decisions |
Choosing how to get your own knowledge into answers.
Documents versus business data
RAG over documents answers "what did we decide and where is it written". It is the wrong tool for "what is Great Lakes Fleet Services' open balance today". That answer lives in SAP S/4HANA, changes by the minute, and must respect the user's authorizations for company code 1710. Copying ledger extracts into a vector store gives you stale numbers and a second copy of financial data to secure. SAP's own stack separates the two kinds of grounding:
- Unstructured documents (specs, runbooks, policies): document grounding in the generative AI hub or your own SAP HANA Cloud vector tables, the subject of this course.
- Live transactional data (open items, stock, order status): Joule skills and agents that call SAP APIs with the user's authorizations, as covered in sap-ai-201.
- Curated analytical data: data products in SAP Business Data Cloud, which SAP positions as the business data foundation for agents on the SAP Business AI Platform.
- Business semantics and relationships: the SAP Knowledge Graph, which SAP describes as a structured map of business entities, processes and relationships that gives agents context. For your own graphs, SAP HANA Cloud has a knowledge graph engine (RDF triples queried with SPARQL), generally available since 2025, with GeoSPARQL support added in the QRC3 2026 release.
What "grounded" does and doesn't promise
Grounding narrows the model's attention; it doesn't make it infallible. The model can still misread a passage, combine two passages wrongly, or fill a gap with general knowledge. Two failure modes matter most. If retrieval returns the wrong passages, the answer will be wrong no matter how good the model is. If retrieval returns the right passages but the model adds claims that aren't in them, the answer is unfaithful. Lesson 5 shows how to measure both separately, because they have different fixes.
What would you do?
Maria, the order-to-cash key user, tests an early prototype. She asks, "What is the PO approval threshold?" and gets "Purchase orders above $50,000 need approval" with no source shown. Contoso's actual threshold is $25,000. What should the team do first?
- RAG retrieves passages from your documents and makes the model answer from them; SAP calls this grounding.
- Indexing (collect, chunk, embed) happens ahead of time; retrieval and generation happen per question.
- For changing program facts, RAG beats fine-tuning because documents stay the source of truth and answers can cite them.
- Wrong retrieval and unfaithful generation are different failures with different fixes.
Check your understanding · not graded
Lesson check
1. What does the retrieval step in RAG add to the model's prompt?
2. Why is fine-tuning usually a poor way to teach a model Contoso's approval thresholds?
3. Which step of the pipeline happens once ahead of time rather than for every question?
4. An answer cites the right runbook but adds a step the runbook doesn't contain. What kind of failure is this?