Lesson 5 of 6
Verifying the answer
9 min
Listen to this lesson
Every provider says some version of the same thing. Anthropic's guide on reducing hallucinations ends with: these techniques "significantly reduce hallucinations" but don't eliminate them, so "always validate critical information". In SAP work, "critical" covers most things you'd act on: a transaction code, a configuration path, a message number, a release fact, a number in a test result. This lesson gives you prompt techniques that make answers easier to check, and a verification routine for checking them.
Prompt techniques that make answers checkable
| Technique (from the providers' guides) | SAP example |
|---|---|
| Allow "I don't know" (Anthropic) | "If the documents below don't say which app replaces this transaction in release 2025, say so." |
| Ground in direct quotes first (Anthropic) | "Quote the passages from the attached SAP Help pages that describe the setting, then answer using only those quotes." |
| Cite sources per claim (Anthropic) | "For each step, name the document and section it comes from; remove any step you can't support." |
| Restrict to supplied material (Anthropic) | "Use only the attached release notes, not general knowledge." |
| Compare several runs (Anthropic: best-of-N) | Run the same question three times; inconsistent transaction codes are a red flag. |
| Pin the model and test changes (OpenAI) | Keep the same model version for a saved prompt template and re-run the test inputs when it changes. |
Techniques from Anthropic's and OpenAI's published guides, applied to SAP questions.
A verification routine
- Is it a fact about SAP software (transaction, app, field, setting, message)?Yes → check SAP Help Portal for your product and release · No → go to the next question
- Does SAP Help confirm it for SAP S/4HANA 2025?Yes → note the page · No or unclear → check the Simplification List or What's New
- Is it a Fiori app?Yes → confirm the app ID and release in the SAP Fiori apps reference library
- Is it about your configuration or data?Yes → check in the sandbox or quality system, never only in a document
- Is it a number (amount, quantity, date)?Yes → recompute it from the source data yourself
Two SAP sources catch most model errors about S/4HANA. The Simplification List describes what changed from SAP ECC, which is exactly where models trained on ECC go wrong: customer and supplier maintenance moved to business partner (BP), and the old inventory transactions such as MB1A, MB1B and MB1C were replaced by MIGO. The SAP Fiori apps reference library tells you whether an app exists for your product and release, and under which ID. When a model names an app, look it up there.
Don't verify with the same model
Asking the model "are you sure?" is not verification. A model may apologize and change a correct answer, or confidently confirm a wrong one. Comparing several runs is useful as a warning signal (if the transaction code changes between runs, distrust all of them), but agreement across runs doesn't prove correctness. Independent sources do: SAP Help, SAP Notes, the Simplification List, the Fiori apps reference library and, above all, the system itself.
Grounded tools still need checks
Grounding reduces errors; it doesn't remove them. SAP Joule for Consultants cites its sources, which makes checking fast: open the cited page and confirm that it says what the answer claims, for your product and release. A cited page for SAP S/4HANA Cloud Public Edition doesn't prove a setting for Private Edition. Agent answers in Joule show a Source and AI Reasoning section; check that the data items it lists are the ones you expected, such as the right customer and company code. The habit is the same with every tool: follow the citation, don't just admire it.
| Claim from the model | Checked in | Result |
|---|---|---|
| "Use XD02 to change payment terms" | Simplification List; system | Replaced: customers are maintained with BP |
| "Goods issue posts movement type 601" | Key-user guide; CMQ dry run | Confirmed for outbound deliveries |
| "App F0000 manages sales orders" | SAP Fiori apps reference library | Wrong ID; correct app found and recorded |
| "Net value 12,600.00 USD" | Recomputed from process description | Wrong; 50 × 240.00 = 12,000.00 USD |
A verification log from Contoso's test design work. The app ID in row 3 is a deliberately invented example of a model error.
What would you do?
Dan asks an LLM which app replaces the old customer line item display in SAP S/4HANA. It names an app with an ID and says it is "available in all releases". What should Dan do before putting it in the finance key-user guide?
- Ask for quotes, citations and permission to say "I don't know"; restrict answers to supplied material.
- Verify SAP facts in SAP Help, the Simplification List, the Fiori apps reference library and the system.
- Re-running a prompt is a warning signal, not proof; never verify with the same model alone.
- Record the source of each SAP-specific fact in the document that uses it.
Check your understanding · not graded
Lesson check
1. Which technique from Anthropic's guide makes an answer about a long document easier to check?
2. Which SAP source best catches model errors that come from ECC habits?
3. Three runs of the same prompt give the same transaction code. What does that prove?
4. Where do you confirm a Fiori app's ID and release availability?