RAG for SAP: Document Grounding, SAP HANA Cloud Vector Engine and Answers Over SAP Documentation

Keep this: job aid

RAG for SAP: build-and-sign-off checklist

• Default to RAG for program facts; fine-tune only for style or narrow task behavior. Live S/4HANA data goes through Joule skills or APIs, not a vector store.
• Managed path: resource group labeled ext.ai.sap.com/document-grounding=true, generic secret, pipeline, then the grounding module (document_grounding_service).
• Custom path: REAL_VECTOR column table, VECTOR_EMBEDDING with DOCUMENT for chunks and QUERY for questions, COSINE_SIMILARITY sorted DESC.
• HNSW is approximate: compare results with NO_VECTOR_INDEX before go-live.
• Chunk on structure (rule, config activity, test case, runbook task) and keep the heading path in every chunk.
• Metadata on every chunk: source URL, process, version/status, cycle, security group.
• Gold set: real questions, labeled sources, key-user reference answers, at least 15% unanswerable.
• Measure retrieval (hit rate at k) before answers (correctness, groundedness, refusal, citation).
• Scope is the strongest control; grounding doesn't filter personal data. Filter by token groups before the model sees anything.
• Rerun the gold set on any change to model, prompt, chunking or scope.