Evaluating retrieval and answer quality

A RAG assistant that "seems good" in a demo is not ready for a hypercare desk. You need evidence, the same way you need evidence before you sign off a business process. The evidence comes from a gold question set: real questions, the documents that should answer them, and the answers a key user agrees are right. With it, you can measure retrieval and generation separately and decide go or no-go with numbers.

Build the gold set

1. Collect real questions. Pull them from the defect tool, the hypercare channel and workshop notes. Contoso collected 120; Priya kept 80 that are answerable from program documents and 20 that deliberately aren't.
2. Label the expected sources. For each answerable question, record the document IDs and sections that contain the answer, for example FS-O2C-014 section 3.2.
3. Write reference answers. A key user writes the short correct answer. Maria owns order-to-cash questions; Dan owns finance; Luis owns cutover.
4. Add the "no answer" cases. For the 20 unanswerable questions, the expected behavior is a clear "I can't find this in the program documents."
5. Freeze and version it. Store the set in the repository so every change to chunking, prompts or models is measured against the same questions.

Measure retrieval first

Run every question through the Retrieval API (or your HANA query) without the model, and compare the returned chunks with the labeled sources. Two simple metrics do most of the work. Hit rate at k is the share of questions where at least one correct source appears in the top k chunks. Mean reciprocal rank rewards having the correct source near the top. If hit rate is low, stop: no prompt can rescue answers built on the wrong passages. Fix chunking, metadata, filters or the embedding model, and rerun.

Then measure the answers

With retrieval in shape, run the full pipeline and score each answer on three questions. Is it correct against the reference answer? Is it faithful (grounded), meaning every claim is supported by the retrieved chunks? Does it refuse properly when the documents don't contain the answer? SAP's generative AI hub evaluation offers LLM-as-a-judge metrics for two of these ideas. Groundedness checks whether the response relies only on the provided context; context relevance checks whether the retrieved context matches the query. SAP reports validating them against public and internal datasets, with judge correlations of roughly 79 to 87 percent against ground truth depending on the judge model. That is useful for screening large sets, not a replacement for a person reviewing a sample.

Keep evaluating after go-live

Documents change daily and models change on SAP's schedule, so a pass today is not a pass forever. Treat the gold set as a regression suite. Rerun it when the pipeline scope changes, when you change the model in the orchestration configuration, and on a fixed cadence (Contoso reruns monthly during hypercare). Log user feedback with the question, the retrieved chunks and the answer, so a bad answer can be reproduced and added to the gold set.