Lesson 5 of 5
Validating analytics figures
9 min
If a sales manager makes a decision on a wrong number, the system has failed even if every document is correct. Analytics therefore need their own test: reconciliation of the figure shown in the app against the documents underneath it, for a selection small enough to check by hand.
The reconciliation method
- Define the selectionFor example: sales org 1710, sold-to Great Lakes Fleet Services, orders created June 1–15
- Read the app figureIncoming Sales Orders – Flexible Analysis: net value and number of orders
- List the documentsSame selection in a document list such as VA05 or Manage Sales Orders
- Add them upTotal net value and count from the list
- Compare and explainEqual → evidence; different → find each difference and its reason
The selection must be identical on both sides: same organizational filter, same customer, same date field and range, same document types. Most "differences" turn out to be selection differences: the app counted by creation date, the list by requested delivery date; the app excluded rejected items, the list included them.
Where differences come from
| Source of difference | Example | How to check |
|---|---|---|
| Date basis | App uses creation date; list uses document date | Compare the date field each side uses |
| Document types included | Returns and credit memos reduce sales volume in the app but not in a simple order list | Check which types and categories the query includes |
| Rejected or cancelled items | Items with a reason for rejection excluded in the app | Filter the list the same way |
| Currency conversion | App converts to display currency at a different date | Compare in document currency first |
| Authorizations | Tester sees only sales org 1710; the app user sees all | Run both sides as the same user |
| Timing | Documents created between the two reads | Use a closed period or freeze test data |
Common reasons why an analytics figure differs from the document list.
- Are the selections identical, including the date field?Yes → compare totals · No → align the selection first
- Do totals match in document currency?Yes → check display currency · No → compare document by document
- Is the difference explained by excluded types or rejected items?Yes → document the rule; it is a definition, not a defect · No → continue
- Is a document missing or double-counted in the app?Yes → raise a defect with the document number · No → escalate to the analytics owner
Validating KPI tiles and thresholds
KPI tiles add two more things to verify: the target and the thresholds that turn the tile green, amber or red. For a tile showing the share of orders blocked for credit, check that the figure reconciles, that the target is the one the business agreed, and that the color changes at the agreed thresholds. Test this with prepared data: create enough blocked orders to cross a threshold and see whether the tile turns red.
| Check | Evidence for sign-off | Owner |
|---|---|---|
| Figure reconciles to documents | App figure, document list and totals for a defined selection | Key user (Maria) |
| Definition matches business meaning | Written definition of the measure, date basis and inclusions agreed | Process owner |
| Authorizations restrict correctly | Same figure seen by two roles with expected differences | Security and key user |
| Thresholds and targets correct | Tile color before and after crossing a threshold | Sales manager |
| Drill-down reaches documents | Navigation from figure to document list works | Tester |
Pick selections that exercise the tricky cases: a customer with a return in the period, a month that contains a cancelled invoice, an order with a rejected item, and, in a multi-currency group, a sales organization in another currency. A reconciliation that only covers clean, single-currency orders proves little about the cases that cause arguments later.
What would you do?
During UAT, Maria finds the Sales Volume app shows $48,000 for Great Lakes Fleet Services in June, but the invoices she lists total $52,000. On closer inspection there is a $4,000 credit memo in June. What should she conclude?
- Reconcile every key figure against the documents behind it for a small, defined selection.
- Align selections first: organization, customer, date field and range, document types, user.
- Most differences come from definitions (date basis, included types, rejections, currency), not defects.
- Validate KPI targets and thresholds with prepared data that crosses them.
- Evidence: the selection, the app figure, the document list total and who ran it when.
Check your understanding · not graded
Lesson check
1. Why reconcile sales analytics to documents during testing?
2. What causes most reconciliation differences?
3. How do you test a KPI threshold?
4. What is good evidence for an analytics test?