Lesson 4 of 6
Duplicate checks and consolidation results
10 min
Listen to this lesson
A duplicate check has two ways to fail. It can miss a real duplicate, and the master data decays again. Or it can raise false alarms, and users learn to click past the warning. Testing a duplicate check means measuring both, with test data built for the purpose. The same thinking applies to consolidation, where matching decides which records become one.
How the duplicate check works
In central governance, the duplicate check runs when a request is checked or submitted. It searches existing records for similar ones, typically with a fuzzy search on name and address fields, scores the candidates, and compares the score with configured thresholds. Above one threshold, MDG shows possible duplicates for the user to review; depending on configuration, a higher score can block the request. The configuration names the fields, the search and match settings, and the thresholds per entity type. If the search depends on an index or search model, it must be built and current in the test system, or the check finds nothing at all.
- 1. Request checkedLucía submits a new customer for Monterrey
- 2. SearchFuzzy search on name, street, city, postal code and tax number
- 3. ScoreEach candidate gets a similarity score
- 4. Compare thresholdsBelow the lower threshold: no hit · between: shown for review · above the upper: shown or blocked, per configuration
- 5. DecideRequester or steward confirms it is new, or uses the existing record
Build a labeled pair set
The key test asset is a labeled pair set: pairs of records where the business has decided in advance whether they are the same partner. Include exact copies, typical variations and deliberate traps. Wei generates the set from anonymized Contoso patterns so it can be loaded fresh for every test cycle.
| Pair type | Example (illustrative) | Label |
|---|---|---|
| Exact copy | Flotillas Monterrey S.A. de C.V. twice, same address | Duplicate |
| Abbreviation | "Flotillas Monterrey SA de CV" versus the full legal form | Duplicate |
| Typo | "Flotilas Monterey" with the same tax number | Duplicate |
| Different depot | Great Lakes Fleet Services sold-to 17100001 versus its Flint depot ship-to 17100007 | Not a duplicate |
| Same name, other city | A different fleet company with the same trade name in Toledo | Not a duplicate |
| Missing field | Same company without postal code | Duplicate |
Pair types for a duplicate check test set.
Report two numbers to the data owner: the share of real duplicates detected, and the share of warnings that were false alarms. Moving the threshold trades one for the other, so the choice is a business decision. At Contoso, Hannah accepts a few false alarms for customers, because a missed duplicate splits credit exposure, but wants almost none for suppliers, where Omar's team creates many similar Lakeside entities on purpose. These are illustrative tuning goals.
Run the pair set through every entry point that performs a duplicate check, not just the single-object request. Depending on configuration, checks can also run in multiple-record requests, in consolidation and in mass processing, with different settings. A pair caught in a single request but missed in a multiple-record request is a real gap, because requesters will find the path of least resistance. Measure the response time of the check on production-like data volumes as well: a check that takes 20 seconds per request will be switched off by popular demand.
Testing consolidation outcomes
Consolidation tests extend the same idea from pairs to groups. For each test source file, the expected result lists the match groups, the expected survivor value for each field (best record), and which records should be created new. Compare actual match groups with expected ones, then check the best record field by field.
| Check | Expected (example) | Evidence |
|---|---|---|
| Match groups | Warranty contact W-0412 grouped with customer 17100001 | Group list exported from the process |
| Best record: phone | Value from the warranty app (newer) | Field comparison sheet |
| Best record: tax number | Value from SAP S/4HANA (verified) | Field comparison sheet |
| Activation | Existing customer updated, no new customer created | Business partner change documents |
| Key mapping | Source ID W-0412 linked to 17100001 | Key mapping display for the source system |
Expected results for one consolidation test group (illustrative).
What goes wrong in testing
- Search index not built in CMQ. Every duplicate test passes as "no duplicates found." Check the index first with a known exact copy.
- Only exact copies tested. The check looks perfect; real duplicates are typos and abbreviations.
- Test data from previous cycles. Old test records become duplicates of new ones and distort counts. Refresh or isolate test data.
- Survivor rules checked on one field. Wrong survivor values hide in fields nobody looked at.
What would you do?
In the first CMQ cycle, the duplicate check finds nothing for any of the 60 pairs, including exact copies. The configuration looks correct. What should the tester check first?
- Duplicate checks fail in two directions: missed duplicates and false alarms.
- Use a labeled pair set with typos, abbreviations and traps such as depots.
- Report detection and false alarm shares; the owner picks the thresholds.
- Consolidation tests check groups, survivor values per field, activation and key mapping.
Check your understanding · not graded
Lesson check
1. What is the key test asset for a duplicate check?
2. Which pair is a trap that should NOT be flagged as a duplicate?
3. What happens to false alarms when the duplicate threshold is lowered?
4. In a consolidation test, what is checked after the match groups?