Lesson 3 of 6
Consolidation and mass processing
10 min
Listen to this lesson
Central governance handles one request at a time, created by a person. Two other situations need different tools. Consolidation is for data that arrives from outside: from another system, an acquisition, or a data provider file. The records must be matched against each other and against existing data before anything is activated. Mass processing is for changing many existing records at once, such as new payment terms for a customer group. Both run as governed processes with steps, checks and an audit trail.
The consolidation process
| Step | What happens | Contoso example |
|---|---|---|
| Import | Records loaded from a file or source system with their source identifier, for example with the Import Master Data app | 900 fleet customer contacts exported from the warranty claims app |
| Standardization | Addresses and formats are cleaned so that matching can compare like with like | Street abbreviations and ZIP+4 codes normalized |
| Matching | Fuzzy matching groups records that probably describe the same partner into match groups | 212 contacts matched to existing customers such as Great Lakes Fleet Services |
| Best record calculation | Rules pick the surviving value for each field from the records in a match group | Phone number from the warranty app, tax number from SAP S/4HANA |
| Validation | Data quality and validation rules check the best records | Missing region rejected for 14 records |
| Activation | Best records are created or update existing records; key mapping links the sources | New contacts activated; source IDs kept for traceability |
Consolidation steps and the Contoso warranty contacts example (illustrative course data).
Each step can be configured in a process template, and steps can be scheduled to run in the background once a number of records is reached. Users with the right authorization approve or reject match groups, and a strict four-eyes principle can stop a user who already worked on a process from continuing it. Since SAP S/4HANA 1909 FPS2, data quality rules can be applied in the validation and activation steps, in both consolidation and mass processing.
Matching is a business decision
Matching tools score how similar two records are, but they cannot know whether two Great Lakes depots are the same customer for credit purposes. Set thresholds so that certain matches merge automatically, clearly different records stay apart, and the band in between goes to a steward. Then review the results with the data owner. A wrong merge is worse than a missed duplicate: it can join two customers' open items and credit exposure.
Keep the source identifiers of imported records. They are what lets a steward trace a golden record back to the warranty app contact or the data provider line it came from, and they feed key mapping when the source system later receives the consolidated record. The import formats offered depend on release (files such as CSV or spreadsheet formats, or source systems); record the file name, row count and import date with each process, because the import file is evidence in the same way a migration extract is.
Mass processing
Mass processing changes existing records in bulk through a governed process: select the records, enter or upload the changes, let validation run, review statistics, and activate. Contoso uses it when sales reorganizes the aftermarket territory: 300 aftermarket customers in sales area 1710 / 10 / 00 move to a new sales district. Instead of 300 individual change requests, Maria's team prepares one mass process, Hannah's team reviews the validation results, and the change is activated in one step with a full record of what changed.
- Select records with a query or an uploaded list of keys.
- Change fields directly or upload a file with new values.
- Run validation and review errors and statistics before activation.
- Activate, with the step optionally scheduled for large volumes.
- Keep the process as evidence: who prepared, who approved, what changed.
For large volumes, the matching, validation and activation steps can be scheduled to run in the background once a threshold number of records is reached, which keeps interactive users out of long waits. Plan the timing with the people who use the records: activating 300 changed customers in the middle of the order-entry day can lock records that Maria's team is editing. Contoso runs mass activations after 18:00 Detroit time.
What goes wrong in testing
| Pitfall | Consequence | Prevention |
|---|---|---|
| Matching tested with obvious duplicates only | Thresholds look perfect until real data arrives | Test near-duplicates, different depots, typos and transliterations |
| Best record rules unchecked per field | A worse value survives, for example an old phone number | Expected survivor per field for each test match group |
| Mass change tested on 5 records only | Performance and lock problems at 300+ | Rehearse with production-like volume in CMQ |
| Source identifiers not checked | No traceability back to the warranty app | Verify key mapping after activation |
Typical consolidation and mass processing test failures.
What would you do?
In the first consolidation run, the matching step groups two Great Lakes Fleet Services records: the sold-to 17100001 and a warranty-app contact for the Flint depot. The score is high. A steward wants to merge them so that only one record remains. What should Hannah decide?
- Consolidation: import, standardize, match, best record, validate, activate.
- Matching scores similarity; the business decides what counts as the same partner.
- Mass processing governs bulk changes with validation, statistics and an audit trail.
- Data quality rules can run in the validation and activation steps of both.
Check your understanding · not graded
Lesson check
1. Which consolidation step picks the surviving value for each field in a match group?
2. What does the strict four-eyes principle do in consolidation and mass processing?
3. Which situation calls for mass processing rather than central governance?
4. Why is a wrong merge often worse than a missed duplicate?