Profiling source data

Every migration inherits the data habits of the old system. Fields that were optional for fifteen years are half empty. Codes that were retired are still on old records. The same customer was created three times by three clerks. None of this stops the old system working, because people learned to work around it. It does stop the new system working, because SAP S/4HANA applies its checks to every record you load. Profiling is how you find these problems early, measure them and give them to the right owner, before they turn into load errors in a mock or wrong data in production.

Profiling means examining the structure and content of the source data systematically: how complete each field is, which values occur and how often, which records break rules, and which records look like duplicates. It is analysis, not repair. The output is a list of findings with counts, examples and a proposed owner, plus a baseline you can measure progress against.

Our running example in this course is Contoso's Wave 2. Contoso converted its ECC system to SAP S/4HANA, and right after that it acquired Alternadores del Norte S.A. de C.V. in Monterrey, Mexico. Alternadores runs a non-SAP legacy ERP. Its customers, suppliers, products, open items and balances must be loaded into Contoso's converted system as new company code 1910 (MXN) with plant 1910 Monterrey, using the Migration Cockpit and a series of mock loads. That legacy data is what we profile, cleanse and map.

The quality dimensions

Data quality practitioners describe problems along a few standard dimensions. Using them gives every finding a clear category and makes metrics comparable across objects.

What to profile, and against what

Profile with the target in mind. A field that is optional in the legacy system may be mandatory in the Migration Cockpit template for the business partner, the product or an open item, or may need a format that SAP S/4HANA validates, such as a tax number or a postal code for a given country. Start from the migration template of each object: list its mandatory fields, its fields that are checked against configuration (payment terms, plants, units of measure) and the fields the business says matter. Those lists decide which source fields you profile first.

• Column profiles: empty counts, distinct values with frequencies, minimum and maximum, and format patterns (for example how many Mexican postal codes have exactly five digits, and how many lost a leading zero in a spreadsheet).
• Rule checks: business rules written as tests, such as "every active supplier has a bank account" or "every product with stock has a valuation class".
• Cross-table checks: references that must exist, such as open items whose customer is blocked or archived.
• Duplicate candidates: records that match on tax number, or closely on name and address. Fuzzy matching finds candidates; owners decide whether they are real duplicates.

Tools vary. SAP offers SAP Information Steward, whose profiling includes content, uniqueness, redundancy and dependency profiles, and SAP Master Data Governance offers consolidation with duplicate matching for business partner and product data where it is licensed. Many teams also profile extracts with SQL or a data preparation tool in the staging area. The tool matters less than repeatability: profiling must be rerun on every new extract, so script it once and keep the scripts with the mapping specification.

Sorting findings

Not every finding has the same urgency. Some will make records fail to load. Others will load cleanly and still be wrong for the business, which is more dangerous because nothing flags them. A third group is records that should not be migrated at all.