Compare ERP and CRM field mappings and the ground truth answer key

This is Lesson 2 of the ERP and CRM reconciliation pipeline series, adapted from the published documentation for the erp-crm-reconciliation-pipeline repository.

What you’ll build

In this tutorial, you build a mental model instead of running new code: the field mapping between the ERP and CRM exports from Lesson 1, and the structure of the ground_truth.json answer key. Both matter before Lesson 4 asks a pipeline to resolve these two exports back into a single source of truth.

Before you begin

You need a generated dataset. Either the one from Lesson 1, or the small, committed examples/sample_dataset (eight customers, --seed 1), works for this lesson.

See it work

Open the same customer in both exports. In PowerShell:

Get-Content examples/sample_dataset/erp/customers.csv | Select-Object -First 2
Get-Content examples/sample_dataset/crm/customers.json | Select-Object -First 20

The first data row in the CSV and the first customer object in the JSON describe the same underlying person, but nothing about their business keys looks alike. That’s the point: both are projections of one internal, seeded canonical entity, deliberately re-encoded per system.

Here are the exact values from the committed seed-1 fixture:

Ignore the business-key pattern and compare the evidence a real migration would have. The name is identical, the normalized emails are identical, and the normalized phone numbers are identical. Those three independent signals make this a strong candidate pair before you know the answer key.

Why two divergent exports instead of one clean dataset

Real ERP and CRM systems almost never agree on field names, ID formats, or structure. datagen reproduces that divergence on purpose so a reconciliation pipeline is tested against the same kind of mismatch a real legacy migration faces:

Both exports are projections of the same internal, seeded canonical entities, so a correct reconciliation pipeline should be able to match them back together despite the different formats.

Compare customer field mapping

Notice that full_name, email, and phone, not the business key, are the fields that actually correlate the same person across both exports. Lesson 4’s entity resolution step matches on exactly these signals.

Compare invoice and payment field mapping

Invoices and payments follow the same pattern, nested one level deeper in the CRM export:

Payments add one field that has no equivalent on the CRM side: erp_amount_drift. It lets the ERP-recorded amount legitimately differ from the CRM, or true, amount, simulating bank fees or foreign-exchange rounding that a real reconciliation pipeline must detect rather than silently average away.

An invoice or payment that exists in only one system is a genuine, valid outcome, not a data error: both exporters simply omit the row or entry when exists_in_erp or exists_in_crm is False.

Read the ground truth answer key

With --with-ground-truth, ground_truth.json records, for every invoice and payment, whether it exists in ERP, CRM, or both (expected_match), its business key in each system (or null if absent), and, for payments, the exact amount_drift applied. A summary block totals orphan and amount-mismatch counts.

For example, the first invoice entry says that INV-000001 and INV000001 should match:

{
  "erp_key": "INV-000001",
  "crm_key": "INV000001",
  "exists_in_erp": true,
  "exists_in_crm": true,
  "expected_match": true
}

The second payment demonstrates the opposite case:

{
  "erp_key": "PMT-000002",
  "crm_key": null,
  "exists_in_erp": true,
  "exists_in_crm": false,
  "expected_match": false,
  "amount_drift": 0.0,
  "has_amount_mismatch": false
}

That payment is an ERP-only orphan. A correct pipeline keeps it with a null CRM key. It doesn’t quarantine it or force it onto the nearest CRM payment.

This file is the answer key you score a downstream reconciliation pipeline against, and Lesson 5 covers exactly that. It’s never read by the pipeline itself: reconcile doesn’t import, read, or reference ground_truth.json under any circumstance. Only eval/score.py, or tests, are allowed to read it.

The hard constraint: no ID-format shortcuts

CUST_ID and customerId are cosmetically different encodings of the same sequence number, but only because this is a synthetic dataset. A real reconciliation pipeline never gets to assume that convenient correlation, so entity resolution in this project never parses, strips, or reverses business-key formats to align records across systems. Instead, it matches customers on name, email, and phone. Within a resolved customer, it matches invoices on exact issue date, currency, and amount. Within a resolved invoice, it matches payments on exact payment date and measures amount drift separately. Lesson 4 builds on this constraint directly.

Verify your work

• In examples/sample_dataset, pick one customer row in erp/customers.csv and find the matching object in crm/customers.json by name, email, or phone.
• Open examples/sample_dataset/ground_truth.json and find one invoice entry. State whether it’s expected in ERP only, CRM only, or both.
• Explain, in your own words, why a pipeline that normalizes digits out of CUST_ID and customerId to match them would be cheating relative to a real migration.

Troubleshooting

Next steps

The next lesson tunes the messiness knobs that decide how many cells get corrupted, how many rows go missing, and how many duplicates appear, and saves those choices to a reusable configuration file. Continue with Customize messiness and build a reproducible configuration file.

For the full field-by-field reference, read the data model reference in the published documentation.