This is Lesson 3 of the ERP and CRM reconciliation pipeline series, adapted from the published documentation for the erp-crm-reconciliation-pipeline repository.

What you’ll build

In this tutorial, you tune the ratios that control how messy a generated dataset is, save the result to a configuration file, and confirm that file reproduces the same dataset on every run. Lesson 1’s default dataset already contains some messiness; this lesson makes those defaults deliberate and reviewable instead of implicit.

Before you begin

You need a working datagen setup from Lesson 1. No new dependencies are required.

See it work

Generate a dataset with a much higher bad-date ratio than the default, then look for the effect:

uv run datagen generate --seed 42 --customers 50 --bad-date-ratio 0.5 --output-dir data-messy
Select-String -Path data-messy/erp/invoices.csv -Pattern '\d{2}/\d{2}/\d{4}' | Select-Object -First 5

At the default bad_date_ratio of 0.08, malformed dates are rare. At 0.5, roughly half of the dates in the export come back rewritten as MM/DD/YYYY, DD-MM-YYYY, a Unix timestamp, or an invalid placeholder like 0000-00-00, instead of the clean ISO format most rows use.

Understand the messiness knobs

Each ratio is a number between 0.0 and 1.0 that applies independently, per export, with its own seeded random-number stream, so the ERP and CRM copies of the same underlying row diverge realistically instead of matching each other’s corruption exactly:

FlagDefaultWhat it does
--missing-value-ratio0.05Replaces cells with None, "", "N/A", "NULL", "--", or "unknown"
--bad-date-ratio0.08Rewrites dates as MM/DD/YYYY, DD-MM-YYYY, a Unix timestamp, or an invalid placeholder
--encoding-issue-ratio0.03Corrupts text with mojibake, UTF-8 bytes misread as CP-1252
--duplicate-ratio0.02Appends near-duplicate rows, with whitespace or case perturbed on a non-key column
--orphan-ratio0.05Decides which invoices and payments exist in only one system
--amount-drift-ratio0.04Decides which payments get a legitimate ERP and CRM amount mismatch

The first four are cosmetic messiness, applied by the messiness package after the canonical data already exists. The last two, orphans and amount drift, are structural discrepancies the generators decide while building invoices and payments, because that decision needs knowledge of both systems at once.

Follow real values through the cleaners

The committed fixture contains enough variation to see the difference between repairable messiness and a row that must be quarantined.

Raw valueCleaner resultOutcome
2025/05/112025-05-11Repaired to ISO 8601
17554644002025-08-18Unix timestamp converted to a date
0000-00-00null, with the invalid flag setQuarantined as unparseable_date
TBDnull, with the invalid flag setQuarantined as unparseable_date
N/A in required ISSUE_DTnullQuarantined as missing_required_field

These aren’t hypothetical values. C000004 has raw CREATED_DT value 2025/05/11, INV-000016 has raw ISSUE_DT value 1755464400, and INV-000009 has raw DUE_DT value 0000-00-00 in examples/sample_dataset/erp/.

The encoding cleaner also reverses the exact UTF-8-as-CP-1252 transformation produced by datagen. Running the source cleaner with seed 42 gives:

raw:     Reneé Gárcia
cleaned: Reneé Gárcia

The cleaner removes mojibake. It doesn’t transliterate legitimate accented characters back to ASCII. If a value already contains the Unicode replacement character, the repair is unsafe, so the row receives unrecoverable_encoding instead.

Start from the fully resolved defaults instead of writing a configuration file from scratch:

uv run datagen show-config > my-config.yaml

Open my-config.yaml. It’s a complete, human-readable GenerationConfig, every flag from Lesson 1 and this lesson, with its current value.

Tune the ratios in your config file

  1. Open my-config.yaml and raise bad_date_ratio to 0.3 and missing_value_ratio to 0.15.

  2. Save the file.

  3. Regenerate using the config file:

    uv run datagen generate --config my-config.yaml --output-dir data

Any generate flag also passed on the command line overrides the corresponding value from --config, so you can still do a one-off override without editing the file, for example uv run datagen generate --config my-config.yaml --customers 500.

Confirm reproducibility survives config-driven runs

  1. Delete data/ and rerun the exact same --config my-config.yaml command.
  2. Confirm the regenerated files are byte-identical to the previous run, the same guarantee Lesson 1 demonstrated with plain flags.
  3. Open the regenerated data/generation_config.yaml and confirm it reflects your edited ratios, not the built-in defaults. This is the file to commit alongside a dataset when you need someone else to reproduce it exactly.

Verify your work

  • Run uv run datagen show-config > my-config.yaml, edit bad_date_ratio, and regenerate with --config my-config.yaml.
  • Grep the regenerated erp/invoices.csv for a non-ISO date pattern and confirm malformed dates appear roughly as often as the configured ratio implies.
  • Regenerate from the same config file a second time and confirm the output is byte-identical to the first config-driven run.
  • Run the cleaner tests with uv run pytest tests/test_dates.py tests/test_encoding.py. Confirm the alternate date formats normalize to ISO 8601 and the encoding round trip is reversible.

Troubleshooting

SymptomResolution
A --config value doesn’t seem to applyCheck for a command-line flag with the same name; flags passed alongside --config always override the file.
datagen generate rejects a ratio valueEvery messiness ratio must be between 0.0 and 1.0.
The regenerated dataset looks unchanged after editing the configConfirm the command actually includes --config my-config.yaml.

Next steps

The next lesson runs the reconcile CLI over a dataset like the one you just tuned, and walks through how it resolves ERP records against CRM records without relying on ID-format shortcuts. Continue with Reconcile ERP and CRM records with the pipeline.

For the complete flag list, read the CLI reference in the published documentation.