The planning problem that is actually a data problem
Every demand planner has lived through a version of this. The forecast is wrong. Not subtly wrong, dramatically wrong. A SKU that sells 500 units a month is forecasted at 5,000. Or a SKU that is in stockout is forecasted at zero. The planner investigates. The model is fine. The actuals are fine. The problem is that the demand history, as it arrived from the ERP, was wrong. A unit conversion was applied twice. A location code was remapped. A discontinued SKU was still active in the item master. The forecast was computed correctly on bad data, and the result was nonsense.
Research on master data management in ERP environments, published across multiple sources including studies indexed on ResearchGate and reports from BARC (the Business Application Research Center), consistently finds that data quality is the single largest predictor of planning system effectiveness. A well-tuned statistical model on clean data outperforms a sophisticated machine learning model on dirty data, every time. The model is not the bottleneck. The data is.
The standard response to this finding is to propose an ERP migration or a master data management (MDM) program. Both are multi-year, multi-million-dollar efforts with high failure rates. This article covers a faster path. The planning tool itself can sit behind a data validation and curation layer that catches and fixes the most common data quality problems before they reach the forecast engine. This does not replace MDM. It makes planning functional while MDM catches up, and in many cases it removes the urgency of MDM entirely.
The four data quality failures that break planning
Across manufacturing planning environments, four categories of data quality failure account for the majority of forecast and inventory errors. All four originate in the ERP and propagate downstream.
Missing and duplicated dates. Demand history arrives with gaps. Some periods are missing entirely, often because of ERP cutover, system upgrades, or data archival. Other periods are duplicated, often because of batch job re-runs or extract-transform-load (ETL) pipeline errors. A forecast model trained on data with missing dates will treat the gaps as zero demand, which biases the forecast low. A model trained on duplicated periods will double-count the demand, which biases the forecast high. Both are common. Both are usually invisible until someone investigates a suspicious forecast.
Unit and conversion errors. Demand is recorded in the ERP in a base unit of measure, often each or case. Planning may need a different unit, often pallet or metric ton. The conversion factor lives in the item master. When the conversion factor is wrong, missing, or applied inconsistently across locations, the demand history that arrives in the planning tool is off by the conversion factor. A SKU measured in cases but planned in each, with a conversion factor of 24, will have its demand understated by a factor of 24 if the conversion is not applied, or overstated by a factor of 24 if applied twice. This is not a subtle error. It produces forecasts that are off by orders of magnitude, and it happens more often than anyone admits.
Item master inconsistencies. The item master is supposed to be the single source of truth for SKU attributes: description, category, lead time, supplier, unit of measure. In practice, the item master in most ERPs is maintained by multiple people across multiple locations, and inconsistencies accumulate. The same SKU has different descriptions in different plants. Lead times are populated for some SKUs and blank for others. Discontinued SKUs are still flagged as active. New SKUs are missing attributes. The forecast model does not care about most of these attributes, but inventory policy, supply netting, and reporting all depend on them.
Lead time errors. Lead time is the input to safety stock calculation, reorder point computation, and supply netting. It is also one of the most poorly maintained fields in most ERP systems. The planned lead time is often set once, at SKU creation, and never updated. The actual lead time, computed from purchase order receipts, may be 50% longer or shorter. Safety stock computed against a planned lead time of 14 days, when actual lead time is 21 days, will systematically understock. This is not a forecast error. It is a data error. And it is invisible because the safety stock formula produces a confident-looking number regardless.
The data validation layer
The fix is a data validation layer between the ERP and the planning tool. This layer does four things, each addressing one of the failure modes above.
Date integrity checks. For every demand history extract, the layer checks that every expected period is present exactly once. Missing periods are flagged and backfilled with zero (with a flag indicating the backfill, so the forecast model can discount them). Duplicated periods are flagged and the duplicate is removed. The check is simple but it must run on every extract, not just at implementation. Demand history is not static. New periods arrive every month, and the arrival process is where errors are introduced.
Unit consistency checks. For every SKU, the layer validates the unit of measure against a canonical mapping. If the ERP reports demand in cases and the planning tool expects each, the conversion is applied once, at the validation layer, with the conversion factor verified against the item master. If the conversion factor is missing or inconsistent across locations, the SKU is flagged and the demand is held until a human resolves the inconsistency. This is slower than letting bad data through. It is faster than explaining a 24x forecast error to the executive team.
Item master validation. The layer cross-checks the item master against a reference taxonomy. SKUs with missing required attributes (lead time, supplier, category) are flagged. SKUs with inconsistent attributes across locations are surfaced for resolution. Discontinued SKUs that still have demand are flagged for investigation. The validation does not fix the item master. It identifies the problems so a human can fix them, and it prevents unvalidated SKUs from contaminating the forecast.
Lead time reconciliation. The layer computes actual lead time from purchase order receipts (receipt date minus PO release date) and compares it to the planned lead time in the item master. Discrepancies above a threshold (typically 20%) are flagged. The planned lead time is not automatically updated, because actual lead time can be noisy and a single bad receipt can skew it. But the flagging ensures that lead time drift is visible and can be reviewed regularly, rather than discovered years later when someone investigates a stockout.
Idempotent imports: the operational property that makes this work
The data validation layer only works if imports are idempotent. Idempotent means that running the same import twice produces the same result as running it once. If you upload the same demand history file twice, the second upload does not duplicate the data. It either updates the existing records or no-ops, depending on whether the data has changed.
Idempotency is not a luxury. It is a requirement for safe data operations. Without it, every re-upload, every re-run of an ETL job, every retry of a failed import risks duplicating data. The team learns to fear re-uploads and avoids them, which means data corrections are deferred and the data quality problem gets worse over time.
Idempotent imports require the import system to identify records by a stable key (a hash of the natural key, or a combination of SKU, location, date, and source) and to use upsert (update or insert) semantics rather than insert. The import system must also handle staging and validation before commit, so that a partially-valid file does not partially land in the database. Errors must be reported per-row, with enough context to fix the row and re-upload.
This is engineering work, not data science work. But it is the engineering work that makes the data science work.
What this does not replace
The data validation layer does not replace master data management. MDM is about governance, ownership, and process for creating and maintaining master data. The validation layer is about catching the errors that governance does not prevent. Both are needed. The validation layer is faster to implement (weeks to months, not years) and produces immediate planning improvement. MDM is slower (years) and produces long-term sustainability.
If you are choosing where to invest first, invest in the validation layer. It will surface the data quality problems that MDM needs to address, with specific examples and evidence. It will make planning functional while MDM catches up. And in many cases, the validation layer plus a disciplined monthly data review process is sufficient, and the full MDM program can be scoped down or deferred.
The takeaway
Most planning failures are data failures. The data failures originate in the ERP and propagate downstream. The fix is not a new ERP and not necessarily a multi-year MDM program. The fix is a data validation layer that checks date integrity, unit consistency, item master completeness, and lead time accuracy on every import, and an import system that is idempotent so corrections can be applied safely.
Build the validation layer first. It is the highest-return investment in planning accuracy available. Then decide whether MDM is still needed. Often, the answer is yes but less urgently, because the validation layer has made planning functional and the MDM program can proceed at a sustainable pace rather than under crisis conditions.