Data lineage before data science

The kickoff deck for an AI programme always shows a model. The first months of the programme usually contain no model at all. They contain arguments about which system holds the authoritative customer record, and why two reports that should agree do not.

One programme began with a telecommunications operator holding enormous volumes of network and customer data and no reliable way to turn any of it into a decision. The data was there. What was missing was any account of where each field came from, who owned it and whether the copy sitting in the warehouse still resembled the copy in the system that produced it. Six months of that work looks like nothing is happening. It is the part that makes everything afterwards defensible.

Lineage is what you need when someone asks a question later

Lineage at the level that matters is per field: source system, extraction time, every transformation applied, the person accountable for it and the retention rule. It sounds like documentation. It behaves like infrastructure.

The test is a dispute. A customer challenges a decision the system made in March. Someone now has to reconstruct the inputs as they stood at the moment of that decision, not as they stand today. If the feature store is overwritten nightly and no snapshot survives, that reconstruction is impossible, and the honest answer to the regulator is that the organisation cannot explain its own decision. We have seen programmes reach production before anyone tested this, which is a bad week to find out.

If you cannot say where a number came from, you cannot defend the decision it produced.

The failure you cannot debug

Model performance degrades. The useful question is whether the model changed or the data changed, and without lineage there is no way to answer it. An upstream team renames a field, switches a unit from seconds to milliseconds, or starts populating a column that used to be null. Nobody tells the data science team, because the upstream team has no idea anyone downstream is reading it. The symptom appears weeks later as a quiet drop in precision.

A cleverer model does not fix this. A contract on the input does: a schema with an owner, a test that fails the pipeline when the shape or distribution moves outside an agreed band and a named person who gets the alert. Boring, and it removes an entire category of incident that would otherwise be diagnosed by guesswork.

Making the plumbing visible

The real difficulty with a long data phase is political. Steering committees get nervous when a quarter passes with nothing to demonstrate, and a nervous committee asks for a demo. A demo built early on unverified data sets an expectation that is then very expensive to walk back, because everyone in the room has now seen the system work.

We handle this by making the plumbing produce artefacts the business can read and use on its own terms:

• A data dictionary with a named owner against every field, not a team name
• A reconciliation report comparing the warehouse against the source system, run on a schedule and reviewed like any other control
• A lineage graph that shows what breaks if a given source goes away
• A quality dashboard with thresholds agreed in advance, so a failing check is a fact rather than an opinion

These outlive the model. Two years on, the reconciliation report is often the most consulted output of the whole programme, because it is the only place anyone can settle an argument about which number is correct.

The fraud platform we built for a financial institution cut fraudulent transactions by a quarter. The longest single item in that build was aggregating transactions, customer profiles and external databases into one feature store that could be trusted. Model selection and training took weeks. The plumbing was the project. When the ratio comes out that way, it usually means the programme was scoped honestly.