There are two kinds of data platforms: those designed knowing an auditor will one day ask 'where did this number come from?', and those that answer that question with a war room. Having built platforms for departments where audit is an annual certainty rather than a tail risk, we have learned to treat auditability as an architectural property — like latency or availability — that you either design in or retrofit at ten times the cost.
Lineage is a schema decision, not a tool purchase
Lineage tooling helps, but the foundation is simpler: every derived table carries the identity of the pipeline run that produced it, and every pipeline run records its inputs, code version and timestamps. With that in place, any figure on any dashboard can be traced to source rows mechanically.
-- Every reporting table carries lineage columns
ALTER TABLE fact_disbursements
ADD COLUMN source_system VARCHAR(64) NOT NULL,
ADD COLUMN ingested_at DATETIME NOT NULL,
ADD COLUMN pipeline_run_id CHAR(36) NOT NULL;
-- And every run is recorded, immutably
CREATE TABLE pipeline_runs (
id CHAR(36) PRIMARY KEY,
pipeline VARCHAR(128) NOT NULL,
code_version VARCHAR(64) NOT NULL,
started_at DATETIME NOT NULL,
finished_at DATETIME NULL,
row_counts JSON NOT NULL
);Notice what this buys you: reconciliation reports become joins, not investigations. When a monthly total is challenged, you can show the exact runs, code versions and source counts that produced it.
Make the raw layer immutable
The single most valuable rule we enforce: the landing zone is append-only. Source extracts are never updated or deleted, only superseded. Corrections happen downstream, in versioned transformation code, where every fix is reviewable and re-runnable. When an auditor asks what the source said on 31 March, you show them — because it is still there.
Append-only raw tables with load timestamps and batch identifiers
All corrections in transformation code, never manual UPDATEs
Retention policies written down and enforced by jobs, not memory
Row counts reconciled source-to-report on every run, with alerts on drift
Access control that answers 'who saw what, when'
Role-based access gets you to 'who could see what'. Audits increasingly ask 'who did see what' — which means query logging on sensitive schemas, service accounts that map to named humans, and a quarterly access review that actually revokes things. None of this is difficult; all of it is tedious, which is why it must be automated from the start.
Audit week should be the most boring week of your year. If it is exciting, the platform failed twelve months ago.
The payoff is speed, not just safety
Teams sometimes fear that audit-grade discipline slows delivery. Our experience is the opposite: platforms with lineage, immutable raw layers and reconciled counts are dramatically faster to debug, because every question that starts with 'why does this number look wrong' has a mechanical answer. The discipline you build for the auditor pays rent every single week.
