Why Data Pipelines Break in Production
Data pipelines fail for reasons that application code rarely encounters. Source systems change their schemas without notice — a column gets renamed, a new required field is added, a date format changes from ISO 8601 to MM/DD/YYYY. External APIs rate-limit or go down at inconvenient times. Source data contains values that your pipeline's type assumptions can't handle: a currency field that occasionally contains text, a user ID column that sometimes contains negative numbers.
The difference between a fragile pipeline and a reliable one isn't the technology — it's whether the engineers who built it anticipated these failure modes and built defences against them. A pipeline without schema validation, retry logic, and monitoring is not a data asset; it's a liability waiting to produce a silently incorrect metric.
Schema Evolution and How to Handle It
The most common pipeline failure mode is schema drift: the source system adds, removes, or renames a column, and the pipeline either breaks or silently drops data. The defence is explicit schema contracts at the ingestion boundary.
For database replication using tools like Fivetran or Airbyte, schema drift is handled automatically — the tool detects changes and updates the destination schema, with configurable alerting when breaking changes occur. For API-based ingestion, the contract is typically enforced by validating the response payload against an expected schema using Pydantic (Python) or JSON Schema before writing to the warehouse. A validation failure triggers an alert, not a silent truncation.
Idempotency: The Property Your Pipelines Must Have
An idempotent pipeline produces the same output regardless of how many times it runs for a given input. If an Airflow task fails and is retried, an idempotent task produces correct results without duplicating data. This sounds obvious but is violated by a surprisingly large number of production pipelines.
The implementation pattern is to use upserts rather than inserts at the load step: if a row with a given key already exists, update it; if it doesn't, insert it. For dbt models, the incremental materialisation strategy with a unique key achieves the same effect. Pipelines that violate idempotency produce duplicate data on retry — a class of data quality issue that is difficult to detect and expensive to remediate.
Observability: You Can't Fix What You Can't See
A production data platform without observability is flying blind. Minimum viable observability for a data pipeline includes: task-level success/failure alerts from Airflow, row count checks that alert when a table grows or shrinks by more than a configurable threshold, freshness checks that alert when a table hasn't been updated within the expected window, and a lineage graph that shows which downstream models and dashboards are affected when a given table is stale or incorrect.
Tools like Monte Carlo, Datafold, or dbt's built-in source freshness checks provide much of this out of the box. The investment in setting them up — typically two to three days of engineer time — pays back in the first incident where a metric discrepancy is caught by automated monitoring rather than by an analyst asking why a dashboard looks wrong.
Incident Response for Data Pipelines
When a data pipeline fails in production, the response process matters as much as the fix. A data incident is distinct from an application incident: the users affected are often internal stakeholders who may not even know that a pipeline exists, and the impact — an incorrect metric, a stale dashboard — may not be immediately visible.
A mature data team has documented runbooks for common failure modes: what to do when the dbt run fails on a specific model, how to re-trigger a backfill for a given date range, how to roll back a bad schema migration. These runbooks don't need to be long — a five-bullet checklist that a junior engineer can follow at 9pm is more valuable than a comprehensive document that nobody reads.