Idempotent Pipeline

An idempotent pipeline produces the same correct end state whether it runs once or five times over the same input. Rerunning yesterday’s load doesn’t double yesterday’s rows; replaying a week after an incident converges on the same tables as if nothing had failed. Idempotency is the property that makes retries, backfills, and CDC replays boring — which is exactly what you want at 3 a.m.

The standard techniques: overwrite deterministic partitions instead of appending, MERGE on stable keys instead of blind inserts, and derive everything from immutable raw inputs (the bronze layer of a medallion architecture) so any table can be rebuilt from history.

Go deeper: How to Make a Data Pipeline Idempotent