Topic — 7 essays
Data Engineering
The mechanics of moving data safely: idempotent pipelines, change data capture, and the batch-versus-streaming decision — built for re-runs, retries, and 3 a.m. failures.
Measured
Parquet vs ORC vs Avro: I Ran the Benchmark, and One Result Surprised Me
Real numbers on 3 million rows: compression, write time, and column-pruned reads across Parquet, ORC, Avro, and CSV. The script is published; rerun it.
02Field Notes
Merge-on-Read vs Copy-on-Write: The Lakehouse Decision With Numbers
Copy-on-write rewrites whole files on update; merge-on-read annotates them and resolves at read time. The trade is write cost against read cost, with arithmetic.
03Field Notes
What Is Reverse ETL? The Warehouse Learns to Talk Back
Reverse ETL syncs modelled warehouse data back into CRMs, ad platforms, and support tools — so business teams act on it where they work. How it works, and its risks.
04Field Notes
ETL vs ELT: The Same Three Letters in a Different Order
ETL transforms data before loading it; ELT loads raw data first and transforms it in the warehouse. A trivial-sounding reorder that encodes the cloud shift.
05Field Notes
Batch vs Streaming: How to Actually Decide
Batch vs streaming isn't legacy vs modern. The real question is what latency the decision actually requires. Default to batch; promote one use case at a time.
06Field Notes
What Is Change Data Capture (CDC), and When Do You Need It?
Change data capture identifies inserts, updates, and deletes in a source database and delivers them downstream. How the three methods compare, and when batch wins.
07Field Notes
How to Make a Data Pipeline Idempotent
An idempotent pipeline gives the same result whether it runs once or five times. The concrete patterns: partition overwrite, merge on keys, delete-insert.