Topic — 8 essays
Data Engineering
The mechanics of moving data safely: idempotent pipelines, change data capture, and the batch-versus-streaming decision — built for re-runs, retries, and 3 a.m. failures.
Field Notes
Parquet vs ORC: What the Benchmark Actually Showed
Received wisdom says ORC compresses better. Measured at matched codecs on 3M rows, Parquet was 23-26% smaller and faster on every axis. Here's the honest call.
02Measured
Parquet vs ORC vs Avro Benchmark: Compression and Read Speed
Benchmarked on 3 million rows: Parquet was 23-26% smaller than ORC at matched codecs, contradicting the usual claim. Script published, rerun it yourself.
03Field Notes
Merge-on-Read vs Copy-on-Write: The Lakehouse Decision With Numbers
Copy-on-write rewrites whole files on update; merge-on-read annotates them and resolves at read time. The trade is write cost against read cost, with arithmetic.
04Field Notes
What Is Reverse ETL? The Warehouse Learns to Talk Back
Reverse ETL syncs modelled warehouse data back into CRMs, ad platforms, and support tools — so business teams act on it where they work. How it works, and its risks.
05Field Notes
ETL vs ELT: The Same Three Letters in a Different Order
ETL transforms data before loading it; ELT loads raw data first and transforms it in the warehouse. A trivial-sounding reorder that encodes the cloud shift.
06Field Notes
Batch vs Streaming: How to Actually Decide
Batch vs streaming isn't legacy vs modern. The real question is what latency the decision actually requires. Default to batch; promote one use case at a time.
07Field Notes
What Is Change Data Capture (CDC), and When Do You Need It?
Change data capture identifies inserts, updates, and deletes in a source database and delivers them downstream. How the three methods compare, and when batch wins.
08Field Notes
How to Make a Data Pipeline Idempotent
An idempotent pipeline gives the same result whether it runs once or five times. The concrete patterns: partition overwrite, merge on keys, delete-insert.