Notes on the architecture of data
Designing the structures that make data trustworthy.
Essays and field notes on data architecture, data modeling, dimensional modeling, data contracts, and the lakehouse — practical writing for data engineers and architects who design systems that make information trustworthy.
Most read start with these
Field Notes
How to Choose an Iceberg Catalog: Unity vs Polaris vs Glue vs Nessie
The table format war is settled; the catalog decides governance and lock-in now. How Iceberg catalogs work, compared honestly — and a decision rule that holds.
★Field Notes
One Big Table vs the Star Schema: The Real Trade-off
One Big Table denormalizes everything into one wide table; the star schema keeps facts and dimensions apart. What each costs, and why the answer is usually both.
★Field Notes
Kimball vs Inmon: Two Ways to Build a Data Warehouse
Kimball and Inmon differ on one decision: build dimensional marts bottom-up, or a normalized enterprise warehouse top-down. The trade-off, and the modern blend.
★Field Notes
Star Schema vs Snowflake Schema: Which to Use and When
Star schema vs snowflake schema comes down to one decision: whether you normalize your dimensions. The difference, a worked example, and why the star wins.
★Field Notes
Fact Table Types: Transaction, Periodic Snapshot, and Accumulating
Three kinds of fact table, distinguished by what one row represents over time: transaction, periodic snapshot, and accumulating snapshot. How to pick one.
★Field Notes
Apache Iceberg vs Parquet: They're Not the Same Layer
Parquet is a file format, Iceberg is a table format that manages Parquet files. The difference, what Iceberg costs, and how to migrate without rewriting.
Essays & field notes 50 pieces
Field Notes
Apache Polaris vs Nessie: The Merger That Never Happened
Polaris and Nessie were announced as merging in 2024. They didn't. Here's where each actually stands in 2026, and which one your estate should run.
02Field Notes
Unity Catalog vs AWS Glue Data Catalog: How to Choose
Glue is the zero-effort default inside AWS; Unity Catalog buys deeper governance for platform commitment. Both now speak Iceberg REST, and they federate.
03Field Notes
Parquet vs ORC: What the Benchmark Actually Showed
Received wisdom says ORC compresses better. Measured at matched codecs on 3M rows, Parquet was 23-26% smaller and faster on every axis. Here's the honest call.
04Measured
Parquet vs ORC vs Avro Benchmark: Compression and Read Speed
Benchmarked on 3 million rows: Parquet was 23-26% smaller than ORC at matched codecs, contradicting the usual claim. Script published, rerun it yourself.
05Field Notes
Merge-on-Read vs Copy-on-Write: The Lakehouse Decision With Numbers
Copy-on-write rewrites whole files on update; merge-on-read annotates them and resolves at read time. The trade is write cost against read cost, with arithmetic.
06Field Notes
Knowledge Graph vs Data Warehouse: Aggregation vs Connection
A warehouse answers how much; a knowledge graph answers how things connect. They're different query shapes, not rivals — and most serious estates end up with both.
07Field Notes
Ontology vs Data Model: Meaning vs Structure
A data model shapes data for one system; an ontology defines what concepts mean, independent of any system — and machines now read your schema.
08Field Notes
Parquet vs ORC vs Avro: Which Data File Format, and When
Parquet and ORC are columnar formats for analytics; Avro is row-based for streaming and schema evolution. A clear comparison and a decision rule that holds.
09Field Notes
Apache Iceberg vs Parquet: They're Not the Same Layer
Parquet is a file format, Iceberg is a table format that manages Parquet files. The difference, what Iceberg costs, and how to migrate without rewriting.
10Field Notes
What Is Reverse ETL? The Warehouse Learns to Talk Back
Reverse ETL syncs modelled warehouse data back into CRMs, ad platforms, and support tools — so business teams act on it where they work. How it works, and its risks.
11Field Notes
How to Build a Data Catalog That People Actually Use
A tool-neutral guide to building a data catalog: start from query logs, document the fifty tables that matter, assign owners, automate the rest — in that order.
12Field Notes
The Inmon Methodology: The Corporate Information Factory
Inmon's method builds one normalized enterprise warehouse first, then serves dimensional marts from it. How the CIF works, a worked example, and where it fits today.
13Field Notes
Data Observability vs Data Quality: What Each One Actually Catches
Data quality checks the rules you wrote; data observability watches for the failures you didn't anticipate. Neither replaces the other — here's the honest split.
14Field Notes
How to Choose an Iceberg Catalog: Unity vs Polaris vs Glue vs Nessie
The table format war is settled; the catalog decides governance and lock-in now. How Iceberg catalogs work, compared honestly — and a decision rule that holds.
15Field Notes
Iceberg vs Delta Lake: How to Actually Choose in 2026
Iceberg and Delta Lake have converged on capabilities — ACID, time travel, deletion vectors. The real decision is your platform and catalog, not the format.
16Field Notes
The Grain of a Fact Table: The First Modeling Decision
The grain is the business definition of what one fact table row represents. Declare it first — every dimension, measure, and bug traces back to it.
17Field Notes
What Is an Open Table Format? Iceberg, Delta, and Hudi Explained
An open table format is a metadata spec that turns raw files in object storage into real tables — with ACID transactions, schema evolution, and time travel.
18Field Notes
Normalization vs Denormalization: When Each Wins
Normalization splits data into many tables to remove redundancy; denormalization combines them to remove joins. One favors writes, the other reads. The trade.
19Field Notes
Data Lake vs Lakehouse: What Changed and Which to Use
A data lake stores raw files cheaply and guarantees nothing. A lakehouse adds a table layer over the same files for ACID, schema, and trust. When each fits.
20Field Notes
Data Engineer vs Data Architect vs Analytics Engineer
Three overlapping data roles, three different jobs: engineers move the data, analytics engineers shape it, architects design the system. The real division.
21Field Notes
ETL vs ELT: The Same Three Letters in a Different Order
ETL transforms data before loading it; ELT loads raw data first and transforms it in the warehouse. A trivial-sounding reorder that encodes the cloud shift.
22Field Notes
Kimball vs Inmon: Two Ways to Build a Data Warehouse
Kimball and Inmon differ on one decision: build dimensional marts bottom-up, or a normalized enterprise warehouse top-down. The trade-off, and the modern blend.
23Field Notes
One Big Table vs the Star Schema: The Real Trade-off
One Big Table denormalizes everything into one wide table; the star schema keeps facts and dimensions apart. What each costs, and why the answer is usually both.
24Field Notes
Batch vs Streaming: How to Actually Decide
Batch vs streaming isn't legacy vs modern. The real question is what latency the decision actually requires. Default to batch; promote one use case at a time.
25Field Notes
What Is Data Lineage, and What Is It Actually For?
Data lineage records where data comes from, how it's transformed, and where it goes. What it's genuinely for, impact analysis and root cause, and its limits.
26Field Notes
What Is Change Data Capture (CDC), and When Do You Need It?
Change data capture identifies inserts, updates, and deletes in a source database and delivers them downstream. How the three methods compare, and when batch wins.
27Reconsidered
Is the Modern Data Stack Dead?
The modern data stack isn't dead, but the era of bolting together a dozen best-of-breed SaaS tools is ending. What's actually happening, and what to keep.
28Field Notes
What Is a Vector Database, and Do You Need One?
A vector database stores embeddings and finds items by meaning rather than exact match. What that's for, how it fits your stack, and when a dedicated one pays.
29Field Notes
Data Vault vs Dimensional Modeling: Which Belongs Where
Data Vault and dimensional modeling aren't rivals; they solve different problems at different layers. What Data Vault costs, and when it's worth paying for.
30Essay
What Does a Data Architect Actually Do?
A data architect's job isn't drawing diagrams or picking tools. It's deciding how data is shaped, defined, owned, and trusted, then defending those decisions.
31Field Notes
What Is a Data Catalog, and Do You Need One?
A data catalog is a searchable inventory of your data: what exists, where it lives, what it means, who owns it. What it's for, and when you actually need one.
32Essay
Most Data Quality Problems Are Org-Chart Problems
When data is wrong, teams buy a tool. But the durable causes are organizational: unclear ownership, misaligned incentives, no accountability. Tools miss those.
33Field Notes
Fact Table Types: Transaction, Periodic Snapshot, and Accumulating
Three kinds of fact table, distinguished by what one row represents over time: transaction, periodic snapshot, and accumulating snapshot. How to pick one.
34Essay
Your AI Is Only as Good as Your Data Architecture
RAG, AI agents, and LLMs querying your warehouse are only as reliable as the data beneath them. GenAI raises the price of getting the architecture wrong.
35Reconsidered
What GenAI Actually Changes About Data Architecture
GenAI adds vector storage and new retrieval patterns to the data stack, but structure, quality, governance, and ownership matter more than ever, not less.
36Field Notes
Slowly Changing Dimensions, Explained Without the Jargon
Slowly changing dimensions answer one question: when an attribute changes, do you overwrite history or preserve it? SCD Types 1, 2, and 3, and when to use each.
37Field Notes
Data Warehouse vs Data Lake vs Lakehouse: A Clear Comparison
A data warehouse stores structured, modeled data. A data lake stores raw data cheaply. A lakehouse tries to be both. A side-by-side, and how to choose.
38Field Notes
How to Make a Data Pipeline Idempotent
An idempotent pipeline gives the same result whether it runs once or five times. The concrete patterns: partition overwrite, merge on keys, delete-insert.
39Essay
What Is a Semantic Layer, and Why Does Your Data Stack Need One?
A semantic layer is the single governed place where business metrics are defined once, independent of any dashboard. What it is, and what it actually fixes.
40Reconsidered
The Medallion Architecture, Reconsidered
Bronze, silver, gold is a useful default and a dangerous dogma. A second look at what the layers get right, and where they quietly fall apart.
41Field Notes
Factless Fact Tables, Explained
A factless fact table records that an event happened, or could have, without any numeric measure. Why a fact table with no facts is useful, and the two types.
42Field Notes
OLTP vs OLAP: Why You Shouldn't Run Analytics on Your App Database
OLTP handles many small transactions fast; OLAP scans huge volumes for analysis. Opposite optimizations, which is why analytics on production is a trap.
43Field Notes
What Are Conformed Dimensions, and Why Do They Matter?
A conformed dimension is one dimension shared identically across several fact tables, so different business processes can be compared on the same terms.
44Essay
Data Contracts Are a Cultural Problem
A schema check is the easy 10% of a data contract. The other 90% is an organizational agreement that no YAML file can enforce for you.
45Field Notes
The Date Dimension: How to Build One and Why You Need It
A date dimension is a table with one row per calendar day, pre-loaded with every attribute of that day. Why every warehouse needs one, and how to build it.
46Field Notes
Star Schema vs Snowflake Schema: Which to Use and When
Star schema vs snowflake schema comes down to one decision: whether you normalize your dimensions. The difference, a worked example, and why the star wins.
47Field Notes
Fact Table vs Dimension Table: The Core Distinction
A fact table stores the measurements you analyze; a dimension table stores the context you analyze them by. The distinction every dimensional model rests on.
48Field Notes
A Field Guide to Dimensional Modeling
Facts, dimensions, and grain — the three ideas that quietly run most analytics, explained without the dogma.
49Field Notes
Surrogate Keys vs Natural Keys: A Practical Rule
Surrogate key vs natural key is a decision every data model faces. The practical rule: surrogate keys for dimensions, natural key kept as an attribute.
50Manifesto
The Shape of Data
Every dataset has a shape. The only question is whether you chose it, or whether it happened to you.