Notes on the architecture of data
Designing the structures that make data trustworthy.
Essays and field notes on data architecture, data modeling, dimensional modeling, data contracts, and the lakehouse — practical writing for data engineers and architects who design systems that make information trustworthy.
Essays & field notes 38 pieces
Field Notes
Data Observability vs Data Quality: What Each One Actually Catches
Data quality checks the rules you wrote; data observability watches for the failures you didn't anticipate. Neither replaces the other — here's the honest split.
02Field Notes
How to Choose an Iceberg Catalog: Unity vs Polaris vs Glue vs Nessie
The table format war is settled; the catalog decides governance and lock-in now. How Iceberg catalogs work, compared honestly — and a decision rule that holds.
03Field Notes
Iceberg vs Delta Lake: How to Actually Choose in 2026
Iceberg and Delta Lake have converged on capabilities — ACID, time travel, deletion vectors. The real decision is your platform and catalog, not the format.
04Field Notes
The Grain of a Fact Table: The First Decision That Decides Everything Else
The grain is the business definition of what one fact table row represents. Declare it first — every dimension, measure, and bug traces back to it.
05Field Notes
What Is an Open Table Format? Iceberg, Delta, and Hudi Explained
An open table format is a metadata spec that turns raw files in object storage into real tables — with ACID transactions, schema evolution, and time travel.
06Field Notes
Normalization vs Denormalization: When Each Wins
Normalization splits data into many tables to remove redundancy; denormalization combines them to remove joins. One favors writes, the other reads. Here's the trade-...
07Field Notes
Data Lake vs Lakehouse: What Changed and Which to Use
A data lake stores raw files cheaply but offers no guarantees. A lakehouse adds a table layer over those same files to give them ACID transactions, schema, and relia...
08Field Notes
Data Engineer vs Data Architect vs Analytics Engineer: Who Does What?
Three overlapping data roles, three different jobs. Data engineers move the data, analytics engineers shape it for analysis, data architects design the system it all...
09Field Notes
ETL vs ELT: The Same Three Letters in a Different Order
ETL transforms data before loading it into the warehouse; ELT loads raw data first and transforms it inside the warehouse. The reorder sounds trivial, but it encodes...
10Field Notes
Kimball vs Inmon: Two Ways to Build a Data Warehouse
Kimball and Inmon are the two foundational approaches to building a data warehouse. The difference is one decision: build dimensional marts bottom-up, or a normalize...
11Field Notes
One Big Table vs the Star Schema: The Real Trade-off
One Big Table (OBT) denormalizes everything into a single wide table; the star schema keeps facts and dimensions separate. Here's what each actually costs, and why t...
12Field Notes
Batch vs Streaming: How to Actually Decide
Batch vs streaming isn't legacy vs modern. The real question: what latency does the decision consuming the data actually require? Default to batch; promote pipelines...
13Field Notes
What Is Data Lineage, and What Is It Actually For?
Data lineage is the record of where data comes from, how it's transformed, and where it goes. Here's what it's genuinely for — impact analysis, root cause, audit — a...
14Field Notes
What Is Change Data Capture (CDC), and When Do You Need It?
Change data capture identifies inserts, updates, and deletes in a source database and delivers them downstream. Here's how log-based, trigger-based, and query-based ...
15Reconsidered
Is the Modern Data Stack Dead?
The modern data stack isn't dead — but the era of bolting together a dozen best-of-breed SaaS tools as the default is ending. Here's what's actually happening, and w...
16Field Notes
What Is a Vector Database, and Do You Need One?
A vector database stores embeddings and finds items by meaning rather than exact match. Here's what that's actually for, how it relates to your existing stack, and w...
17Field Notes
Data Vault vs Dimensional Modeling: Which Belongs Where
Data Vault and dimensional modeling aren't rivals — they solve different problems at different layers. Here's what Data Vault actually is, what it costs, and when it...
18Essay
What Does a Data Architect Actually Do?
A data architect's job isn't drawing diagrams or picking tools. It's making the deliberate decisions about how data is shaped, defined, owned, and trusted — and defe...
19Field Notes
What Is a Data Catalog, and Do You Need One?
A data catalog is a searchable inventory of an organization's data — what exists, where it lives, what it means, and who owns it. Here's what it's for, what it isn't...
20Essay
Most Data Quality Problems Are Org-Chart Problems
When data is wrong, teams buy a data-quality tool. But the durable causes are organizational — unclear ownership, misaligned incentives, no accountability — and no t...
21Field Notes
Fact Table Types: Transaction, Periodic Snapshot, and Accumulating
There are three kinds of fact table, distinguished by what one row represents over time: transaction, periodic snapshot, and accumulating snapshot. Here's how each w...
22Essay
Your AI Is Only as Good as Your Data Architecture
Retrieval-augmented generation, AI agents, and LLMs querying your warehouse are all only as reliable as the data beneath them. GenAI doesn't replace data architectur...
23Reconsidered
What GenAI Actually Changes About Data Architecture — and What It Doesn't
Cutting through the hype: GenAI adds vector storage and new retrieval patterns to the data stack, but the fundamentals — structure, quality, governance, ownership — ...
24Field Notes
Slowly Changing Dimensions, Explained Without the Jargon
Slowly changing dimensions answer one question: when a dimension attribute changes, do you overwrite history or preserve it? Here are SCD Types 1, 2, and 3, and exac...
25Field Notes
Data Warehouse vs Data Lake vs Lakehouse: A Clear Comparison
A data warehouse stores structured, modeled data. A data lake stores raw data of any shape, cheaply. A lakehouse tries to be both. Here's a side-by-side comparison, ...
26Field Notes
How to Make a Data Pipeline Idempotent
An idempotent data pipeline produces the same result whether it runs once or five times. Here are the concrete patterns — partition overwrite, merge on keys, delete-...
27Essay
What Is a Semantic Layer, and Why Does Your Data Stack Need One?
A semantic layer is the single, governed place where business metrics are defined once — independent of any dashboard. Here's what it is, what it isn't, and why it f...
28Reconsidered
The Medallion Architecture, Reconsidered
Bronze, silver, gold is a useful default and a dangerous dogma. A second look at what the layers get right, and where they quietly fall apart.
29Field Notes
Factless Fact Tables, Explained
A factless fact table records that an event happened — or could have — without any numeric measure. Here's why a fact table with no facts is useful, the two types, a...
30Field Notes
OLTP vs OLAP: Why You Shouldn't Run Analytics on Your App Database
OLTP systems handle many small transactions fast. OLAP systems scan huge volumes for analysis. They're optimized for opposite things — which is why querying your pro...
31Field Notes
What Are Conformed Dimensions, and Why Do They Matter?
A conformed dimension is a single dimension shared identically across multiple fact tables, so different business processes can be compared on the same terms. Here's...
32Essay
Data Contracts Are a Cultural Problem
A schema check is the easy 10% of a data contract. The other 90% is an organizational agreement that no YAML file can enforce for you.
33Field Notes
The Date Dimension: How to Build One and Why You Need It
A date dimension is a table with one row per calendar day, pre-loaded with every attribute of that day. Here's why every warehouse needs one, what columns to include...
34Field Notes
Star Schema vs Snowflake Schema: Which to Use and When
Star schema vs snowflake schema comes down to one decision — whether you normalize your dimensions. Here's the difference, a worked example, a diagram, and why the s...
35Field Notes
Fact Table vs Dimension Table: The Core Distinction
A fact table stores the measurements — the numbers you analyze. A dimension table stores the context you analyze them by. Here's the distinction every dimensional mo...
36Field Notes
A Field Guide to Dimensional Modeling
Facts, dimensions, and grain — the three ideas that quietly run most analytics, explained without the dogma.
37Field Notes
Surrogate Keys vs Natural Keys: A Practical Rule
Surrogate key vs natural key is a decision every data model faces. The practical rule: use surrogate keys for dimensions, keep the natural key as an attribute, and h...
38Manifesto
The Shape of Data
Every dataset has a shape. The only question is whether you chose it, or whether it happened to you.
Essays by email
One new essay on data architecture, straight to your inbox. No noise, unsubscribe anytime.