<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://dataarchitect.studio/feed.xml" rel="self" type="application/atom+xml" /><link href="https://dataarchitect.studio/" rel="alternate" type="text/html" hreflang="en" /><updated>2026-11-09T13:04:53+05:30</updated><id>https://dataarchitect.studio/feed.xml</id><title type="html">dataarchitect.studio</title><subtitle>Essays and field notes on data architecture, data modeling, dimensional modeling, data contracts, and the lakehouse — practical writing for data engineers and architects who design systems that make information trustworthy.</subtitle><author><name>dataarchitect.studio</name></author><entry><title type="html">The Small File Problem, Measured: When Compaction Pays</title><link href="https://dataarchitect.studio/essays/small-file-problem-measured/" rel="alternate" type="text/html" title="The Small File Problem, Measured: When Compaction Pays" /><published>2026-11-08T07:00:00+05:30</published><updated>2026-11-08T00:00:00+05:30</updated><id>https://dataarchitect.studio/essays/small-file-problem-measured</id><content type="html" xml:base="https://dataarchitect.studio/essays/small-file-problem-measured/"><![CDATA[<p>“Small files kill query performance, so compact your tables” is one of the most
repeated claims in lakehouse writing. It is repeated on this site too — the word
<em>compaction</em> appears in about twenty places here, and until today not one of them
was backed by a measurement.</p>

<p>So here is the measurement. Same 2,000,000 rows, same eleven columns, same zstd
codec, written once as a single file and then split across 10, 100, 1,000 and
5,000 files. <strong>Nothing varies except the file count.</strong> The
<a href="/benchmarks/">script is published</a> and runs in about a minute.</p>

<h2 id="the-results">The results</h2>

<table>
  <thead>
    <tr>
      <th>Files</th>
      <th>Avg file size</th>
      <th>Total on disk</th>
      <th>Full scan</th>
      <th>2 of 11 columns</th>
      <th>DuckDB aggregation</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>1</strong></td>
      <td>24 MB</td>
      <td>24.9 MB</td>
      <td>0.289 s</td>
      <td>0.028 s</td>
      <td>0.026 s</td>
    </tr>
    <tr>
      <td><strong>10</strong></td>
      <td>3.2 MB</td>
      <td>32.4 MB</td>
      <td>0.244 s</td>
      <td>0.028 s</td>
      <td>0.028 s</td>
    </tr>
    <tr>
      <td><strong>100</strong></td>
      <td>428 KB</td>
      <td>43.9 MB</td>
      <td>0.292 s</td>
      <td>0.061 s</td>
      <td>0.039 s</td>
    </tr>
    <tr>
      <td><strong>1,000</strong></td>
      <td>46 KB</td>
      <td>47.2 MB</td>
      <td>0.887 s</td>
      <td>0.354 s</td>
      <td>0.169 s</td>
    </tr>
    <tr>
      <td><strong>5,000</strong></td>
      <td>12.7 KB</td>
      <td><strong>64.9 MB</strong></td>
      <td>2.544 s</td>
      <td>1.247 s</td>
      <td>0.954 s</td>
    </tr>
    <tr>
      <td> </td>
      <td> </td>
      <td><strong>2.6× more</strong></td>
      <td><strong>8.8× slower</strong></td>
      <td><strong>45.3× slower</strong></td>
      <td>36× slower</td>
    </tr>
  </tbody>
</table>

<p>Three things in there are worth separating, because they are usually collapsed
into one vague warning.</p>

<figure style="margin:2rem auto;text-align:center;">
<svg class="dia-mob" viewBox="0 0 400 560" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="sf-mt sf-md">
  <title id="sf-mt">Selective reads degrade far faster than full scans</title>
  <desc id="sf-md">Slowdown relative to storing the same two million rows in a single file, on a logarithmic scale. At 10 files neither full scans nor two-column reads are affected. At 100 files the full scan is unchanged while the two-column read is 2.2 times slower. At 1,000 files the full scan is 3.1 times slower and the two-column read 12.9 times. At 5,000 files the full scan is 8.8 times slower while the two-column read is 45.3 times slower, roughly five times worse.</desc>
  <text x="200" y="20" font-size="12" fill="#1c1a17" text-anchor="middle" font-weight="700">Slowdown vs a single file (log scale)</text>
  <rect x="36" y="34" width="14" height="10" fill="#cabfac" /><text x="56" y="43" font-size="10" fill="#56514a">full scan</text>
  <rect x="166" y="34" width="14" height="10" fill="#c8472b" /><text x="186" y="43" font-size="10" fill="#56514a">2 of 11 columns</text>
  <line x1="60" y1="300" x2="380" y2="300" stroke="#1c1a17" stroke-width="1.5" />
  <text x="92" y="318" font-size="10" fill="#56514a" text-anchor="middle">10</text>
  <rect x="78" y="298" width="12" height="2" fill="#cabfac" /><rect x="92" y="298" width="12" height="2" fill="#c8472b" />
  <text x="92" y="292" font-size="10" fill="#8b857a" text-anchor="middle">no penalty</text>
  <text x="170" y="318" font-size="10" fill="#56514a" text-anchor="middle">100</text>
  <rect x="156" y="298" width="12" height="2" fill="#cabfac" />
  <rect x="170" y="267" width="12" height="33" fill="#c8472b" /><text x="176" y="261" font-size="10" fill="#a4391f" text-anchor="middle">2.2x</text>
  <text x="248" y="318" font-size="10" fill="#56514a" text-anchor="middle">1,000</text>
  <rect x="234" y="253" width="12" height="47" fill="#cabfac" /><text x="240" y="247" font-size="10" fill="#8b857a" text-anchor="middle">3.1x</text>
  <rect x="248" y="193" width="12" height="107" fill="#c8472b" /><text x="254" y="187" font-size="10" fill="#a4391f" text-anchor="middle">12.9x</text>
  <text x="326" y="318" font-size="10" fill="#56514a" text-anchor="middle">5,000</text>
  <rect x="312" y="209" width="12" height="91" fill="#cabfac" /><text x="318" y="203" font-size="10" fill="#8b857a" text-anchor="middle">8.8x</text>
  <rect x="326" y="140" width="12" height="160" fill="#c8472b" /><text x="332" y="134" font-size="10" fill="#a4391f" text-anchor="middle">45.3x</text>
  <text x="200" y="340" font-size="10" fill="#56514a" text-anchor="middle">number of files holding the same 2,000,000 rows</text>
  <line x1="20" y1="368" x2="380" y2="368" stroke="#ddd6c8" stroke-width="1" />
  <text x="200" y="396" font-size="11" fill="#1c1a17" text-anchor="middle" font-weight="700">Selective reads degrade about</text>
  <text x="200" y="414" font-size="11" fill="#1c1a17" text-anchor="middle" font-weight="700">five times faster than full scans.</text>
  <text x="200" y="444" font-size="10" fill="#8b857a" text-anchor="middle">Column pruning is the reason to use a</text>
  <text x="200" y="459" font-size="10" fill="#8b857a" text-anchor="middle">columnar format. Fragmentation attacks</text>
  <text x="200" y="474" font-size="10" fill="#8b857a" text-anchor="middle">that specifically.</text>
  <text x="200" y="504" font-size="10" fill="#a4391f" text-anchor="middle">Storage also rises 2.6x for identical rows.</text>
  <text x="200" y="532" font-size="10" fill="#8b857a" text-anchor="middle">Local disk. On object storage every file is</text>
  <text x="200" y="547" font-size="10" fill="#8b857a" text-anchor="middle">an HTTP request — treat this as a floor.</text>
</svg>
<svg class="dia-desk" viewBox="0 0 800 330" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="sf-t sf-d">
  <title id="sf-t">Selective reads degrade far faster than full scans</title>
  <desc id="sf-d">Slowdown relative to storing the same two million rows in a single file, on a logarithmic scale. At 10 files neither full scans nor two-column reads are affected. At 100 files the full scan is unchanged while the two-column read is 2.2 times slower. At 1,000 files the full scan is 3.1 times slower and the two-column read 12.9 times. At 5,000 files the full scan is 8.8 times slower while the two-column read is 45.3 times slower, roughly five times worse. Storage also rises 2.6 times for identical rows.</desc>
  <text x="400" y="22" font-size="13" fill="#1c1a17" text-anchor="middle" font-weight="700">Slowdown vs a single file, same 2,000,000 rows (log scale)</text>
  <rect x="300" y="36" width="14" height="10" fill="#cabfac" /><text x="320" y="45" font-size="11" fill="#56514a">full scan</text>
  <rect x="420" y="36" width="14" height="10" fill="#c8472b" /><text x="440" y="45" font-size="11" fill="#56514a">2 of 11 columns</text>
  <line x1="90" y1="250" x2="730" y2="250" stroke="#1c1a17" stroke-width="1.5" />
  <text x="160" y="268" font-size="11" fill="#56514a" text-anchor="middle">10 files</text>
  <rect x="140" y="248" width="18" height="2" fill="#cabfac" /><rect x="162" y="248" width="18" height="2" fill="#c8472b" />
  <text x="160" y="240" font-size="11" fill="#8b857a" text-anchor="middle">no penalty</text>
  <text x="320" y="268" font-size="11" fill="#56514a" text-anchor="middle">100 files</text>
  <rect x="300" y="248" width="18" height="2" fill="#cabfac" />
  <rect x="322" y="217" width="18" height="33" fill="#c8472b" /><text x="331" y="210" font-size="11" fill="#a4391f" text-anchor="middle">2.2x</text>
  <text x="480" y="268" font-size="11" fill="#56514a" text-anchor="middle">1,000 files</text>
  <rect x="460" y="203" width="18" height="47" fill="#cabfac" /><text x="469" y="196" font-size="11" fill="#8b857a" text-anchor="middle">3.1x</text>
  <rect x="482" y="143" width="18" height="107" fill="#c8472b" /><text x="491" y="136" font-size="11" fill="#a4391f" text-anchor="middle">12.9x</text>
  <text x="640" y="268" font-size="11" fill="#56514a" text-anchor="middle">5,000 files</text>
  <rect x="620" y="159" width="18" height="91" fill="#cabfac" /><text x="629" y="152" font-size="11" fill="#8b857a" text-anchor="middle">8.8x</text>
  <rect x="642" y="90" width="18" height="160" fill="#c8472b" /><text x="651" y="83" font-size="11" fill="#a4391f" text-anchor="middle">45.3x</text>
  <text x="400" y="294" font-size="12" fill="#1c1a17" text-anchor="middle" font-weight="700">Selective reads degrade about five times faster than full scans.</text>
  <text x="400" y="316" font-size="11" fill="#8b857a" text-anchor="middle">Storage also rises 2.6x for identical rows. Local disk, so treat this as a floor.</text>
</svg>
<figcaption style="font-family:'IBM Plex Mono',monospace;font-size:0.75rem;color:#8b857a;margin-top:0.6rem;">Log scale, because a linear one would hide everything below a thousand files. Fragmentation attacks column pruning specifically.</figcaption>
</figure>

<h2 id="1-selective-reads-suffer-about-five-times-more-than-full-scans">1. Selective reads suffer about five times more than full scans</h2>

<p>Full scans got <strong>8.8× slower</strong>. Reading two columns of eleven got <strong>45.3×
slower.</strong></p>

<p>That gap is the real finding, and it is the opposite of reassuring. <strong>Column
pruning is the main reason to use a columnar format at all.</strong> A full scan
amortises the per-file cost across a lot of useful work — you were going to read
every byte anyway. A selective read pays the same per-file overhead for a
fraction of the payload, so the overhead dominates sooner.</p>

<p>Put differently: fragmentation attacks the specific thing you chose
<a href="/essays/apache-iceberg-vs-parquet/">Parquet</a> for. Measured as the benefit that
column pruning still delivers:</p>

<table>
  <thead>
    <tr>
      <th>Files</th>
      <th>Pruning still pays</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>1</td>
      <td><strong>10.5× faster</strong> than a full scan</td>
    </tr>
    <tr>
      <td>10</td>
      <td>8.6× faster</td>
    </tr>
    <tr>
      <td>100</td>
      <td>4.8× faster</td>
    </tr>
    <tr>
      <td>1,000</td>
      <td>2.5× faster</td>
    </tr>
    <tr>
      <td>5,000</td>
      <td><strong>2.0× faster</strong></td>
    </tr>
  </tbody>
</table>

<p>Pruning never stops helping — it just stops helping <em>much</em>. You keep the format
and lose most of the advantage.</p>

<p><strong>A correction, since this site’s rule is that published numbers are measured
numbers.</strong> The first run of this benchmark showed the 5,000-file pruned read at
4.51 s, <em>slower</em> than its own full scan — pruning going negative. It was a
striking result and it was about to be the headline here. It does not replicate:
three further runs at five trials each gave 1.257, 1.242 and 1.252 s, within one
percent of each other. The original was an outlier, almost certainly memory
pressure during the first large write. The honest finding is the duller one in
the table above.</p>

<h2 id="2-the-same-data-costs-26-the-storage">2. The same data costs 2.6× the storage</h2>

<p>This one I nearly missed, because I had assumed total bytes would be constant and
wrote that assumption into the script’s own caveats before checking it.</p>

<p>They are not constant. <strong>24.9 MB as one file, 64.9 MB as 5,000</strong> — identical
rows, 2.6 times the disk.</p>

<p>The cause is not mysterious. Compression and dictionary encoding work on blocks,
so a smaller block has less context to compress against: a dictionary of country
codes that pays for itself across a million rows is dead weight across four
hundred. On top of that every file carries its own footer, schema and statistics.</p>

<p>You therefore pay twice. Once in storage, and again in the read path, because
part of that 8.8× slowdown is simply <strong>more bytes to move.</strong> This benchmark does
not separate “more bytes” from “per-file overhead”, and I would not trust anyone
who claims a clean split without isolating it deliberately.</p>

<h2 id="3-under-a-hundred-files-there-is-nothing-to-fix">3. Under a hundred files, there is nothing to fix</h2>

<p>Look at the 10-file row again. The full scan was <strong>0.244 s against 0.289 s for a
single file</strong> — marginally <em>faster</em>, because several files parallelise where one
does not.</p>

<p>At 100 files averaging 428 KB, the pruned read is 2.2× slower: measurable, rarely
urgent. The damage becomes serious between there and 1,000 files, as average size
falls through the tens of kilobytes.</p>

<p><strong>So “compact everything” is as wrong as ignoring the problem.</strong> Compaction
rewrites data. It costs compute, churns storage, invalidates caches, and on a
table format it generates new snapshots that themselves need expiring. Running it
against a table with forty healthy files is pure cost for no benefit.</p>

<p>The useful question is not <em>how many files does this table have</em> but <strong>what is
the average file size, and has it fallen into the range where the penalty
starts.</strong> On this data that range began in the hundreds of kilobytes.</p>

<h2 id="why-tables-end-up-here">Why tables end up here</h2>

<p>Almost always because <strong>write frequency is decoupled from file size, and nobody
reconnected them.</strong></p>

<p>A streaming or <a href="/essays/what-is-change-data-capture/">CDC</a> job that commits every
minute writes 1,440 files a day whether a minute carries a megabyte or a
kilobyte, and the quietest hours produce the worst files. Over-partitioning does
the same thing structurally: partition by day <em>and</em> region <em>and</em> product and most
partitions hold almost nothing. And frequent small commits are exactly how
<a href="/essays/merge-on-read-vs-copy-on-write/">merge-on-read</a> tables accumulate delete
files alongside the data.</p>

<p>None of these is a mistake at the moment it is made. Each is a reasonable
freshness decision whose storage consequence shows up months later.</p>

<h2 id="what-to-actually-do">What to actually do</h2>

<ul>
  <li><strong>Measure average file size, not file count.</strong> It is the number that predicts
the penalty. File count alone tells you nothing without total volume.</li>
  <li><strong>Set a target size and compact toward it</strong>, rather than compacting on a fixed
schedule. Hundreds of megabytes is a common target and a reasonable default.</li>
  <li><strong>Commit for freshness, compact for size.</strong> They are different jobs with
different triggers and should not be the same job.</li>
  <li><strong>Expect it to be worse than this on object storage.</strong> These numbers are local
disk, where opening a file is nearly free. On S3 every file is an HTTP request
with latency in the tens of milliseconds, paid per file regardless of size.
<strong>Treat this curve as a floor.</strong></li>
  <li><strong>If selective queries matter to you, care sooner.</strong> They degrade five times
faster than the full scans you might be benchmarking with.</li>
</ul>

<h2 id="the-honest-limits-of-this">The honest limits of this</h2>

<p>One dataset shape, eleven columns, zstd, default row-group sizes, local disk,
warm page cache, single writer, no concurrency. A single 2-million-row file still
contains many row groups internally, so the one-file case is not an artificially
perfect baseline.</p>

<p>What transfers is the <strong>shape</strong>: selective reads degrade far faster than full
scans, storage inflates meaningfully, and the knee sits in the hundreds of files
rather than the tens. The absolute seconds are a property of a two-core machine
and are worth nothing to you. <a href="/benchmarks/">Rerun it</a> on your own data, which
takes about a minute and will tell you something the generic advice cannot.</p>]]></content><author><name>dataarchitect.studio</name></author><summary type="html"><![CDATA[Splitting the same rows across more files costs 2.6x the storage and 45x the query time. But under a hundred files it is free. Where the knee actually is.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://dataarchitect.studio/assets/og-default.png" /><media:content medium="image" url="https://dataarchitect.studio/assets/og-default.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">What Is Master Data Management, and Why Does MDM Fail?</title><link href="https://dataarchitect.studio/essays/what-is-master-data-management/" rel="alternate" type="text/html" title="What Is Master Data Management, and Why Does MDM Fail?" /><published>2026-10-17T07:00:00+05:30</published><updated>2026-10-17T00:00:00+05:30</updated><id>https://dataarchitect.studio/essays/what-is-master-data-management</id><content type="html" xml:base="https://dataarchitect.studio/essays/what-is-master-data-management/"><![CDATA[<p>Every large organisation eventually discovers it does not know how many customers
it has. Sales counts accounts, finance counts billable entities, support counts
contacts, and the three numbers disagree by a margin nobody can explain. The
standard response is to buy master data management software.</p>

<p>That response is usually premature, and understanding why is most of what is
worth knowing about MDM.</p>

<h2 id="what-master-data-actually-is">What master data actually is</h2>

<p><strong>Master data is the set of entities a business shares across systems</strong> —
customer, product, supplier, location, employee. It is the nouns, as distinct
from transactions, which are the verbs. An order is a transaction; the customer
who placed it and the product they bought are master data.</p>

<p>The defining property is that master data is <em>referenced by many systems and
owned by none of them in practice</em>. Every system that touches a customer has
opinions about what a customer is, and each of those opinions is encoded in a
schema somebody shipped years ago.</p>

<p><strong>Master data management</strong> is the practice of maintaining one agreed version of
those entities. It has three parts, and they are not equally hard.</p>

<h2 id="the-mechanical-part-matching-merging-survivorship">The mechanical part: matching, merging, survivorship</h2>

<figure style="margin:2rem auto;text-align:center;">
<svg class="dia-mob" viewBox="0 0 400 620" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="mdm-mt mdm-md">
  <title id="mdm-mt">Every mechanical step in MDM rests on a human decision</title>
  <desc id="mdm-md">A pipeline runs from source systems through matching, merging and survivorship to a golden record. Beside each mechanical step sits the decision it depends on: matching needs agreement on what counts as the same customer and what match confidence is good enough; merging needs someone empowered to approve or reverse a merge; survivorship needs a per-attribute ranking of which source is more trustworthy. The software performs the steps; it cannot make the decisions.</desc>
  <text x="200" y="22" font-size="12" fill="#1c1a17" text-anchor="middle" font-weight="700">What the software does</text>
  <rect x="30" y="36" width="340" height="40" rx="4" fill="#1c1a17" />
  <text x="200" y="53" font-size="11" fill="#f6f3ec" text-anchor="middle">CRM · ERP · billing · support</text>
  <text x="200" y="69" font-size="10" fill="#cabfac" text-anchor="middle">each with its own idea of "customer"</text>
  <line x1="200" y1="76" x2="200" y2="98" stroke="#cabfac" stroke-width="2" />
  <rect x="30" y="98" width="340" height="34" rx="4" fill="#c8472b" />
  <text x="200" y="120" font-size="11" fill="#f6f3ec" text-anchor="middle">MATCH — are these the same thing?</text>
  <text x="200" y="150" font-size="10" fill="#a4391f" text-anchor="middle">needs: what counts as the same customer,</text>
  <text x="200" y="165" font-size="10" fill="#a4391f" text-anchor="middle">and what confidence is good enough</text>
  <line x1="200" y1="176" x2="200" y2="196" stroke="#cabfac" stroke-width="2" />
  <rect x="30" y="196" width="340" height="34" rx="4" fill="#c8472b" />
  <text x="200" y="218" font-size="11" fill="#f6f3ec" text-anchor="middle">MERGE — combine the matches</text>
  <text x="200" y="248" font-size="10" fill="#a4391f" text-anchor="middle">needs: someone empowered to approve,</text>
  <text x="200" y="263" font-size="10" fill="#a4391f" text-anchor="middle">reject and reverse a merge</text>
  <line x1="200" y1="274" x2="200" y2="294" stroke="#cabfac" stroke-width="2" />
  <rect x="30" y="294" width="340" height="34" rx="4" fill="#c8472b" />
  <text x="200" y="316" font-size="11" fill="#f6f3ec" text-anchor="middle">SURVIVORSHIP — which value wins?</text>
  <text x="200" y="346" font-size="10" fill="#a4391f" text-anchor="middle">needs: a per-attribute ranking of which</text>
  <text x="200" y="361" font-size="10" fill="#a4391f" text-anchor="middle">source is more trustworthy, field by field</text>
  <line x1="200" y1="372" x2="200" y2="392" stroke="#cabfac" stroke-width="2" />
  <rect x="30" y="392" width="340" height="44" rx="6" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <text x="200" y="412" font-size="12" fill="#1c1a17" text-anchor="middle" font-weight="700">golden record</text>
  <text x="200" y="429" font-size="10" fill="#56514a" text-anchor="middle">assembled, not found</text>
  <line x1="20" y1="466" x2="380" y2="466" stroke="#ddd6c8" stroke-width="1" />
  <text x="200" y="494" font-size="11" fill="#1c1a17" text-anchor="middle" font-weight="700">The software performs every step.</text>
  <text x="200" y="514" font-size="11" fill="#1c1a17" text-anchor="middle" font-weight="700">It cannot make a single decision.</text>
  <text x="200" y="546" font-size="10" fill="#8b857a" text-anchor="middle">Each decision needs a person with the</text>
  <text x="200" y="561" font-size="10" fill="#8b857a" text-anchor="middle">authority to overrule a department.</text>
  <text x="200" y="588" font-size="10" fill="#8b857a" text-anchor="middle">If that person does not exist, the project</text>
  <text x="200" y="603" font-size="10" fill="#8b857a" text-anchor="middle">stalls here — after the licence is signed.</text>
</svg>
<svg class="dia-desk" viewBox="0 0 800 330" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="mdm-t mdm-d">
  <title id="mdm-t">Every mechanical step in MDM rests on a human decision</title>
  <desc id="mdm-d">On the left, a pipeline runs from source systems through matching, merging and survivorship to a golden record. On the right, the decision each mechanical step depends on: matching needs agreement on what counts as the same customer and what match confidence is good enough; merging needs someone empowered to approve or reverse a merge; survivorship needs a per-attribute ranking of which source is more trustworthy. The software performs the steps; it cannot make the decisions.</desc>
  <text x="200" y="22" font-size="12" fill="#1c1a17" text-anchor="middle" font-weight="700">What the software does</text>
  <text x="600" y="22" font-size="12" fill="#1c1a17" text-anchor="middle" font-weight="700">What it cannot do</text>
  <rect x="40" y="36" width="320" height="40" rx="4" fill="#1c1a17" />
  <text x="200" y="53" font-size="11" fill="#f6f3ec" text-anchor="middle">CRM · ERP · billing · support</text>
  <text x="200" y="69" font-size="10" fill="#cabfac" text-anchor="middle">each with its own idea of "customer"</text>
  <line x1="200" y1="76" x2="200" y2="96" stroke="#cabfac" stroke-width="2" />
  <rect x="40" y="96" width="320" height="34" rx="4" fill="#c8472b" />
  <text x="200" y="118" font-size="11" fill="#f6f3ec" text-anchor="middle">MATCH — are these the same thing?</text>
  <text x="440" y="112" font-size="11" fill="#a4391f">what counts as the same customer,</text>
  <text x="440" y="128" font-size="11" fill="#a4391f">and what confidence is good enough</text>
  <line x1="200" y1="130" x2="200" y2="150" stroke="#cabfac" stroke-width="2" />
  <rect x="40" y="150" width="320" height="34" rx="4" fill="#c8472b" />
  <text x="200" y="172" font-size="11" fill="#f6f3ec" text-anchor="middle">MERGE — combine the matches</text>
  <text x="440" y="166" font-size="11" fill="#a4391f">someone empowered to approve,</text>
  <text x="440" y="182" font-size="11" fill="#a4391f">reject and reverse a merge</text>
  <line x1="200" y1="184" x2="200" y2="204" stroke="#cabfac" stroke-width="2" />
  <rect x="40" y="204" width="320" height="34" rx="4" fill="#c8472b" />
  <text x="200" y="226" font-size="11" fill="#f6f3ec" text-anchor="middle">SURVIVORSHIP — which value wins?</text>
  <text x="440" y="220" font-size="11" fill="#a4391f">a per-attribute ranking of which source</text>
  <text x="440" y="236" font-size="11" fill="#a4391f">is more trustworthy, field by field</text>
  <line x1="200" y1="238" x2="200" y2="258" stroke="#cabfac" stroke-width="2" />
  <rect x="40" y="258" width="320" height="40" rx="6" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <text x="200" y="276" font-size="12" fill="#1c1a17" text-anchor="middle" font-weight="700">golden record</text>
  <text x="200" y="292" font-size="10" fill="#56514a" text-anchor="middle">assembled, not found</text>
  <line x1="420" y1="30" x2="420" y2="300" stroke="#ddd6c8" stroke-width="1.5" stroke-dasharray="4 4" />
  <text x="600" y="268" font-size="11" fill="#1c1a17" text-anchor="middle" font-weight="700">Each needs authority to</text>
  <text x="600" y="286" font-size="11" fill="#1c1a17" text-anchor="middle" font-weight="700">overrule a department.</text>
  <text x="400" y="322" font-size="12" fill="#8b857a" text-anchor="middle">The software performs every step and cannot make a single decision.</text>
</svg>
<figcaption style="font-family:'IBM Plex Mono',monospace;font-size:0.75rem;color:#8b857a;margin-top:0.6rem;">The pipeline is the easy half. Every box on the left waits on an answer from the right.</figcaption>
</figure>

<p><strong>Matching</strong> asks whether two records refer to the same real-world thing.
<em>Deterministic</em> matching compares exact keys — a tax identifier, a registration
number, an email. <em>Fuzzy</em> or probabilistic matching scores similarity when no
shared key exists, which is the usual case for customers and suppliers whose
names and addresses are typed by humans in a hurry.</p>

<p><strong>Merging</strong> combines matched records. Most implementations auto-merge above a
confidence threshold, queue the middle band for human review, and leave the rest
alone.</p>

<p><strong>Survivorship</strong> decides, attribute by attribute, which value wins. The standard
approach ranks sources per field: ERP wins on legal name and tax data, CRM wins
on contact preferences, the billing system wins on the address that actually
receives invoices. A single blanket rule across every field is the classic
mistake, because it lets a source that is authoritative for one attribute
overwrite a better source for another.</p>

<p>The output is the <strong>golden record</strong> — and the word “record” oversells it. It is
not a row that existed somewhere waiting to be found. <strong>It is assembled</strong>, and it
reflects a set of decisions about trust. Change the survivorship ranking and you
get a different golden record from identical inputs.</p>

<h2 id="the-four-implementation-styles">The four implementation styles</h2>

<p>Most MDM writing organises implementations into four styles. They are a genuinely
useful ladder, and they are usually presented as a vendor-neutral industry
standard. A caveat on provenance: this taxonomy is widely attributed, and I could
not trace it to a single authoritative primary source — almost every citation
leads to a vendor glossary. Treat the names as common usage rather than a
specification.</p>

<table>
  <thead>
    <tr>
      <th>Style</th>
      <th>Where master data lives</th>
      <th>Writes back?</th>
      <th>What it costs you</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Registry</strong></td>
      <td>Sources keep it; hub holds IDs and match keys only</td>
      <td>No</td>
      <td>Least disruption, least benefit</td>
    </tr>
    <tr>
      <td><strong>Consolidation</strong></td>
      <td>Copied into a hub, cleansed, published</td>
      <td>No</td>
      <td>Good for analytics, fixes nothing operationally</td>
    </tr>
    <tr>
      <td><strong>Coexistence</strong></td>
      <td>Both hub and sources, synchronised</td>
      <td>Yes</td>
      <td>Sync complexity, and the hub is not the system of record</td>
    </tr>
    <tr>
      <td><strong>Centralized / transaction hub</strong></td>
      <td>The hub, authored directly</td>
      <td>N/A — it <em>is</em> the source</td>
      <td>Maximum benefit, maximum organisational commitment</td>
    </tr>
  </tbody>
</table>

<p>Read down that last column. <strong>The ladder is not primarily technical.</strong> Each rung
increases how much authority the hub has over systems other teams own, and that
is a political quantity, not an architectural one. Programmes that start at
registry or consolidation and earn their way upward tend to survive. Programmes
that begin at transaction hub because that is where the value is have skipped the
part where everyone agrees to be overruled.</p>

<h2 id="why-mdm-projects-fail">Why MDM projects fail</h2>

<p>Not because the software is bad. Modern matching engines are genuinely good, and
the fuzzy-matching problem is largely solved for ordinary cases.</p>

<p>They fail because <strong>every mechanical step above has a prerequisite that is a
decision, and the decision requires authority somebody has to actually hold.</strong></p>

<ul>
  <li>Matching needs agreement on what counts as <em>the same customer</em>. Is a
subsidiary the same customer as its parent? Is a person who moved house the
same person? Finance and sales will answer differently, and both answers are
defensible within their own department’s purpose.</li>
  <li>Merging needs someone who can approve a merge over the objection of the team
whose record loses.</li>
  <li>Survivorship needs a per-field ranking that says, in writing, that one
department’s data is less trustworthy than another’s.</li>
</ul>

<p>None of these is a configuration question. All of them are <em>whose definition
wins</em>, which is a question about org charts. A tool makes them unavoidable and
explicit, which is useful — but it surfaces them at the most expensive possible
moment, after a licence is signed and a delivery date is public.</p>

<p>This is the same shape as
<a href="/essays/data-quality-problems-are-org-chart-problems/">data quality problems being org-chart problems</a>,
and the same reason a <a href="/glossary/data-contract/">data contract</a> is
<a href="/essays/data-contracts-are-a-cultural-problem/">mostly a cultural artefact</a>. The
pattern repeats because the underlying constraint repeats: software can enforce
an agreement and cannot manufacture one.</p>

<h2 id="what-to-do-before-buying-anything">What to do before buying anything</h2>

<p>Two diagnostics, both cheap, both revealing.</p>

<p><strong>Count the claimants.</strong> For your most contested entity, list every system that
believes it defines that entity. Not systems that <em>store</em> it — systems whose
owners would argue if you told them their version was wrong. The number is
usually higher than expected and is the real scope of the problem.</p>

<p><strong>Ask who can overrule.</strong> Name the person who can tell finance that sales owns
the customer definition, and make them say it out loud. If no such person exists,
you do not have an MDM project; you have an escalation waiting to happen.</p>

<p>If the second question has a clear answer, a tool will accelerate you
considerably. If it does not, the useful first step is cheaper and less
political: a <a href="/essays/what-is-a-data-catalog/">data catalog</a> to document how many
conflicting definitions exist, or a
<a href="/essays/what-is-a-semantic-layer/">semantic layer</a> to make the disagreement
visible in reporting before you try to resolve it at source.</p>

<h2 id="where-it-meets-the-warehouse">Where it meets the warehouse</h2>

<p>For analytics specifically, there is a narrower and much cheaper version of this
problem, and it is worth knowing you may only need that one.</p>

<p>If your goal is consistent reporting rather than consistent operations, you do
not need to fix the source systems. You need
<a href="/essays/conformed-dimensions/">conformed dimensions</a>: one shared customer
dimension that every fact table references, with the reconciliation done once on
the way in. That is MDM’s analytical benefit without MDM’s political cost, and it
is a well-trodden path.</p>

<p>MDM proper earns its keep when the <em>operational</em> systems need to agree — when a
shipment goes to the wrong address because billing and logistics disagree about
which one is current. That is a real problem and worth real money. It is just a
different problem from the one most MDM purchases are trying to solve.</p>

<p><strong>The summary worth keeping:</strong> master data management is an organisational
agreement with software attached. Buy the software when you have the agreement,
or at minimum when you know who is empowered to make it. Buying it to <em>produce</em>
the agreement is the expensive way to discover you needed it first.</p>]]></content><author><name>dataarchitect.studio</name></author><summary type="html"><![CDATA[Master data management is sold as a product category. It is an organisational agreement with software attached, which is why buying the tool first fails.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://dataarchitect.studio/assets/og-default.png" /><media:content medium="image" url="https://dataarchitect.studio/assets/og-default.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">HOOK vs Data Vault: What the Hook Pattern Actually Is</title><link href="https://dataarchitect.studio/essays/hook-vs-data-vault/" rel="alternate" type="text/html" title="HOOK vs Data Vault: What the Hook Pattern Actually Is" /><published>2026-09-29T07:00:00+05:30</published><updated>2026-09-29T00:00:00+05:30</updated><id>https://dataarchitect.studio/essays/hook-vs-data-vault</id><content type="html" xml:base="https://dataarchitect.studio/essays/hook-vs-data-vault/"><![CDATA[<p>If you search for the “hook pattern in Data Vault”, you will find very little, and
the reason is that the question contains a category error. <strong>Data Vault has no
construct called a hook.</strong> It has hubs, links and satellites, and that is the
whole vocabulary.</p>

<p>HOOK is a <em>different</em> data warehousing approach, created by
<a href="https://hookcookbook.substack.com/">Andrew Foad</a>, explicitly as an alternative to
Data Vault rather than an extension of it. In his own words, “HOOK is my attempt
to solve problems that I encountered with data vault.” The two are competitors.</p>

<p>That confusion is worth clearing up properly, because the underlying question
people are actually asking — <em>is there a lighter way to do what Data Vault does?</em>
— is a good one.</p>

<h2 id="why-hook-exists">Why HOOK exists</h2>

<p>Foad is unusually candid about the motivation, and about the originality of the
idea. From the cookbook:</p>

<blockquote>
  <p>There is nothing inherently wrong with the current data warehousing approaches;
Data Vault and Dimensional modelling work well if done right. Unfortunately,
they are rarely done right.</p>
</blockquote>

<p>His argument is not that Data Vault is wrong. It is that Data Vault demands a
level of rigour that real projects abandon the moment delivery pressure arrives,
and a method that only works when followed perfectly is a method that mostly
fails. He also says plainly that HOOK “is nothing new, not really… a
re-imagining of ideas put forward by” Inmon, Linstedt and Kimball.</p>

<p>That is a more honest framing than most methodology pitches manage, and it sets
up the actual design goal: keep the integration benefits, drop the up-front
restructuring.</p>

<h2 id="the-four-components">The four components</h2>

<p>HOOK has four parts, and genuinely only four.</p>

<p><strong>The business glossary</strong> is the foundation, and it is a hard gate: you may only
align data to a business concept once that concept has been formally defined. If
the organisation cannot say what it means by <em>Customer</em>, HOOK’s position is that
you have no business integrating customer data yet. This is the same instinct as a
<a href="/essays/what-is-a-semantic-layer/">semantic layer</a>, applied earlier.</p>

<p><strong>A hook</strong> is a structure recording the business keys that identify instances of a
concept. A Customer hook records all the business keys for customers. Hooks can
also express relationships — a Sales hook associating Customer and Product. The
detail that surprises people: <strong>hooks can be physical tables but do not have to
be.</strong> Their existence may simply be implied by the keys projected onto the data.</p>

<p><strong>A bag</strong> is a virtual construct — normally a view — that aligns raw,
source-shaped data to glossary concepts without changing it. The source table
stays exactly as it landed; the bag wraps it and exposes qualified business keys
alongside the original columns.</p>

<p>A note on vocabulary, because the sources disagree: the cookbook (2022) calls this
construct a <strong>bag</strong>. Foad’s more recent talks call it a <strong>frame</strong>. Same thing,
newer name. If you read both and think you have missed a fifth component, you
haven’t.</p>

<p><strong>A keyset</strong> qualifies business keys across sources. Customer #123 in System X is
very probably not Customer #123 in System Y, so each keyset has an identifier that
prefixes the keys originating from that system. This is the same problem that
<a href="/essays/surrogate-keys-vs-natural-keys/">surrogate keys</a> solve in a dimensional
model, handled by qualification rather than by substitution.</p>

<h2 id="the-actual-difference-from-data-vault">The actual difference from Data Vault</h2>

<figure style="margin:2rem auto;text-align:center;">
<svg class="dia-mob" viewBox="0 0 400 600" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="hvd-mt hvd-md">
  <title id="hvd-mt">Data Vault restructures; HOOK annotates</title>
  <desc id="hvd-md">Two approaches shown one above the other from the same source tables. Data Vault, first, transforms source data into hubs, links and satellites, so the physical model is rebuilt before loading and modelling must happen up front. HOOK, below, leaves the source tables unchanged and defines a bag view over each one that adds qualified business keys, so the structure is untouched and modelling is deferred to the consumption layer.</desc>
  <text x="200" y="22" font-size="13" fill="#1c1a17" text-anchor="middle" font-weight="700">Data Vault — restructure</text>
  <rect x="40" y="36" width="320" height="26" rx="4" fill="#1c1a17" />
  <text x="200" y="54" font-size="11" fill="#f6f3ec" text-anchor="middle">source tables</text>
  <line x1="200" y1="62" x2="200" y2="88" stroke="#cabfac" stroke-width="2" />
  <text x="248" y="80" font-size="10" fill="#a4391f">transform</text>
  <rect x="40" y="88" width="96" height="40" rx="4" fill="#c8472b" />
  <text x="88" y="113" font-size="11" fill="#f6f3ec" text-anchor="middle">hubs</text>
  <rect x="152" y="88" width="96" height="40" rx="4" fill="#c8472b" />
  <text x="200" y="113" font-size="11" fill="#f6f3ec" text-anchor="middle">links</text>
  <rect x="264" y="88" width="96" height="40" rx="4" fill="#c8472b" />
  <text x="312" y="113" font-size="11" fill="#f6f3ec" text-anchor="middle">satellites</text>
  <text x="200" y="152" font-size="10" fill="#56514a" text-anchor="middle">the physical model IS the deliverable</text>
  <text x="200" y="170" font-size="10" fill="#a4391f" text-anchor="middle">modelling happens before loading</text>
  <line x1="20" y1="196" x2="380" y2="196" stroke="#ddd6c8" stroke-width="1.5" stroke-dasharray="4 4" />
  <text x="200" y="232" font-size="13" fill="#1c1a17" text-anchor="middle" font-weight="700">HOOK — annotate</text>
  <rect x="40" y="246" width="320" height="26" rx="4" fill="#1c1a17" />
  <text x="200" y="264" font-size="11" fill="#f6f3ec" text-anchor="middle">source tables</text>
  <line x1="200" y1="272" x2="200" y2="298" stroke="#cabfac" stroke-width="2" />
  <text x="252" y="290" font-size="10" fill="#8b857a">wrap in a view</text>
  <rect x="40" y="298" width="320" height="56" rx="6" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <text x="200" y="320" font-size="11" fill="#1c1a17" text-anchor="middle" font-weight="700">bag (a view over the source)</text>
  <text x="200" y="340" font-size="10" fill="#56514a" text-anchor="middle">same columns + qualified business keys</text>
  <rect x="40" y="372" width="150" height="34" rx="4" fill="#c8472b" />
  <text x="115" y="393" font-size="10" fill="#f6f3ec" text-anchor="middle">hook: customer</text>
  <rect x="210" y="372" width="150" height="34" rx="4" fill="#c8472b" />
  <text x="285" y="393" font-size="10" fill="#f6f3ec" text-anchor="middle">hook: product</text>
  <text x="200" y="430" font-size="10" fill="#56514a" text-anchor="middle">source structure untouched</text>
  <text x="200" y="448" font-size="10" fill="#a4391f" text-anchor="middle">modelling deferred to consumption</text>
  <line x1="20" y1="476" x2="380" y2="476" stroke="#ddd6c8" stroke-width="1" />
  <text x="200" y="504" font-size="11" fill="#1c1a17" text-anchor="middle" font-weight="700">The trade</text>
  <text x="200" y="526" font-size="10" fill="#8b857a" text-anchor="middle">Data Vault pays up front and delivers</text>
  <text x="200" y="542" font-size="10" fill="#8b857a" text-anchor="middle">a queryable integrated model.</text>
  <text x="200" y="566" font-size="10" fill="#8b857a" text-anchor="middle">HOOK pays later and delivers</text>
  <text x="200" y="582" font-size="10" fill="#8b857a" text-anchor="middle">annotated source data.</text>
</svg>
<svg class="dia-desk" viewBox="0 0 800 330" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="hvd-t hvd-d">
  <title id="hvd-t">Data Vault restructures; HOOK annotates</title>
  <desc id="hvd-d">Two approaches from the same source tables, side by side. Data Vault, on the left, transforms source data into hubs, links and satellites, so the physical model is rebuilt before loading and modelling must happen up front. HOOK, on the right, leaves the source tables unchanged and defines a bag view over each one that adds qualified business keys, so the structure is untouched and modelling is deferred to the consumption layer.</desc>
  <text x="200" y="24" font-size="13" fill="#1c1a17" text-anchor="middle" font-weight="700">Data Vault — restructure</text>
  <rect x="50" y="40" width="300" height="28" rx="4" fill="#1c1a17" />
  <text x="200" y="59" font-size="11" fill="#f6f3ec" text-anchor="middle">source tables</text>
  <line x1="200" y1="68" x2="200" y2="98" stroke="#cabfac" stroke-width="2" />
  <text x="250" y="88" font-size="11" fill="#a4391f">transform</text>
  <rect x="50" y="98" width="90" height="44" rx="4" fill="#c8472b" />
  <text x="95" y="125" font-size="11" fill="#f6f3ec" text-anchor="middle">hubs</text>
  <rect x="155" y="98" width="90" height="44" rx="4" fill="#c8472b" />
  <text x="200" y="125" font-size="11" fill="#f6f3ec" text-anchor="middle">links</text>
  <rect x="260" y="98" width="90" height="44" rx="4" fill="#c8472b" />
  <text x="305" y="125" font-size="11" fill="#f6f3ec" text-anchor="middle">satellites</text>
  <text x="200" y="172" font-size="11" fill="#56514a" text-anchor="middle">the physical model IS the deliverable</text>
  <text x="200" y="194" font-size="11" fill="#a4391f" text-anchor="middle">modelling happens before loading</text>
  <line x1="400" y1="34" x2="400" y2="250" stroke="#ddd6c8" stroke-width="1.5" stroke-dasharray="4 4" />
  <text x="600" y="24" font-size="13" fill="#1c1a17" text-anchor="middle" font-weight="700">HOOK — annotate</text>
  <rect x="450" y="40" width="300" height="28" rx="4" fill="#1c1a17" />
  <text x="600" y="59" font-size="11" fill="#f6f3ec" text-anchor="middle">source tables</text>
  <line x1="600" y1="68" x2="600" y2="98" stroke="#cabfac" stroke-width="2" />
  <text x="652" y="88" font-size="11" fill="#8b857a">wrap in a view</text>
  <rect x="450" y="98" width="300" height="52" rx="6" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <text x="600" y="120" font-size="11" fill="#1c1a17" text-anchor="middle" font-weight="700">bag (a view over the source)</text>
  <text x="600" y="140" font-size="10" fill="#56514a" text-anchor="middle">same columns + qualified business keys</text>
  <rect x="450" y="166" width="142" height="32" rx="4" fill="#c8472b" />
  <text x="521" y="187" font-size="10" fill="#f6f3ec" text-anchor="middle">hook: customer</text>
  <rect x="608" y="166" width="142" height="32" rx="4" fill="#c8472b" />
  <text x="679" y="187" font-size="10" fill="#f6f3ec" text-anchor="middle">hook: product</text>
  <text x="600" y="222" font-size="11" fill="#56514a" text-anchor="middle">source structure untouched</text>
  <text x="600" y="244" font-size="11" fill="#a4391f" text-anchor="middle">modelling deferred to consumption</text>
  <line x1="40" y1="272" x2="760" y2="272" stroke="#ddd6c8" stroke-width="1" />
  <text x="400" y="300" font-size="12" fill="#8b857a" text-anchor="middle">Data Vault pays up front and delivers a queryable integrated model.</text>
  <text x="400" y="320" font-size="12" fill="#8b857a" text-anchor="middle">HOOK pays later and delivers annotated source data.</text>
</svg>
<figcaption style="font-family:'IBM Plex Mono',monospace;font-size:0.75rem;color:#8b857a;margin-top:0.6rem;">The same integration idea, on opposite sides of the load. One restructures; the other annotates.</figcaption>
</figure>

<p><a href="/patterns/data-vault/">Data Vault</a> restructures. Source data is decomposed into
hubs (business keys), links (relationships) and satellites (descriptive attributes
over time). The physical model is the deliverable, and it has to be designed
before you can load anything into it.</p>

<p>HOOK annotates. The source table lands and stays as it is. A bag view sits on top
and adds qualified business keys as extra columns. Foad’s summary is <strong>ELO rather
than ELT</strong> — extract, load, organise, with no transformation step. Loading and
modelling become independent: the model is a layer of definitions over loaded
data, not a structure maintained alongside it.</p>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>Data Vault</th>
      <th>HOOK</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Core objects</strong></td>
      <td>Hubs, links, satellites</td>
      <td>Glossary, hooks, bags, keysets</td>
    </tr>
    <tr>
      <td><strong>Source structure</strong></td>
      <td>Decomposed</td>
      <td>Preserved</td>
    </tr>
    <tr>
      <td><strong>When you model</strong></td>
      <td>Before loading</td>
      <td>Deferred to consumption</td>
    </tr>
    <tr>
      <td><strong>Physical footprint</strong></td>
      <td>New tables</td>
      <td>Usually just views</td>
    </tr>
    <tr>
      <td><strong>Changing your mind</strong></td>
      <td>Refactor the model</td>
      <td>Add or drop a column</td>
    </tr>
    <tr>
      <td><strong>What you hand a consumer</strong></td>
      <td>An integrated model</td>
      <td>Annotated source data</td>
    </tr>
    <tr>
      <td><strong>Maturity</strong></td>
      <td>20 years, certification, tooling</td>
      <td>One author, small community</td>
    </tr>
  </tbody>
</table>

<p>The last row is the one that decides most real decisions, and I’ll come back to it.</p>

<h2 id="where-the-argument-is-strongest">Where the argument is strongest</h2>

<p>The genuinely good idea here is <strong>decoupling integration from restructuring.</strong></p>

<p>Data Vault’s integration value comes from agreeing business keys and recording
relationships. Almost none of that value strictly requires physically splitting
tables into hubs and satellites — that is an implementation choice which happens
to be expensive, and which has to be right before the first load. HOOK’s claim is
that you can get the subject-orientation and integration by <em>qualifying and
projecting keys</em>, leaving the bytes alone.</p>

<p>That is a real insight, and it maps onto something the industry already learned
elsewhere: <a href="/essays/what-is-an-open-table-format/">open table formats</a> also won by
adding a metadata layer over untouched files rather than by rewriting data into a
proprietary shape.</p>

<p>It also makes being wrong cheap. Getting a business key wrong in Data Vault means
refactoring hubs and reloading satellites. In HOOK it means editing a view.</p>

<h2 id="where-i-would-be-careful">Where I would be careful</h2>

<p><strong>The lakehouse claim is contested, including by his own readers.</strong> Foad argues
that data lake storage already satisfies Inmon’s non-volatile and time-variant
criteria, so layering HOOK on top supplies the remaining two and yields a
<a href="/glossary/data-lakehouse/">lakehouse</a>. A commenter on that very chapter pushes
back that Inmon’s four criteria describe one integrated layer rather than a
checklist to tick off separately, and that storing files immutably does not make
them a warehouse. I find the objection reasonable. It does not damage HOOK as a
method; it just means the equation is rhetoric rather than a definition.</p>

<p><strong>Deferring modelling is not the same as avoiding it.</strong> The structural work moves
to the consumption layer under the banner of “just enough modelling”. For a small
number of well-understood consumers that is a genuine saving. For many consumers
with conflicting requirements, it may just relocate the cost — and relocate it to
the layer where it is least likely to be governed. Data Vault’s rigour is
expensive precisely because it forces that conversation early.</p>

<p><strong>Adoption is the real constraint.</strong> Data Vault has two decades of books,
certification, consultancies and automation tooling, and you can hire for it.
HOOK is essentially one person’s body of work. Adopting it means no hiring pool,
no certification path, no mature tooling, and a bus factor of one for the
methodology itself. That is a serious cost for a warehouse expected to outlive
several teams.</p>

<h2 id="so-which-should-you-use">So which should you use</h2>

<p>For most organisations, <strong>Data Vault if you need Data Vault’s guarantees and can
actually sustain its discipline; dimensional modelling if you cannot.</strong> That
second clause is doing a lot of work, and it is the same conclusion as
<a href="/essays/data-vault-vs-dimensional-modeling/">Data Vault vs dimensional modelling</a>:
Data Vault earns its keep on auditability and multi-source integration, and is
overkill almost everywhere else.</p>

<p>HOOK is worth reading rather than adopting wholesale, and it is <em>most</em> worth
reading if Data Vault’s rigour is currently failing on your project — because
that is the precise failure mode it was designed around. The keyset idea in
particular transfers cleanly into other methods. You can qualify business keys
across source systems without adopting anything else.</p>

<p>What you should not do is go looking for hooks in your Data Vault
implementation. They are not in there.</p>]]></content><author><name>dataarchitect.studio</name></author><summary type="html"><![CDATA[HOOK is not a Data Vault pattern. It is a separate, lighter alternative built by someone who found Data Vault too heavy. What it is, and where each fits.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://dataarchitect.studio/assets/og-default.png" /><media:content medium="image" url="https://dataarchitect.studio/assets/og-default.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Apache Polaris vs Nessie: The Merger That Never Happened</title><link href="https://dataarchitect.studio/essays/apache-polaris-vs-nessie/" rel="alternate" type="text/html" title="Apache Polaris vs Nessie: The Merger That Never Happened" /><published>2026-08-08T11:00:00+05:30</published><updated>2026-08-08T00:00:00+05:30</updated><id>https://dataarchitect.studio/essays/apache-polaris-vs-nessie</id><content type="html" xml:base="https://dataarchitect.studio/essays/apache-polaris-vs-nessie/"><![CDATA[<p>Start with the thing most comparisons of these two get wrong. In 2024 it was
announced that
<a href="https://www.dremio.com/newsroom/polaris-catalog-to-be-merged-with-nessie-now-available-on-github/">Polaris and Nessie would merge</a>,
with Nessie’s capabilities folded into Polaris and Nessie eventually retired.
<strong>It did not happen.</strong> As of mid-2026 both projects are alive and separate:
<a href="https://amdatalakehouse.substack.com/p/the-state-of-apache-polaris-in-july">Polaris graduated from incubation</a>
and ships on a regular release train, while Nessie continues to serve the
git-for-data niche it invented.</p>

<p>That matters practically, because a fair amount of the guidance you’ll find is
still written against the 2024 plan — advising people to pick Polaris on the
grounds that Nessie is going away. Nessie isn’t going away. Choose on what each
actually does.</p>

<p><strong>The short version: default to Polaris. Choose Nessie only if you want Git-like
branching of the catalog itself, which is a real capability that most estates
don’t need.</strong></p>

<h2 id="polaris-vs-nessie-side-by-side">Polaris vs Nessie, side by side</h2>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>Apache Polaris</th>
      <th>Nessie</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Origin</strong></td>
      <td>Donated by Snowflake, now ASF</td>
      <td>Dremio-backed, open source</td>
    </tr>
    <tr>
      <td><strong>Status, mid-2026</strong></td>
      <td>Graduated, monthly release train</td>
      <td>Independent, active</td>
    </tr>
    <tr>
      <td><strong>Core idea</strong></td>
      <td>Spec-faithful REST catalog + RBAC</td>
      <td>Git semantics over the catalog</td>
    </tr>
    <tr>
      <td><strong>Access control</strong></td>
      <td>Granular table and namespace privileges</td>
      <td>Branch- and commit-oriented</td>
    </tr>
    <tr>
      <td><strong>Credential vending</strong></td>
      <td>Yes</td>
      <td>Limited</td>
    </tr>
    <tr>
      <td><strong>Multi-table transactions</strong></td>
      <td>No</td>
      <td>Yes — that is the point</td>
    </tr>
    <tr>
      <td><strong>Branch / tag / rollback</strong></td>
      <td>No</td>
      <td>Yes</td>
    </tr>
    <tr>
      <td><strong>Engine ecosystem</strong></td>
      <td>Broad: Spark, Flink, Trino, Doris, DuckDB, Presto, Starburst</td>
      <td>Narrower, Dremio-centred</td>
    </tr>
    <tr>
      <td><strong>Managed option</strong></td>
      <td>Snowflake Open Catalog</td>
      <td>Dremio</td>
    </tr>
    <tr>
      <td><strong>Choose it when</strong></td>
      <td>You want a neutral default</td>
      <td>You want data as branchable state</td>
    </tr>
    <tr>
      <td><strong>Fails by</strong></td>
      <td>Being unremarkable, which is the goal</td>
      <td>Solving a problem you don’t have</td>
    </tr>
  </tbody>
</table>

<h2 id="what-nessie-actually-gives-you">What Nessie actually gives you</h2>

<p>Nessie’s distinctive claim is not access control or interop. It’s that the catalog
behaves like Git: you can branch it, commit to the branch, test against it, and
merge or discard. Because the branch covers the <em>whole catalog</em> rather than one
table, you get something genuinely hard to obtain otherwise — <strong>atomic changes
across several tables at once.</strong></p>

<p>The workflow that justifies it looks like this:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">-- Branch the entire catalog, not one table</span>
<span class="k">CREATE</span> <span class="n">BRANCH</span> <span class="n">etl_run_2026_08_08</span> <span class="k">IN</span> <span class="n">nessie</span> <span class="k">FROM</span> <span class="n">main</span><span class="p">;</span>
<span class="n">USE</span> <span class="n">REFERENCE</span> <span class="n">etl_run_2026_08_08</span> <span class="k">IN</span> <span class="n">nessie</span><span class="p">;</span>

<span class="c1">-- Rewrite several related tables; nothing is visible on main yet</span>
<span class="n">MERGE</span> <span class="k">INTO</span> <span class="n">sales</span><span class="p">.</span><span class="n">orders</span>   <span class="p">...;</span>
<span class="n">MERGE</span> <span class="k">INTO</span> <span class="n">sales</span><span class="p">.</span><span class="n">line_item</span> <span class="p">...;</span>
<span class="n">MERGE</span> <span class="k">INTO</span> <span class="n">sales</span><span class="p">.</span><span class="n">customer</span>  <span class="p">...;</span>

<span class="c1">-- Validate the branch, then publish all three tables atomically</span>
<span class="n">MERGE</span> <span class="n">BRANCH</span> <span class="n">etl_run_2026_08_08</span> <span class="k">INTO</span> <span class="n">main</span> <span class="k">IN</span> <span class="n">nessie</span><span class="p">;</span>
</code></pre></div></div>

<p>Anyone who has shipped a pipeline that half-updated three tables and left the
warehouse internally inconsistent for forty minutes can see the appeal. It is the
same instinct as
<a href="/essays/how-to-make-a-data-pipeline-idempotent/">idempotent pipelines</a>, pushed up
to the catalog: make the unsafe window not exist rather than making it short.</p>

<h2 id="what-polaris-actually-gives-you">What Polaris actually gives you</h2>

<p>Polaris is deliberately less interesting, and that is its argument. It implements
the <a href="https://iceberg.apache.org/spec/">Iceberg REST catalog specification</a>
faithfully, vends credentials so the catalog can act as the real security
boundary, and applies conventional role-based privileges at table and namespace
level. Every spec-faithful engine talks to it without special handling.</p>

<p>Its second argument is governance of the project rather than of the data. It sits
with the Apache Software Foundation with contributions from multiple vendors, it
has graduated, and it has a predictable release cadence. If your reason for
avoiding Unity Catalog and Glue is that you don’t want a single vendor holding the
component you cannot casually swap, Polaris is the answer that reasoning points
to. Snowflake’s managed
<a href="https://www.snowflake.com/en/blog/polaris-catalog-open-source/">Open Catalog</a> is
the same codebase operated for you.</p>

<figure style="margin:2rem auto;text-align:center;">
<svg viewBox="0 0 800 340" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="polaris-nessie-t polaris-nessie-d">
  <title id="polaris-nessie-t">Polaris and Nessie optimize for different things</title>
  <desc id="polaris-nessie-d">Two panels. Polaris is shown as a single main line of commits with role-based privileges applied per table and namespace, serving many engines through the Iceberg REST specification. Nessie is shown as a main line with a branch drawn off it, where three tables are rewritten in isolation and then merged back atomically, illustrating catalog-level branching and multi-table transactions. A note beneath records that the 2024 announcement to merge the two projects has not happened as of 2026 and both remain active.</desc>
  <text x="200" y="26" font-size="13" fill="#1c1a17" text-anchor="middle" font-weight="700">Polaris — one governed line</text>
  <text x="600" y="26" font-size="13" fill="#1c1a17" text-anchor="middle" font-weight="700">Nessie — branchable catalog</text>
  <line x1="60" y1="110" x2="340" y2="110" stroke="#1c1a17" stroke-width="2.5" />
  <circle cx="110" cy="110" r="7" fill="#c8472b" />
  <circle cx="200" cy="110" r="7" fill="#c8472b" />
  <circle cx="290" cy="110" r="7" fill="#c8472b" />
  <text x="200" y="136" font-size="10" fill="#56514a" text-anchor="middle">main — commits land directly</text>
  <rect x="60" y="156" width="280" height="26" rx="4" fill="#f6f3ec" stroke="#ddd6c8" stroke-width="1.2" />
  <text x="200" y="174" font-size="10" fill="#56514a" text-anchor="middle">RBAC per table + namespace</text>
  <rect x="60" y="190" width="280" height="26" rx="4" fill="#f6f3ec" stroke="#ddd6c8" stroke-width="1.2" />
  <text x="200" y="208" font-size="10" fill="#56514a" text-anchor="middle">credential vending · REST spec</text>
  <text x="200" y="240" font-size="10" fill="#8b857a" text-anchor="middle">Spark · Flink · Trino · DuckDB · Presto</text>
  <line x1="460" y1="110" x2="740" y2="110" stroke="#1c1a17" stroke-width="2.5" />
  <circle cx="500" cy="110" r="7" fill="#1c1a17" />
  <circle cx="710" cy="110" r="7" fill="#c8472b" />
  <path d="M540 110 C 570 110, 570 176, 600 176" fill="none" stroke="#c8472b" stroke-width="2" />
  <path d="M660 176 C 690 176, 690 110, 710 110" fill="none" stroke="#c8472b" stroke-width="2" />
  <circle cx="600" cy="176" r="6" fill="#c8472b" />
  <circle cx="630" cy="176" r="6" fill="#c8472b" />
  <circle cx="660" cy="176" r="6" fill="#c8472b" />
  <text x="630" y="200" font-size="10" fill="#a4391f" text-anchor="middle">branch: 3 tables rewritten together</text>
  <text x="712" y="98" font-size="10" fill="#a4391f" text-anchor="end">merged atomically →</text>
  <text x="600" y="240" font-size="10" fill="#8b857a" text-anchor="middle">multi-table transactions · rollback · zero-copy test envs</text>
  <line x1="400" y1="46" x2="400" y2="258" stroke="#ddd6c8" stroke-width="1.5" stroke-dasharray="4 4" />
  <rect x="140" y="276" width="520" height="42" rx="6" fill="#f6f3ec" stroke="#ddd6c8" stroke-width="1.5" />
  <text x="400" y="294" font-size="11" fill="#1c1a17" text-anchor="middle">announced 2024: Nessie merges into Polaris, Nessie retired</text>
  <text x="400" y="311" font-size="11" fill="#a4391f" text-anchor="middle">mid-2026: it has not happened. Both projects are active and separate.</text>
</svg>
<figcaption style="font-family:'IBM Plex Mono',monospace;font-size:0.75rem;color:#8b857a;margin-top:0.6rem;">One line versus branchable state. And a merger announcement that guidance still repeats two years on.</figcaption>
</figure>

<h2 id="why-the-merger-stalled-and-why-you-shouldnt-plan-around-it-resuming">Why the merger stalled, and why you shouldn’t plan around it resuming</h2>

<p>The two projects were never as complementary as the announcement implied. Polaris
is organised around a single authoritative pointer per table with privileges
layered on top — the model the
<a href="/essays/what-is-an-open-table-format/">Iceberg REST spec</a> assumes. Nessie is
organised around versioned references, where “current” is a property of the branch
you’re reading. Merging them is not a matter of porting features; it means
choosing which of two models of catalog state is the real one.</p>

<p>Practical implication: <strong>do not choose either catalog on the assumption they will
converge.</strong> If branching is what you need, run Nessie and accept a narrower engine
ecosystem. If it isn’t, run Polaris and don’t pay for a capability you won’t use.</p>

<h2 id="the-decision-rule">The decision rule</h2>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Do you need atomic changes across MULTIPLE tables,
or branch-and-test against a copy of production catalog state?
   → Nessie. Nothing else does this cleanly.

Do you want a neutral, spec-faithful catalog with broad engine support
and conventional RBAC?
   → Polaris. Managed: Snowflake Open Catalog.

Not sure, and no multi-table transaction has ever hurt you?
   → Polaris. The branching model has real operational cost,
     and inventing a use for it after the fact is how estates
     accumulate machinery nobody maintains.
</code></pre></div></div>

<p>And the honest framing that applies to
<a href="/essays/how-to-choose-an-iceberg-catalog/">every catalog decision</a>: the data
files never move, so this feels like a reversible choice. It isn’t. Every engine
config, every access policy, and all accumulated audit history live in the
catalog, which is exactly why it deserves the evaluation weeks the format choice
no longer needs.</p>]]></content><author><name>dataarchitect.studio</name></author><summary type="html"><![CDATA[Polaris and Nessie were announced as merging in 2024. They didn't. Here's where each actually stands in 2026, and which one your estate should run.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://dataarchitect.studio/assets/og-default.png" /><media:content medium="image" url="https://dataarchitect.studio/assets/og-default.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Unity Catalog vs AWS Glue Data Catalog: How to Choose</title><link href="https://dataarchitect.studio/essays/unity-catalog-vs-glue-catalog/" rel="alternate" type="text/html" title="Unity Catalog vs AWS Glue Data Catalog: How to Choose" /><published>2026-08-08T09:00:00+05:30</published><updated>2026-08-29T00:00:00+05:30</updated><id>https://dataarchitect.studio/essays/unity-catalog-vs-glue-catalog</id><content type="html" xml:base="https://dataarchitect.studio/essays/unity-catalog-vs-glue-catalog/"><![CDATA[<p>Both of these hold the pointer that makes an
<a href="/essays/what-is-an-open-table-format/">Iceberg table</a> a table, and the choice
between them is not really a technical one. <strong>AWS Glue Data Catalog is the
zero-effort default if your compute lives in AWS: serverless, IAM-governed, with
crawlers that register tables for you. Unity Catalog buys materially deeper
governance — fine-grained access control, lineage, and audit for data and AI
assets in one place — and charges for it in platform commitment.</strong></p>

<p>The decision rule is short: <strong>follow your compute.</strong> Deep-AWS shops take Glue.
Databricks-centred estates take Unity. And the case that trips people up — running
Databricks <em>on</em> AWS — usually ends with both, which is the part most comparisons
miss entirely.</p>

<h2 id="unity-catalog-vs-glue-side-by-side">Unity Catalog vs Glue, side by side</h2>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>AWS Glue Data Catalog</th>
      <th>Unity Catalog</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Steward</strong></td>
      <td>AWS</td>
      <td>Databricks; core is Apache 2.0 under LF AI &amp; Data</td>
    </tr>
    <tr>
      <td><strong>Operating model</strong></td>
      <td>Serverless, no infrastructure</td>
      <td>Managed in Databricks, or self-hosted OSS</td>
    </tr>
    <tr>
      <td><strong>Access control</strong></td>
      <td>IAM, plus Lake Formation for finer grain</td>
      <td>Native fine-grained: catalog, schema, table, row, column</td>
    </tr>
    <tr>
      <td><strong>Lineage</strong></td>
      <td>Limited</td>
      <td>First-class, captured automatically</td>
    </tr>
    <tr>
      <td><strong>Audit</strong></td>
      <td>CloudTrail</td>
      <td>Unified across data and AI assets</td>
    </tr>
    <tr>
      <td><strong>Table registration</strong></td>
      <td>Crawlers discover and register automatically</td>
      <td>Explicit registration</td>
    </tr>
    <tr>
      <td><strong>Iceberg REST API</strong></td>
      <td>Yes</td>
      <td>Yes, with credential vending</td>
    </tr>
    <tr>
      <td><strong>Non-tabular assets</strong></td>
      <td>Tables</td>
      <td>Tables, volumes, models, functions</td>
    </tr>
    <tr>
      <td><strong>Gravity</strong></td>
      <td>AWS</td>
      <td>Databricks</td>
    </tr>
    <tr>
      <td><strong>Costs you</strong></td>
      <td>Cloud lock-in</td>
      <td>Platform commitment</td>
    </tr>
    <tr>
      <td><strong>Fails by</strong></td>
      <td>Governance thinning out as the estate grows</td>
      <td>Being overkill without the platform around it</td>
    </tr>
  </tbody>
</table>

<h2 id="what-glue-is-actually-good-at">What Glue is actually good at</h2>

<p>Glue’s advantage is that it is <em>already there</em>. If your estate is Athena, EMR,
Redshift Spectrum, and Glue ETL, the catalog requires no decision, no deployment,
and no separate bill of its own. Crawlers infer schemas and register tables
without anyone writing DDL, which for a lake accumulating files from many
producers is a genuine reduction in toil.</p>

<p>The governance is IAM-shaped, which is a real strength if your organisation
already runs on IAM and a real limit otherwise. Table- and column-level control
means adding Lake Formation on top, and the lineage story is thin. That’s fine
until the estate has enough consumers that “who can see this column, and who
touched it last quarter” becomes a question someone has to answer in writing.</p>

<h2 id="what-unity-catalog-is-actually-good-at">What Unity Catalog is actually good at</h2>

<p>Unity’s advantage is that governance is one surface rather than four. Permissions,
lineage, and audit cover tables, files, ML models, and functions together, which
matters once AI assets stop being a side project. Where Glue tells you IAM allowed
a call, Unity tells you which notebook read which column and what it fed.</p>

<p>It’s also less closed than its reputation suggests. Databricks
<a href="https://www.databricks.com/blog/open-sourcing-unity-catalog">open-sourced the core under Apache 2.0</a>
with a published spec, and the project sits with the LF AI &amp; Data Foundation.
Worth being precise about what that means, though: the OSS project and the managed
Databricks service are not the same thing, and most of the governance depth people
buy Unity Catalog <em>for</em> lives in the managed product. Treat “it’s open source” as a
statement about lock-in risk, not a statement about features.</p>

<figure style="margin:2rem auto;text-align:center;">
<svg viewBox="0 0 800 330" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="uc-glue-t uc-glue-d">
  <title id="uc-glue-t">Choosing between Unity Catalog and AWS Glue Data Catalog</title>
  <desc id="uc-glue-d">A decision tree starting from where compute lives. Estates centred on AWS services such as Athena, EMR and Redshift branch to Glue Data Catalog, which is serverless and IAM-governed. Estates centred on Databricks branch to Unity Catalog, which provides fine-grained access control, lineage and audit. A third path shows Databricks running on AWS, which commonly ends with both: Unity as the governing catalog with Glue federated in for legacy tables. A final note records that both now expose an Iceberg REST endpoint, so Iceberg support no longer separates them.</desc>
  <rect x="250" y="16" width="300" height="50" rx="6" fill="#1c1a17" />
  <text x="400" y="38" font-size="13" fill="#f6f3ec" text-anchor="middle">Where does your compute live?</text>
  <text x="400" y="56" font-size="10" fill="#8b857a" text-anchor="middle">not: which catalog has more features</text>
  <line x1="310" y1="66" x2="150" y2="110" stroke="#cabfac" stroke-width="2" />
  <line x1="400" y1="66" x2="400" y2="110" stroke="#cabfac" stroke-width="2" />
  <line x1="490" y1="66" x2="650" y2="110" stroke="#cabfac" stroke-width="2" />
  <rect x="40" y="110" width="220" height="72" rx="6" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <text x="150" y="134" font-size="12" fill="#1c1a17" text-anchor="middle" font-weight="700">Athena · EMR · Redshift</text>
  <text x="150" y="155" font-size="11" fill="#c8472b" text-anchor="middle" font-weight="700">→ Glue Data Catalog</text>
  <text x="150" y="173" font-size="11" fill="#56514a" text-anchor="middle">serverless · IAM · crawlers</text>
  <rect x="290" y="110" width="220" height="72" rx="6" fill="#c8472b" />
  <text x="400" y="134" font-size="12" fill="#f6f3ec" text-anchor="middle" font-weight="700">Databricks on AWS</text>
  <text x="400" y="155" font-size="11" fill="#f6f3ec" text-anchor="middle" font-weight="700">→ usually BOTH</text>
  <text x="400" y="173" font-size="11" fill="#f6f3ec" text-anchor="middle">Unity governs · Glue federated in</text>
  <rect x="540" y="110" width="220" height="72" rx="6" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <text x="650" y="134" font-size="12" fill="#1c1a17" text-anchor="middle" font-weight="700">Databricks-centred</text>
  <text x="650" y="155" font-size="11" fill="#c8472b" text-anchor="middle" font-weight="700">→ Unity Catalog</text>
  <text x="650" y="173" font-size="11" fill="#56514a" text-anchor="middle">lineage · fine-grained · audit</text>
  <rect x="170" y="216" width="460" height="46" rx="6" fill="#f6f3ec" stroke="#ddd6c8" stroke-width="1.5" />
  <text x="400" y="236" font-size="11" fill="#1c1a17" text-anchor="middle">both now expose an Iceberg REST endpoint</text>
  <text x="400" y="253" font-size="10" fill="#8b857a" text-anchor="middle">Iceberg support stopped being the differentiator</text>
  <text x="400" y="294" font-size="12" fill="#8b857a" text-anchor="middle">the middle box is the case most comparisons leave out, and it is the common one</text>
</svg>
<figcaption style="font-family:'IBM Plex Mono',monospace;font-size:0.75rem;color:#8b857a;margin-top:0.6rem;">Follow the compute. And notice that the middle branch is not a compromise, it is the usual answer.</figcaption>
</figure>

<h2 id="the-case-nobody-writes-about-both">The case nobody writes about: both</h2>

<p>Running Databricks on AWS is not an edge case, and it does not force a choice.
Unity Catalog ships federation connectors for Glue and Hive Metastore, so an
existing Glue catalog mounts as a foreign catalog inside Unity, and
<a href="https://aws.amazon.com/blogs/big-data/access-databricks-unity-catalog-data-using-catalog-federation-in-the-aws-glue-data-catalog/">AWS built federation in the other direction</a>
so Glue-side engines can reach Unity-governed tables.</p>

<p>The pattern that works: <strong>Unity as the governing catalog for anything new, Glue
federated in for the legacy estate</strong>, migrating tables when there’s a reason
rather than as a project. You get one governance surface without a big-bang
migration, which matters because
<a href="/essays/how-to-choose-an-iceberg-catalog/">switching catalogs is the painful migration in a lakehouse</a>
— the data files never move, but every engine config, every access policy, and
all accumulated audit history do.</p>

<h2 id="where-the-iceberg-story-landed">Where the Iceberg story landed</h2>

<p>This used to be a real differentiator and isn’t anymore. Unity Catalog implements
the <a href="https://iceberg.apache.org/spec/">Iceberg REST Catalog API</a> including
credential vending, so Spark, Flink, Trino and anything else spec-faithful can
read and write Unity-managed tables. Glue exposes an Iceberg endpoint too.</p>

<div class="language-properties highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Same Spark client, either catalog behind the URI.
</span><span class="py">spark.sql.extensions</span> <span class="p">=</span> <span class="s">org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions</span>
<span class="py">spark.sql.catalog.lake</span> <span class="p">=</span> <span class="s">org.apache.iceberg.spark.SparkCatalog</span>
<span class="py">spark.sql.catalog.lake.type</span> <span class="p">=</span> <span class="s">rest</span>
<span class="py">spark.sql.catalog.lake.uri</span> <span class="p">=</span> <span class="s">https://&lt;unity-or-glue-endpoint&gt;/iceberg</span>
<span class="py">spark.sql.catalog.lake.warehouse</span> <span class="p">=</span> <span class="s">my_warehouse</span>
<span class="py">spark.sql.catalog.lake.rest.auth.type</span> <span class="p">=</span> <span class="s">oauth2</span>
<span class="c"># token injected from a secret manager at submit time, never in a notebook cell
</span><span class="py">spark.sql.catalog.lake.token</span> <span class="p">=</span> <span class="s">${ICEBERG_CATALOG_TOKEN}</span>
</code></pre></div></div>

<p>That config is nearly identical either way, which is the point: <strong>the wire
protocol converged, so the decision moved entirely to governance and gravity.</strong></p>

<h2 id="the-decision-rule">The decision rule</h2>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Compute is Athena / EMR / Redshift, governance needs are IAM-shaped?
   → Glue. Stop optimising; it is already there.

Compute is Databricks, and you need lineage, column-level control,
or one audit surface across data AND AI assets?
   → Unity Catalog.

Databricks on AWS with a legacy Glue estate?
   → Both. Unity governs, Glue federates in. This is the common case.

Deliberately avoiding both vendors' gravity?
   → Neither — that is the Polaris question, one layer over.
</code></pre></div></div>

<p>The trap worth naming: choosing Unity Catalog for governance you have no
organisational capacity to operate. Fine-grained access control is only worth its
setup cost if someone owns the policies and reviews them. Absent that, you’ve
bought a more expensive metastore and inherited the same
<a href="/essays/data-quality-problems-are-org-chart-problems/">org-chart problem</a> you
started with. Glue plus genuine ownership beats Unity plus nobody, every time.</p>]]></content><author><name>dataarchitect.studio</name></author><summary type="html"><![CDATA[Glue is the zero-effort default inside AWS; Unity Catalog buys deeper governance for platform commitment. Both now speak Iceberg REST, and they federate.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://dataarchitect.studio/assets/og-default.png" /><media:content medium="image" url="https://dataarchitect.studio/assets/og-default.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Parquet vs ORC: What the Benchmark Actually Showed</title><link href="https://dataarchitect.studio/essays/parquet-vs-orc/" rel="alternate" type="text/html" title="Parquet vs ORC: What the Benchmark Actually Showed" /><published>2026-08-08T07:00:00+05:30</published><updated>2026-08-08T00:00:00+05:30</updated><id>https://dataarchitect.studio/essays/parquet-vs-orc</id><content type="html" xml:base="https://dataarchitect.studio/essays/parquet-vs-orc/"><![CDATA[<p>Both are columnar, both prune columns, both skip data with embedded statistics.
The architecture is close enough that the choice usually gets settled by a piece
of received wisdom instead: <em>ORC compresses a bit better, Parquet has the better
ecosystem.</em></p>

<p>The second half is true. <strong>The first half did not survive measurement.</strong>
<a href="/essays/parquet-vs-orc-vs-avro-benchmark/">Benchmarked at matched compression codecs</a>
on a realistic 3-million-row table, Parquet was <strong>23–26% smaller than ORC</strong>, and
it also wrote faster and scanned faster. Not marginally, and not in the direction
the guidance predicts.</p>

<p><strong>For a new lakehouse: Parquet. Choose ORC when a Hive-centric stack expects it,
not because you believe it compresses better.</strong></p>

<h2 id="where-the-compression-claim-comes-from">Where the compression claim comes from</h2>

<p>Most published comparisons compare <strong>Parquet with Snappy</strong> against <strong>ORC with
Zlib</strong>. Zlib compresses harder than Snappy, so ORC’s file comes out smaller, and
the difference gets written up as a format difference. It’s a codec difference.</p>

<p>Matched properly, at 1,000,000 rows:</p>

<table>
  <thead>
    <tr>
      <th>Compression</th>
      <th>Parquet</th>
      <th>ORC</th>
      <th>Difference</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Snappy</strong></td>
      <td>24.3 MB</td>
      <td>32.9 MB</td>
      <td>Parquet <strong>26% smaller</strong></td>
    </tr>
    <tr>
      <td><strong>DEFLATE</strong> (gzip / zlib)</td>
      <td>17.8 MB</td>
      <td>23.1 MB</td>
      <td>Parquet <strong>23% smaller</strong></td>
    </tr>
  </tbody>
</table>

<p>And at 3,000,000 rows across the full set of measurements:</p>

<table>
  <thead>
    <tr>
      <th>Format</th>
      <th>Size</th>
      <th>Write</th>
      <th>Full scan</th>
      <th>2 of 11 columns</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>ORC (snappy)</td>
      <td>96.2 MB</td>
      <td>1.84 s</td>
      <td>0.676 s</td>
      <td>0.109 s</td>
    </tr>
    <tr>
      <td>Parquet (snappy)</td>
      <td>72.3 MB</td>
      <td>1.35 s</td>
      <td>0.420 s</td>
      <td>0.055 s</td>
    </tr>
    <tr>
      <td>ORC (zlib)</td>
      <td>68.1 MB</td>
      <td>4.67 s</td>
      <td>1.289 s</td>
      <td>0.262 s</td>
    </tr>
    <tr>
      <td><strong>Parquet (zstd)</strong></td>
      <td><strong>56.6 MB</strong></td>
      <td><strong>1.32 s</strong></td>
      <td><strong>0.377 s</strong></td>
      <td><strong>0.054 s</strong></td>
    </tr>
  </tbody>
</table>

<p>The row worth staring at is ORC with zlib, because zlib is the default nobody
changes: it writes <strong>3.5× slower</strong> than Parquet with zstd and still produces a
<strong>20% larger file</strong>.</p>

<p>I want to be careful about the scope of this. It is one dataset shape, using
pyarrow’s ORC writer at default settings — not Hive’s, and not tuned. ORC’s
reputation was built in tuned Hive stacks on different data and may well hold
there. What the measurement supports is narrower and still useful: <strong>“ORC
compresses better” is not a default you can assume.</strong></p>

<figure style="margin:2rem auto;text-align:center;">
<svg viewBox="0 0 800 300" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="pq-orc-t pq-orc-d">
  <title id="pq-orc-t">Parquet versus ORC at matched compression codecs</title>
  <desc id="pq-orc-d">Paired bars at one million rows. With Snappy, Parquet is 24.3 megabytes against ORC's 32.9, making Parquet 26 percent smaller. With DEFLATE, Parquet is 17.8 megabytes against ORC's 23.1, making Parquet 23 percent smaller. A note explains that most published comparisons instead pit Parquet with Snappy against ORC with Zlib, which measures a codec difference and reports it as a format difference.</desc>
  <text x="20" y="24" font-size="12" fill="#1c1a17" font-weight="700">Matched codec, 1M rows — smaller is better</text>
  <text x="150" y="62" font-size="11" fill="#56514a" text-anchor="end">Parquet · snappy</text>
  <rect x="160" y="50" width="243" height="17" rx="2" fill="#c8472b" />
  <text x="411" y="63" font-size="10" fill="#a4391f" font-weight="700">24.3 MB</text>
  <text x="150" y="88" font-size="11" fill="#56514a" text-anchor="end">ORC · snappy</text>
  <rect x="160" y="76" width="329" height="17" rx="2" fill="#cabfac" />
  <text x="497" y="89" font-size="10" fill="#56514a">32.9 MB — Parquet 26% smaller</text>
  <text x="150" y="128" font-size="11" fill="#56514a" text-anchor="end">Parquet · DEFLATE</text>
  <rect x="160" y="116" width="178" height="17" rx="2" fill="#c8472b" />
  <text x="346" y="129" font-size="10" fill="#a4391f" font-weight="700">17.8 MB</text>
  <text x="150" y="154" font-size="11" fill="#56514a" text-anchor="end">ORC · DEFLATE</text>
  <rect x="160" y="142" width="231" height="17" rx="2" fill="#cabfac" />
  <text x="399" y="155" font-size="10" fill="#56514a">23.1 MB — Parquet 23% smaller</text>
  <line x1="20" y1="182" x2="780" y2="182" stroke="#ddd6c8" stroke-width="1" />
  <text x="20" y="208" font-size="11" fill="#1c1a17" font-weight="700">How the usual comparison gets it backwards</text>
  <text x="20" y="230" font-size="11" fill="#56514a">Parquet + Snappy  vs  ORC + Zlib  →  ORC looks smaller</text>
  <text x="20" y="250" font-size="11" fill="#a4391f">but that compares CODECS, not formats. Zlib compresses harder than Snappy.</text>
  <text x="20" y="276" font-size="11" fill="#8b857a">Match the codec and the result reverses. One dataset, pyarrow's ORC writer, defaults.</text>
</svg>
<figcaption style="font-family:'IBM Plex Mono',monospace;font-size:0.75rem;color:#8b857a;margin-top:0.6rem;">The comparison everyone runs measures the codec. Match it, and the ranking flips.</figcaption>
</figure>

<h2 id="what-genuinely-separates-them">What genuinely separates them</h2>

<p>Strip out the compression argument and three real differences remain.</p>

<p><strong>Ecosystem gravity.</strong> Parquet is the default across Spark, the cloud warehouses,
dbt, and every <a href="/essays/what-is-an-open-table-format/">open table format</a>. ORC’s
home is Hive and Trino. This is the difference that actually decides most cases,
and it always was.</p>

<p><strong>Hive ACID.</strong> ORC supports ACID transactions <em>within Hive</em>, which mattered a great
deal before table formats existed. It matters much less now that
<a href="/essays/iceberg-vs-delta-lake/">Iceberg and Delta</a> provide transactions over any
underlying file format — including ORC, but conventionally Parquet.</p>

<p><strong>Tooling depth.</strong> More readers, more writers, more debugging tools, more people
who have hit your problem before. Unglamorous and worth a lot at 2am.</p>

<h2 id="the-codec-choice-matters-more-than-the-format-choice">The codec choice matters more than the format choice</h2>

<p>This is the part of the measurement with the largest practical payoff, and it
applies to both formats:</p>

<table>
  <thead>
    <tr>
      <th>Codec (Parquet, 1M rows)</th>
      <th>Size</th>
      <th>Write time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>zstd</td>
      <td>19.3 MB</td>
      <td>0.54 s</td>
    </tr>
    <tr>
      <td>gzip</td>
      <td>17.8 MB</td>
      <td>16.00 s</td>
    </tr>
  </tbody>
</table>

<p>Gzip bought an 8% smaller file for a <strong>30× slower write</strong>. For analytical tables,
written once and read many times, that is a bad trade; for continuously written
tables it is an indefensible one.</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">-- Usually a bigger win than switching format:</span>
<span class="k">ALTER</span> <span class="k">TABLE</span> <span class="n">lake</span><span class="p">.</span><span class="n">sales</span><span class="p">.</span><span class="n">orders</span>
  <span class="k">SET</span> <span class="n">TBLPROPERTIES</span> <span class="p">(</span><span class="s1">'write.parquet.compression-codec'</span> <span class="o">=</span> <span class="s1">'zstd'</span><span class="p">);</span>
</code></pre></div></div>

<p>If you’re on Parquet with gzip or ORC with zlib because that was the default when
the pipeline was written, changing that one property is probably the cheapest
performance work available to you, and nothing downstream has to change.</p>

<h2 id="the-decision-rule">The decision rule</h2>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Greenfield lakehouse, or any Spark / cloud-warehouse / dbt stack?
   → Parquet, with zstd. Do not deliberate.

Established Hive or Trino platform that already standardised on ORC,
or you depend on Hive ACID?
   → Stay on ORC. Migrating for a compression claim that
     did not survive measurement is not worth the churn.

Choosing ORC because you read that it compresses better?
   → Measure it on your data first. On mine it was 23-26% larger.
</code></pre></div></div>

<p>The broader point, which is why this essay exists at all: the ORC compression
claim is repeated in nearly every comparison of the two formats, including one
I published on this site before I measured anything. It propagated because it
sounds plausible and because the benchmark that would test it takes an afternoon
that nobody spends. <a href="/benchmarks/file-format-benchmark.py">The script is published</a>
and the dataset is seeded. If your table behaves differently, that is worth
knowing, and I’d rather be corrected than repeated.</p>]]></content><author><name>dataarchitect.studio</name></author><summary type="html"><![CDATA[Received wisdom says ORC compresses better. Measured at matched codecs on 3M rows, Parquet was 23-26% smaller and faster on every axis. Here's the honest call.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://dataarchitect.studio/assets/og-default.png" /><media:content medium="image" url="https://dataarchitect.studio/assets/og-default.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Parquet vs ORC vs Avro Benchmark: Compression and Read Speed</title><link href="https://dataarchitect.studio/essays/parquet-vs-orc-vs-avro-benchmark/" rel="alternate" type="text/html" title="Parquet vs ORC vs Avro Benchmark: Compression and Read Speed" /><published>2026-07-26T14:00:00+05:30</published><updated>2026-08-08T00:00:00+05:30</updated><id>https://dataarchitect.studio/essays/parquet-vs-orc-vs-avro-benchmark</id><content type="html" xml:base="https://dataarchitect.studio/essays/parquet-vs-orc-vs-avro-benchmark/"><![CDATA[<p><strong>Parquet came out 23–26% smaller than ORC at matched compression codecs</strong> — the
opposite of the claim you’ll find in most comparisons, including one I published
on this site. That result is the reason this piece exists.</p>

<p>Every comparison of file formats I have written, including
<a href="/essays/parquet-vs-orc-vs-avro/">the one on this site</a>, asserts things about
compression and scan speed without measuring them. So I measured them.</p>

<p>What follows is one benchmark on one dataset: 3 million rows, 11 columns,
322 MB uncompressed in memory, written to Parquet, ORC, Avro, and CSV and read
back in several ways. <a href="/benchmarks/file-format-benchmark.py">The script is published</a>,
the dataset is generated from a fixed seed, and
<a href="/benchmarks/results-2026-07-26.json">the raw results are in the repo</a>. If you
rerun it and get something different, I want to know.</p>

<p>One result contradicted what I had previously written on this site. That one is
first.</p>

<h2 id="the-result-i-did-not-expect">The result I did not expect</h2>

<p>The received wisdom, repeated in most comparisons including my own, is that ORC
usually edges Parquet on compression ratio. On this dataset it didn’t — not at
any matched codec, and not by a small margin.</p>

<p>Comparing like with like matters here, because most published comparisons put
Parquet with Snappy against ORC with Zlib and call the difference a format
difference. It isn’t; it’s a codec difference. Matched properly, at 1,000,000
rows:</p>

<table>
  <thead>
    <tr>
      <th>Compression</th>
      <th>Parquet</th>
      <th>ORC</th>
      <th>Difference</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Snappy</strong></td>
      <td>24.3 MB</td>
      <td>32.9 MB</td>
      <td>Parquet <strong>26% smaller</strong></td>
    </tr>
    <tr>
      <td><strong>DEFLATE</strong> (gzip / zlib)</td>
      <td>17.8 MB</td>
      <td>23.1 MB</td>
      <td>Parquet <strong>23% smaller</strong></td>
    </tr>
  </tbody>
</table>

<p>I want to be careful about what this does and does not show. It does <em>not</em> show
that ORC compresses worse than Parquet in general. It shows that on one
realistically-shaped table, using the ORC writer most people in a Python or
Spark-adjacent stack will actually reach for, at default settings, the claim
went the other way. ORC’s reputation was built in tuned Hive stacks on different
data, and it may well hold there.</p>

<p>What it does show is narrower and still useful: <strong>“ORC compresses better” is not
a default you can assume.</strong> It is a claim that needs measuring on your data
before it drives an architectural decision. I had repeated it without measuring
it, which is the point of this essay.</p>

<h2 id="the-main-table">The main table</h2>

<p>3,000,000 rows × 11 columns. Sizes are on disk; times are medians, not
best-of-N. Read times are warm-cache.</p>

<table>
  <thead>
    <tr>
      <th>Format</th>
      <th>Size</th>
      <th>vs CSV</th>
      <th>Write</th>
      <th>Full scan</th>
      <th>2 of 11 columns</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>CSV</td>
      <td>342.6 MB</td>
      <td>1.0×</td>
      <td>1.80 s</td>
      <td>1.398 s</td>
      <td>0.499 s</td>
    </tr>
    <tr>
      <td>CSV + gzip</td>
      <td>92.1 MB</td>
      <td>3.7×</td>
      <td>13.40 s</td>
      <td>—</td>
      <td>—</td>
    </tr>
    <tr>
      <td>ORC (snappy)</td>
      <td>96.2 MB</td>
      <td>3.6×</td>
      <td>1.84 s</td>
      <td>0.676 s</td>
      <td>0.109 s</td>
    </tr>
    <tr>
      <td><strong>Parquet (snappy)</strong></td>
      <td>72.3 MB</td>
      <td>4.7×</td>
      <td>1.35 s</td>
      <td>0.420 s</td>
      <td>0.055 s</td>
    </tr>
    <tr>
      <td>ORC (zlib)</td>
      <td>68.1 MB</td>
      <td>5.0×</td>
      <td>4.67 s</td>
      <td>1.289 s</td>
      <td>0.262 s</td>
    </tr>
    <tr>
      <td><strong>Parquet (zstd)</strong></td>
      <td><strong>56.6 MB</strong></td>
      <td><strong>6.1×</strong></td>
      <td><strong>1.32 s</strong></td>
      <td><strong>0.377 s</strong></td>
      <td><strong>0.054 s</strong></td>
    </tr>
  </tbody>
</table>

<p>Parquet with zstd won every column on this dataset: smallest file, fastest
write, fastest full scan, fastest pruned scan. That is an unusually clean
result and I’d treat it with suspicion if I hadn’t run it myself.</p>

<p>Note the ORC/zlib row in particular, because zlib is the default a lot of people
never change: it writes <strong>3.5× slower than Parquet/zstd</strong> and still produces a
<strong>20% larger file</strong>.</p>

<figure style="margin:2rem auto;text-align:center;">
<svg viewBox="0 0 800 340" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="fmt-bench-t fmt-bench-d">
  <title id="fmt-bench-t">Measured file size and pruned-read time by format</title>
  <desc id="fmt-bench-d">Horizontal bars for on-disk size across six format and codec combinations on three million rows: CSV at 342.6 megabytes, gzipped CSV at 92.1, ORC with Snappy at 96.2, Parquet with Snappy at 72.3, ORC with Zlib at 68.1, and Parquet with Zstd smallest at 56.6 megabytes. Beneath, the time to read two of eleven columns: 0.499 seconds for CSV, 0.262 for ORC Zlib, 0.109 for ORC Snappy, and 0.054 for Parquet Zstd, a nine-fold gap between CSV and Parquet.</desc>
  <text x="20" y="22" font-size="12" fill="#1c1a17" font-weight="700">On-disk size — 3M rows, 11 columns</text>
  <text x="150" y="52" font-size="11" fill="#56514a" text-anchor="end">CSV</text>
  <rect x="158" y="41" width="560" height="15" rx="2" fill="#1c1a17" />
  <text x="726" y="53" font-size="10" fill="#56514a">342.6 MB</text>
  <text x="150" y="76" font-size="11" fill="#56514a" text-anchor="end">ORC snappy</text>
  <rect x="158" y="65" width="157" height="15" rx="2" fill="#cabfac" />
  <text x="323" y="77" font-size="10" fill="#56514a">96.2</text>
  <text x="150" y="100" font-size="11" fill="#56514a" text-anchor="end">CSV gzip</text>
  <rect x="158" y="89" width="151" height="15" rx="2" fill="#cabfac" />
  <text x="317" y="101" font-size="10" fill="#56514a">92.1</text>
  <text x="150" y="124" font-size="11" fill="#56514a" text-anchor="end">Parquet snappy</text>
  <rect x="158" y="113" width="118" height="15" rx="2" fill="#c8472b" opacity="0.55" />
  <text x="284" y="125" font-size="10" fill="#56514a">72.3</text>
  <text x="150" y="148" font-size="11" fill="#56514a" text-anchor="end">ORC zlib</text>
  <rect x="158" y="137" width="111" height="15" rx="2" fill="#cabfac" />
  <text x="277" y="149" font-size="10" fill="#56514a">68.1</text>
  <text x="150" y="172" font-size="11" fill="#1c1a17" text-anchor="end" font-weight="700">Parquet zstd</text>
  <rect x="158" y="161" width="93" height="15" rx="2" fill="#c8472b" />
  <text x="259" y="173" font-size="10" fill="#a4391f" font-weight="700">56.6</text>
  <line x1="20" y1="196" x2="780" y2="196" stroke="#ddd6c8" stroke-width="1" />
  <text x="20" y="220" font-size="12" fill="#1c1a17" font-weight="700">Time to read 2 of 11 columns</text>
  <text x="150" y="250" font-size="11" fill="#56514a" text-anchor="end">CSV</text>
  <rect x="158" y="239" width="560" height="15" rx="2" fill="#1c1a17" />
  <text x="726" y="251" font-size="10" fill="#56514a">0.499 s</text>
  <text x="150" y="274" font-size="11" fill="#56514a" text-anchor="end">ORC zlib</text>
  <rect x="158" y="263" width="294" height="15" rx="2" fill="#cabfac" />
  <text x="460" y="275" font-size="10" fill="#56514a">0.262</text>
  <text x="150" y="298" font-size="11" fill="#56514a" text-anchor="end">ORC snappy</text>
  <rect x="158" y="287" width="122" height="15" rx="2" fill="#cabfac" />
  <text x="288" y="299" font-size="10" fill="#56514a">0.109</text>
  <text x="150" y="322" font-size="11" fill="#1c1a17" text-anchor="end" font-weight="700">Parquet zstd</text>
  <rect x="158" y="311" width="61" height="15" rx="2" fill="#c8472b" />
  <text x="227" y="323" font-size="10" fill="#a4391f" font-weight="700">0.054 — 9.2× faster than CSV</text>
</svg>
<figcaption style="font-family:'IBM Plex Mono',monospace;font-size:0.75rem;color:#8b857a;margin-top:0.6rem;">Measured, not asserted. Absolute times are specific to a slow two-core box; the ratios are the finding.</figcaption>
</figure>

<h2 id="column-pruning-is-where-the-money-is">Column pruning is where the money is</h2>

<p>Compression gets the attention. Pruning is the bigger effect.</p>

<p>Reading 2 of 11 columns instead of all 11:</p>

<ul>
  <li><strong>Parquet: 6.9× faster</strong> (0.377 s → 0.054 s)</li>
  <li><strong>ORC: 4.9× faster</strong> (1.289 s → 0.262 s)</li>
  <li><strong>CSV: 2.8× faster</strong> (1.398 s → 0.499 s)</li>
  <li><strong>Avro: 0.96× — i.e. 4% <em>slower</em></strong></li>
</ul>

<p>The CSV number is the instructive one, because 2.8× looks like pruning and
isn’t. CSV still reads and parses every byte of every row; all it saves is
converting the columns you didn’t ask for. The I/O and the parse are unavoidable
because the bytes of <code class="language-plaintext highlighter-rouge">country</code> are physically interleaved with the bytes of
everything else.</p>

<p>And Avro’s result is the cleanest measurement in the whole exercise, because it
compares the same library reading the same file two ways: asking for two fields
out of eleven was <em>marginally slower</em> than taking all eleven, since the whole
record is decoded either way and then you throw most of it away. That is not a
defect. It is what row-based means, and it is precisely why Avro belongs on the
write path rather than the scan path.</p>

<h2 id="through-a-query-engine">Through a query engine</h2>

<p>Sizes and raw reads are one thing; what an engine does with them is another.
Same query, same data, DuckDB reading each file directly:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">country</span><span class="p">,</span> <span class="k">sum</span><span class="p">(</span><span class="n">quantity</span> <span class="o">*</span> <span class="n">unit_price</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">discount_pct</span><span class="p">))</span>
<span class="k">FROM</span>   <span class="o">&lt;</span><span class="n">file</span><span class="o">&gt;</span>
<span class="k">WHERE</span>  <span class="n">status</span> <span class="o">=</span> <span class="s1">'delivered'</span>
<span class="k">GROUP</span>  <span class="k">BY</span> <span class="n">country</span><span class="p">;</span>
</code></pre></div></div>

<table>
  <thead>
    <tr>
      <th>Source</th>
      <th>Time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>CSV</td>
      <td>0.879 s</td>
    </tr>
    <tr>
      <td>Parquet (zstd)</td>
      <td><strong>0.064 s</strong></td>
    </tr>
  </tbody>
</table>

<p><strong>13.8× faster.</strong> The query touches 4 of 11 columns and filters on one of them,
which is the shape almost every analytical query has, and it is the shape
columnar storage was built for.</p>

<h2 id="avro-on-its-own-terms">Avro, on its own terms</h2>

<p>Avro was measured on a 300,000-row subset, because <code class="language-plaintext highlighter-rouge">fastavro</code> is pure Python
while Arrow’s readers are compiled C++ — mixing them at full scale measures the
library, not the format. For the same reason I am <strong>not</strong> publishing a
Parquet-vs-Avro read-time comparison. It would be a language benchmark wearing a
format benchmark’s clothes, and you should be suspicious of anyone who does
publish one without saying so.</p>

<p>Sizes are a fair comparison, though, since bytes are bytes:</p>

<table>
  <thead>
    <tr>
      <th>Format (300k rows)</th>
      <th>Size</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Avro, uncompressed</td>
      <td>24.2 MB</td>
    </tr>
    <tr>
      <td>Avro, snappy</td>
      <td>13.7 MB</td>
    </tr>
    <tr>
      <td>Avro, deflate</td>
      <td>10.6 MB</td>
    </tr>
    <tr>
      <td>ORC, zlib</td>
      <td>6.8 MB</td>
    </tr>
    <tr>
      <td><strong>Parquet, zstd</strong></td>
      <td><strong>6.5 MB</strong></td>
    </tr>
  </tbody>
</table>

<p>Even at its best compression, Avro was <strong>63% larger than Parquet</strong> on identical
data. Again: not a defect. Row-based layouts interleave values of different
types and cardinalities, which is exactly the arrangement that defeats the
run-length, dictionary, and delta encodings columnar formats depend on. You
accept that cost to get cheap appends and schema evolution, and on a
<a href="/essays/batch-vs-streaming/">streaming path</a> that is a good trade.</p>

<h2 id="zstd-versus-gzip-not-close">Zstd versus gzip: not close</h2>

<table>
  <thead>
    <tr>
      <th>Codec</th>
      <th>Size (1M rows)</th>
      <th>Write time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>zstd</td>
      <td>19.3 MB</td>
      <td>0.54 s</td>
    </tr>
    <tr>
      <td>gzip</td>
      <td>17.8 MB</td>
      <td>16.00 s</td>
    </tr>
  </tbody>
</table>

<p>Gzip bought an 8% smaller file for a <strong>30× slower write</strong>. For analytical tables
— written once, read many times — that is a bad trade, and for tables written
continuously it is an indefensible one. Zstd is the sensible default and has
been for a while; a lot of pipelines are still on gzip because nobody revisited
a setting chosen in 2016.</p>

<h2 id="what-this-benchmark-does-not-prove">What this benchmark does not prove</h2>

<p>Stating this properly matters more than the numbers, because a benchmark whose
limits aren’t declared is marketing.</p>

<ul>
  <li><strong>One dataset shape.</strong> Eleven columns mixing low-cardinality strings, sorted
timestamps, high-cardinality integers, floats, and one free-text column.
Change the mix and the compression figures move. A test built only from
low-cardinality columns would flatter every columnar format enormously.</li>
  <li><strong>One ORC writer.</strong> pyarrow’s, at defaults — not Hive’s, and not tuned. This
is a real-world result for a common stack, not a statement about ORC’s
ceiling.</li>
  <li><strong>No tuning at all.</strong> No sorting or clustering, no bloom filters, no adjusted
row group or stripe sizes. Every format here can be made to do better, and
sorting on a filtered column would move the numbers most.</li>
  <li><strong>Warm page cache</strong>, which <em>understates</em> the advantage of the smaller files.
Cold reads would widen the gaps.</li>
  <li><strong>A slow two-core laptop CPU.</strong> The absolute seconds mean nothing to you. The
ratios are the deliverable.</li>
</ul>

<h2 id="what-id-take-from-it">What I’d take from it</h2>

<p>The decision rule from <a href="/essays/parquet-vs-orc-vs-avro/">the earlier essay</a>
survives, with one correction and one addition.</p>

<p><strong>Unchanged:</strong> choose the axis first. Read-heavy analytics wants columnar;
write-heavy streaming and cross-system exchange want Avro. That split is
structural, and the Avro pruning result is about as direct a confirmation as a
measurement can give.</p>

<p><em>(The Parquet-vs-ORC half of this now has its own field note:
<a href="/essays/parquet-vs-orc/">Parquet vs ORC</a>.)</em></p>

<p><strong>Corrected:</strong> I previously wrote that ORC “often achieves slightly higher
compression.” I no longer think that should be stated as a general fact. On this
data, at matched codecs, it was 23–26% larger. ORC remains the right call where
a Hive-centric stack expects it; the compression argument for choosing it should
be measured, not inherited.</p>

<p><strong>Added:</strong> if you are on Parquet with gzip or ORC with zlib because that was the
default, changing that one setting to zstd is likely the cheapest performance
work available to you. It cost 30× less write time for a file within 8% of
gzip’s size, and nothing downstream has to change.</p>

<p>And the broader point, which is why I ran this at all: most of what circulates
about file formats is inherited rather than measured, this site included until
today. The script is thirty lines of real work and
<a href="/benchmarks/file-format-benchmark.py">it’s published</a>. Run it on your data. The
answer for your table may differ from the answer for mine, and that is the
finding either way.</p>]]></content><author><name>dataarchitect.studio</name></author><summary type="html"><![CDATA[Benchmarked on 3 million rows: Parquet was 23-26% smaller than ORC at matched codecs, contradicting the usual claim. Script published, rerun it yourself.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://dataarchitect.studio/assets/og-default.png" /><media:content medium="image" url="https://dataarchitect.studio/assets/og-default.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Merge-on-Read vs Copy-on-Write: The Lakehouse Decision With Numbers</title><link href="https://dataarchitect.studio/essays/merge-on-read-vs-copy-on-write/" rel="alternate" type="text/html" title="Merge-on-Read vs Copy-on-Write: The Lakehouse Decision With Numbers" /><published>2026-07-26T09:00:00+05:30</published><updated>2026-09-14T00:00:00+05:30</updated><id>https://dataarchitect.studio/essays/merge-on-read-vs-copy-on-write</id><content type="html" xml:base="https://dataarchitect.studio/essays/merge-on-read-vs-copy-on-write/"><![CDATA[<p>Every <a href="/essays/what-is-an-open-table-format/">open table format</a> has to answer one
awkward question: the data files are <strong>immutable</strong>, so what happens when a row
changes? There are exactly two answers, and choosing between them is the most
consequential operational decision in a lakehouse that nobody writes about.</p>

<p><strong>Copy-on-write</strong> rewrites every data file that contains a changed row. The table
stays clean; the writer pays. <strong>Merge-on-read</strong> leaves the original files
untouched and records the change separately — a delete file, or a deletion vector
marking which rows in which file are no longer live — and makes the <em>reader</em>
reconcile them at query time. Writers get cheap; queries get slower until
someone compacts.</p>

<p>That’s the whole trade. What follows is why it bites harder than it sounds, and
the arithmetic that tells you which side you’re on.</p>

<h2 id="copy-on-write-vs-merge-on-read-side-by-side">Copy-on-write vs merge-on-read, side by side</h2>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>Copy-on-write</th>
      <th>Merge-on-read</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>On update, the writer</strong></td>
      <td>Rewrites whole files containing changed rows</td>
      <td>Writes new rows + delete files / deletion vectors</td>
    </tr>
    <tr>
      <td><strong>Write cost</strong></td>
      <td>High, and disproportionate to rows changed</td>
      <td>Low, roughly proportional to rows changed</td>
    </tr>
    <tr>
      <td><strong>Read cost</strong></td>
      <td>Baseline — files are already correct</td>
      <td>Baseline + reconciling accumulated deletes</td>
    </tr>
    <tr>
      <td><strong>Read cost over time</strong></td>
      <td>Flat</td>
      <td>Degrades until compaction</td>
    </tr>
    <tr>
      <td><strong>Compaction</strong></td>
      <td>Optional tuning</td>
      <td><strong>Structural requirement</strong></td>
    </tr>
    <tr>
      <td><strong>Storage churn</strong></td>
      <td>Rewrites amplify; old files await expiry</td>
      <td>Small files accumulate instead</td>
    </tr>
    <tr>
      <td><strong>Best when</strong></td>
      <td>Updates rare, batched, recent-partition</td>
      <td>Updates frequent, small, scattered</td>
    </tr>
    <tr>
      <td><strong>Iceberg</strong></td>
      <td>Default (<code class="language-plaintext highlighter-rouge">write.*.mode = copy-on-write</code>)</td>
      <td><code class="language-plaintext highlighter-rouge">write.update.mode = merge-on-read</code></td>
    </tr>
    <tr>
      <td><strong>Delta Lake</strong></td>
      <td>Default</td>
      <td><code class="language-plaintext highlighter-rouge">delta.enableDeletionVectors = true</code></td>
    </tr>
    <tr>
      <td><strong>Fails by</strong></td>
      <td>Write amplification</td>
      <td>Read amplification + small-file sprawl</td>
    </tr>
  </tbody>
</table>

<h2 id="why-copy-on-write-hurts-more-than-it-looks">Why copy-on-write hurts more than it looks</h2>

<p>The intuition that breaks people is this: <strong>copy-on-write’s cost tracks the number
of files touched, not the number of rows changed.</strong> Change one row in a 128 MB
Parquet file and you rewrite 128 MB. Change one row each in nine hundred files and
you rewrite 112 GB — to modify a few hundred kilobytes of actual data.</p>

<p>This is why the pattern that most reliably destroys a copy-on-write table is
<a href="/essays/what-is-change-data-capture/">change data capture</a>. CDC updates are
small, continuous, and — crucially — <em>scattered</em>: a customer amends an order from
four months ago and your writer reaches into a partition it hasn’t touched since
April. Batch loads confined to yesterday’s partition are fine. Corrections
sprayed across two years of history are not.</p>

<p>Merge-on-read inverts it. The writer appends the new row versions and records
which old rows are dead. Nothing gets rewritten, so write cost scales with rows
changed rather than files touched. The bill moves to the reader, who now has to
subtract the dead rows from every file they scan.</p>

<figure style="margin:2rem auto;text-align:center;">
<svg viewBox="0 0 800 340" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="mor-cow-t mor-cow-d">
  <title id="mor-cow-t">Copy-on-write versus merge-on-read</title>
  <desc id="mor-cow-d">Two panels showing what happens when one row changes. Under copy-on-write, the entire data file containing that row is rewritten as a new file, so the writer pays and the reader gets a clean table. Under merge-on-read, the original file is left in place and a small deletion vector is written alongside it, so the writer is cheap and the reader must apply the deletion vector at query time until compaction folds it in.</desc>
  <text x="200" y="28" font-size="13" fill="#1c1a17" text-anchor="middle" font-weight="700">Copy-on-write</text>
  <text x="600" y="28" font-size="13" fill="#1c1a17" text-anchor="middle" font-weight="700">Merge-on-read</text>
  <text x="400" y="58" font-size="11" fill="#8b857a" text-anchor="middle">one row changes in a 128 MB file</text>
  <rect x="70" y="76" width="260" height="46" rx="5" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <text x="200" y="98" font-size="11" fill="#1c1a17" text-anchor="middle">part-001.parquet · 128 MB</text>
  <text x="200" y="114" font-size="10" fill="#8b857a" text-anchor="middle">marked for expiry</text>
  <line x1="200" y1="122" x2="200" y2="152" stroke="#cabfac" stroke-width="2" />
  <text x="248" y="142" font-size="10" fill="#a4391f">rewrite all</text>
  <rect x="70" y="152" width="260" height="46" rx="5" fill="#c8472b" />
  <text x="200" y="174" font-size="11" fill="#f6f3ec" text-anchor="middle">part-009.parquet · 128 MB</text>
  <text x="200" y="190" font-size="10" fill="#f6f3ec" text-anchor="middle">128 MB written for one row</text>
  <text x="200" y="232" font-size="11" fill="#56514a" text-anchor="middle">writer pays · reader sees a clean table</text>
  <rect x="470" y="76" width="260" height="46" rx="5" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <text x="600" y="98" font-size="11" fill="#1c1a17" text-anchor="middle">part-001.parquet · 128 MB</text>
  <text x="600" y="114" font-size="10" fill="#8b857a" text-anchor="middle">untouched</text>
  <line x1="600" y1="122" x2="600" y2="152" stroke="#cabfac" stroke-width="2" />
  <text x="648" y="142" font-size="10" fill="#a4391f">annotate</text>
  <rect x="470" y="152" width="120" height="46" rx="5" fill="#1c1a17" />
  <text x="530" y="174" font-size="10" fill="#f6f3ec" text-anchor="middle">deletes.puffin</text>
  <text x="530" y="190" font-size="11" fill="#8b857a" text-anchor="middle">~KB</text>
  <rect x="606" y="152" width="124" height="46" rx="5" fill="#c8472b" />
  <text x="668" y="174" font-size="10" fill="#f6f3ec" text-anchor="middle">new rows</text>
  <text x="668" y="190" font-size="11" fill="#f6f3ec" text-anchor="middle">~KB</text>
  <text x="600" y="232" font-size="11" fill="#56514a" text-anchor="middle">writer is cheap · reader subtracts at query time</text>
  <line x1="400" y1="70" x2="400" y2="250" stroke="#ddd6c8" stroke-width="1.5" stroke-dasharray="4 4" />
  <text x="400" y="286" font-size="12" fill="#8b857a" text-anchor="middle">the cost doesn't disappear — it moves, from the writer to the reader</text>
  <text x="400" y="310" font-size="12" fill="#8b857a" text-anchor="middle">and compaction is the bill coming due on a schedule you choose</text>
</svg>
<figcaption style="font-family:'IBM Plex Mono',monospace;font-size:0.75rem;color:#8b857a;margin-top:0.6rem;">Same change, two places to put the cost: rewrite now, or reconcile on every read until you compact.</figcaption>
</figure>

<h2 id="measured-what-a-small-delete-actually-costs">Measured: what a small delete actually costs</h2>

<p>The amplification above is usually asserted. Here it is measured, with
<a href="/benchmarks/">a script you can run</a>. One table, five columns, deleted at three
selectivities, with the matching rows spread across every data file rather than
conveniently clustered in one — which is the realistic case, since a GDPR erasure
or a late-arriving correction does not land tidily.</p>

<p><strong>Bytes written per byte of data actually removed</strong>, at 1,000,000 rows:</p>

<table>
  <thead>
    <tr>
      <th>Rows deleted</th>
      <th>delta-rs</th>
      <th>pyiceberg</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>0.1%</strong></td>
      <td><strong>259×</strong></td>
      <td><strong>1,073×</strong></td>
    </tr>
    <tr>
      <td><strong>1%</strong></td>
      <td><strong>24×</strong></td>
      <td><strong>105×</strong></td>
    </tr>
    <tr>
      <td><strong>10%</strong></td>
      <td>2.2×</td>
      <td>9.4×</td>
    </tr>
  </tbody>
</table>

<p>Read the top row twice. Removing one row in a thousand wrote <strong>259 times</strong> the
volume of the data removed on delta-rs, and <strong>over a thousand times</strong> on
pyiceberg. The two engines differ mainly because they chose different default
file sizes and codecs, which is the point rather than a flaw in the comparison:
<strong>amplification is a function of file size, not of the format’s logo.</strong> Bigger
files make it worse.</p>

<p>The pattern held when the table was 5× smaller, so this is not an artifact of one
run. And the trend is the whole argument for merge-on-read: the <em>less</em> you delete,
the <em>worse</em> copy-on-write looks. At 10% it is a reasonable 2–9×. At 0.1% it is
indefensible.</p>

<h3 id="the-catch-you-probably-cannot-turn-merge-on-read-on">The catch: you probably cannot turn merge-on-read on</h3>

<p>Both engines accept a merge-on-read setting. <strong>Neither honours it.</strong></p>

<p>Every row in that table is identical whether the table was created with
copy-on-write or with merge-on-read requested. No delete file, no deletion vector,
same bytes rewritten.</p>

<ul>
  <li><strong>pyiceberg 0.12.0</strong> at least says so. Set <code class="language-plaintext highlighter-rouge">write.delete.mode=merge-on-read</code>
and it emits: <em>“Merge on read is not yet supported, falling back to
copy-on-write.”</em></li>
  <li><strong>delta-rs 1.6.3</strong> says nothing at all. Set
<code class="language-plaintext highlighter-rouge">delta.enableDeletionVectors=true</code>, and the property is faithfully stored in
the table metadata — where a reader will see it and believe it. The delete then
rewrites files exactly as copy-on-write would.</li>
</ul>

<p>That second one is the trap. The setting persists, so the table <em>looks</em>
merge-on-read to anyone inspecting its properties, while behaving as
copy-on-write. If you have configured deletion vectors from Python and assumed
they were active, check.</p>

<p><strong>Merge-on-read is a JVM feature today.</strong> Spark with the Iceberg or Delta runtime
does it properly. The pure-Python clients that most pipelines reach for do not,
as of these versions. That is a materially different situation from the one most
writing on this topic implies, including the earlier version of this essay.</p>

<p>Versions matter and will date quickly: pyarrow 25.0.0, deltalake 1.6.3,
pyiceberg 0.12.0, Python 3.10.12. The script re-runs in under a minute and prints
what your versions actually do, which is the only answer that counts.</p>

<h2 id="scaling-that-up-the-arithmetic-at-2-tb">Scaling that up: the arithmetic at 2 TB</h2>

<p>The measurement above is small enough to run on a laptop. Production tables are
not, so here is the same arithmetic at realistic scale. <strong>The amplification ratio
is measured; the table size, batch rate and file count below are assumptions</strong>
chosen to be plausible — substitute your own. What holds regardless is the shape.</p>

<p>Take an <code class="language-plaintext highlighter-rouge">orders</code> table on object storage:</p>

<ul>
  <li><strong>2 TB</strong>, Parquet, partitioned by day, two years of history</li>
  <li>Target file size <strong>128 MB</strong> → roughly <strong>16,000 data files</strong></li>
  <li>CDC delivers <strong>400,000 changed rows per day</strong>, applied <strong>hourly</strong></li>
  <li>So each batch carries about <strong>16,700 rows</strong>, averaging <strong>1 KB</strong> → <strong>~17 MB</strong> of
genuinely changed data per hour</li>
  <li>Because corrections land across history, those 16,700 rows are scattered across
about <strong>900 distinct files</strong></li>
</ul>

<p>Now price each strategy per hourly batch.</p>

<p><strong>Copy-on-write.</strong> Every one of those 900 files gets rewritten in full:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>900 files × 128 MB          = 115,200 MB  ≈ 112.5 GB written
actual changed data                        ≈ 17 MB
write amplification         = 112.5 GB / 17 MB  ≈ 6,800×
</code></pre></div></div>

<p>Across 24 batches that’s roughly <strong>2.7 TB rewritten per day</strong> to apply 400 MB of
changes — on a 2 TB table. You are rewriting more than the table, daily.</p>

<p><strong>Merge-on-read.</strong> The writer appends the new row versions and a deletion vector
per touched file:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>new row data                              ≈ 17 MB
deletion vectors, 900 files × ~2 KB       ≈  1.8 MB
total written                             ≈ 19 MB
write amplification                       ≈ 1.1×
</code></pre></div></div>

<p>Three orders of magnitude less written. That is the entire reason merge-on-read
exists, and why streaming and CDC workloads reach for it.</p>

<p><strong>Then the reader’s bill arrives.</strong> Every scan must apply the accumulated
deletes. After one batch, a query touching those partitions resolves 900 deletion
vectors; after a full day without compaction, it may face <strong>21,600</strong>. Read
latency degrades roughly in step with that accumulation — a scan that was
comfortable in the morning is noticeably slower by evening, and the degradation
is gradual enough that people blame the query rather than the table.</p>

<p><strong>Compaction is what closes the loop.</strong> Rewriting the affected files <em>once
nightly</em> rather than 24 times a day:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>distinct files touched across the whole day  ≈ 4,000
nightly rewrite                              = 4,000 × 128 MB ≈ 500 GB
vs copy-on-write's 24 × 112.5 GB             ≈ 2.7 TB
</code></pre></div></div>

<p>Same correctness, roughly <strong>five times less data rewritten</strong>, and reads reset to
baseline every morning. Merge-on-read plus scheduled compaction isn’t a
compromise between the two strategies — it’s copy-on-write’s work, batched, with
the scheduling under your control instead of the CDC stream’s.</p>

<h2 id="the-decision-rule">The decision rule</h2>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Are updates rare, large, and confined to recent partitions?
   → Copy-on-write. Reads stay flat, writers rarely suffer.

Are updates frequent, small, or scattered across old partitions?
   → Merge-on-read — AND schedule compaction before you ship it.

Do you not know yet?
   → Copy-on-write (the default), and instrument write volume.
     If bytes-written per batch dwarfs rows-changed, you have your answer.
</code></pre></div></div>

<p>The threshold worth watching is the ratio in that last line: <strong>bytes rewritten
divided by bytes actually changed.</strong> Below roughly 10× copy-on-write is fine.
Somewhere past 100× it is quietly burning your compute budget, and past 1,000× it
is the reason your hourly job now takes fifty minutes.</p>

<h2 id="how-to-set-it">How to set it</h2>

<p>Iceberg exposes the choice per operation, which is more useful than it first
appears — deletes and updates often have different profiles from merges:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">ALTER</span> <span class="k">TABLE</span> <span class="n">lake</span><span class="p">.</span><span class="n">sales</span><span class="p">.</span><span class="n">orders</span> <span class="k">SET</span> <span class="n">TBLPROPERTIES</span> <span class="p">(</span>
  <span class="s1">'write.delete.mode'</span>  <span class="o">=</span> <span class="s1">'merge-on-read'</span><span class="p">,</span>
  <span class="s1">'write.update.mode'</span>  <span class="o">=</span> <span class="s1">'merge-on-read'</span><span class="p">,</span>
  <span class="s1">'write.merge.mode'</span>   <span class="o">=</span> <span class="s1">'merge-on-read'</span>
<span class="p">);</span>

<span class="c1">-- Compaction is the other half of the decision, not an afterthought:</span>
<span class="k">CALL</span> <span class="n">lake</span><span class="p">.</span><span class="k">system</span><span class="p">.</span><span class="n">rewrite_data_files</span><span class="p">(</span>
  <span class="k">table</span> <span class="o">=&gt;</span> <span class="s1">'sales.orders'</span><span class="p">,</span>
  <span class="k">options</span> <span class="o">=&gt;</span> <span class="k">map</span><span class="p">(</span><span class="s1">'target-file-size-bytes'</span><span class="p">,</span> <span class="s1">'134217728'</span><span class="p">)</span>
<span class="p">);</span>
<span class="k">CALL</span> <span class="n">lake</span><span class="p">.</span><span class="k">system</span><span class="p">.</span><span class="n">rewrite_position_delete_files</span><span class="p">(</span><span class="k">table</span> <span class="o">=&gt;</span> <span class="s1">'sales.orders'</span><span class="p">);</span>
</code></pre></div></div>

<p>Delta reaches the same place through deletion vectors:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">ALTER</span> <span class="k">TABLE</span> <span class="n">lake</span><span class="p">.</span><span class="n">sales</span><span class="p">.</span><span class="n">orders</span>
  <span class="k">SET</span> <span class="n">TBLPROPERTIES</span> <span class="p">(</span><span class="s1">'delta.enableDeletionVectors'</span> <span class="o">=</span> <span class="k">true</span><span class="p">);</span>

<span class="c1">-- Fold the soft deletes back into the data files:</span>
<span class="n">REORG</span> <span class="k">TABLE</span> <span class="n">lake</span><span class="p">.</span><span class="n">sales</span><span class="p">.</span><span class="n">orders</span> <span class="n">APPLY</span> <span class="p">(</span><span class="n">PURGE</span><span class="p">);</span>
</code></pre></div></div>

<p>The mechanisms differ in detail — Iceberg’s modes are documented in the
<a href="https://iceberg.apache.org/spec/">table specification</a>, Delta’s behaviour and
the version in which each operation gained support in the
<a href="https://docs.delta.io/latest/delta-deletion-vectors.html">deletion vectors docs</a>
— but the trade-off underneath is identical, which is a good sign it’s a property
of the problem rather than of either implementation.</p>

<h2 id="the-part-that-actually-goes-wrong">The part that actually goes wrong</h2>

<p>Almost nobody chooses merge-on-read badly. They choose it <em>correctly</em> and then
don’t schedule compaction, because in the first week the table is fast and the
setting looks free. The degradation is gradual, it lands on readers rather than
on the pipeline that caused it, and by the time anyone investigates, the small
files have multiplied and the metadata planning cost has grown alongside the scan
cost.</p>

<p>So treat it as one decision with two halves. Turning on merge-on-read without a
compaction schedule isn’t a tuning choice; it’s a deferred cost with no due date,
which is the same category of mistake as
<a href="/essays/how-to-make-a-data-pipeline-idempotent/">a pipeline that isn’t idempotent</a>
— it works until precisely the moment it matters.</p>

<p>Pick the side your update pattern puts you on. Then pay the bill on a schedule
you chose, rather than one your CDC stream chose for you.</p>]]></content><author><name>dataarchitect.studio</name></author><summary type="html"><![CDATA[Copy-on-write rewrites whole files on update; merge-on-read annotates them and resolves at read time. The trade is write cost against read cost, with arithmetic.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://dataarchitect.studio/assets/og-default.png" /><media:content medium="image" url="https://dataarchitect.studio/assets/og-default.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Knowledge Graph vs Data Warehouse: Aggregation vs Connection</title><link href="https://dataarchitect.studio/essays/knowledge-graph-vs-data-warehouse/" rel="alternate" type="text/html" title="Knowledge Graph vs Data Warehouse: Aggregation vs Connection" /><published>2026-07-24T11:00:00+05:30</published><updated>2026-08-29T00:00:00+05:30</updated><id>https://dataarchitect.studio/essays/knowledge-graph-vs-data-warehouse</id><content type="html" xml:base="https://dataarchitect.studio/essays/knowledge-graph-vs-data-warehouse/"><![CDATA[<p>The comparison is usually framed as a contest, and that framing is why teams get
it wrong. <strong>A data warehouse is built to aggregate: sum, group, and filter across
enormous numbers of rows. A knowledge graph is built to traverse: follow the
relationships between entities, several hops out, without knowing the path in
advance.</strong> Those are different query <em>shapes</em>, not competing products — and the
right question isn’t which to buy, it’s which shape your hard questions actually
have. Most mature estates end up with both: the
<a href="/glossary/data-warehouse/">warehouse</a> as the system of record for metrics, the
graph as the layer that understands how things connect.</p>

<h2 id="data-warehouse-vs-knowledge-graph-side-by-side">Data warehouse vs knowledge graph, side by side</h2>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>Data warehouse</th>
      <th>Knowledge graph</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Optimized for</strong></td>
      <td>Aggregation over many rows</td>
      <td>Traversal across relationships</td>
    </tr>
    <tr>
      <td><strong>Answers</strong></td>
      <td>How much, by what, over what period</td>
      <td>What connects to what, and how far</td>
    </tr>
    <tr>
      <td><strong>Shape</strong></td>
      <td>Tables, <a href="/essays/fact-table-vs-dimension-table/">facts and dimensions</a></td>
      <td>Nodes and edges, both first-class</td>
    </tr>
    <tr>
      <td><strong>Schema</strong></td>
      <td>Declared up front, enforced</td>
      <td>Flexible, extended as you learn</td>
    </tr>
    <tr>
      <td><strong>Query language</strong></td>
      <td>SQL</td>
      <td>Cypher, SPARQL, GQL</td>
    </tr>
    <tr>
      <td><strong>Multi-hop questions</strong></td>
      <td>Painful — self-joins pile up</td>
      <td>Native — that’s the design</td>
    </tr>
    <tr>
      <td><strong>Aggregate questions</strong></td>
      <td>Native and fast</td>
      <td>Possible, rarely as fast</td>
    </tr>
    <tr>
      <td><strong>Meaning lives in</strong></td>
      <td>Models + <a href="/essays/what-is-a-semantic-layer/">semantic layer</a></td>
      <td>The graph itself, plus its ontology</td>
    </tr>
    <tr>
      <td><strong>Typical role</strong></td>
      <td>System of record for reporting</td>
      <td>Reasoning and context layer</td>
    </tr>
  </tbody>
</table>

<h2 id="the-query-shape-is-the-whole-argument">The query shape is the whole argument</h2>

<p>The clearest way to feel the difference is to write the same business question
both ways. <em>“Which suppliers are we exposed to through our suppliers’ suppliers?”</em>
In SQL, unknown depth means either a recursive CTE or a stack of self-joins that
grows with every hop:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">-- Warehouse: multi-hop needs recursion, and gets awkward fast</span>
<span class="k">WITH</span> <span class="k">RECURSIVE</span> <span class="k">chain</span> <span class="k">AS</span> <span class="p">(</span>
  <span class="k">SELECT</span> <span class="n">supplier_id</span><span class="p">,</span> <span class="n">depends_on_id</span><span class="p">,</span> <span class="mi">1</span> <span class="k">AS</span> <span class="n">hop</span>
  <span class="k">FROM</span>   <span class="n">fact_supplier_dependency</span>
  <span class="k">WHERE</span>  <span class="n">supplier_id</span> <span class="o">=</span> <span class="p">:</span><span class="n">root</span>
  <span class="k">UNION</span> <span class="k">ALL</span>
  <span class="k">SELECT</span> <span class="k">c</span><span class="p">.</span><span class="n">supplier_id</span><span class="p">,</span> <span class="n">d</span><span class="p">.</span><span class="n">depends_on_id</span><span class="p">,</span> <span class="k">c</span><span class="p">.</span><span class="n">hop</span> <span class="o">+</span> <span class="mi">1</span>
  <span class="k">FROM</span>   <span class="k">chain</span> <span class="k">c</span>
  <span class="k">JOIN</span>   <span class="n">fact_supplier_dependency</span> <span class="n">d</span> <span class="k">ON</span> <span class="n">d</span><span class="p">.</span><span class="n">supplier_id</span> <span class="o">=</span> <span class="k">c</span><span class="p">.</span><span class="n">depends_on_id</span>
  <span class="k">WHERE</span>  <span class="k">c</span><span class="p">.</span><span class="n">hop</span> <span class="o">&lt;</span> <span class="mi">5</span>                       <span class="c1">-- depth you must guess in advance</span>
<span class="p">)</span>
<span class="k">SELECT</span> <span class="k">DISTINCT</span> <span class="n">depends_on_id</span> <span class="k">FROM</span> <span class="k">chain</span><span class="p">;</span>
</code></pre></div></div>

<div class="language-cypher highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Graph: depth is a parameter of the traversal, not a rewrite of the query</span>
<span class="k">MATCH</span><span class="w"> </span><span class="ss">(</span><span class="py">s:</span><span class="n">Supplier</span> <span class="ss">{</span><span class="py">id:</span> <span class="n">$root</span><span class="ss">})</span><span class="o">-</span><span class="ss">[</span><span class="nc">:DEPENDS_ON</span><span class="o">*</span><span class="m">1</span><span class="o">..</span><span class="m">5</span><span class="ss">]</span><span class="o">-&gt;</span><span class="ss">(</span><span class="py">exposed:</span><span class="n">Supplier</span><span class="ss">)</span>
<span class="k">RETURN</span> <span class="k">DISTINCT</span> <span class="n">exposed</span><span class="ss">;</span>
</code></pre></div></div>

<p>Now reverse it. <em>“What was revenue by region by quarter?”</em> is one clean
<code class="language-plaintext highlighter-rouge">GROUP BY</code> in the warehouse and an awkward, slower aggregation in most graphs.
Neither engine is better; each is built for a different question, and choosing
by hype rather than by query shape is how teams end up with an expensive graph
serving dashboards, or a warehouse full of self-joins nobody can maintain.</p>

<figure style="margin:2rem auto;text-align:center;">
<svg viewBox="0 0 800 330" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="knowledge-graph-vs-data-warehouse-t knowledge-graph-vs-data-warehouse-d">
  <title id="knowledge-graph-vs-data-warehouse-t">Aggregation versus traversal</title>
  <desc id="knowledge-graph-vs-data-warehouse-d">On the left, a warehouse: stacked rows collapsing into a single aggregated number via SUM and GROUP BY. On the right, a knowledge graph: a starting node with edges followed outward through intermediate nodes to entities three hops away. Choose by the shape of your hardest question.</desc>
  <text x="200" y="30" font-size="13" fill="#1c1a17" text-anchor="middle" font-weight="700">Warehouse — aggregation</text>
  <rect x="70" y="46" width="260" height="22" rx="3" fill="#c8472b" />
  <rect x="70" y="72" width="260" height="14" rx="3" fill="#f6f3ec" stroke="#ddd6c8" stroke-width="1.2" />
  <rect x="70" y="90" width="260" height="14" rx="3" fill="#f6f3ec" stroke="#ddd6c8" stroke-width="1.2" />
  <rect x="70" y="108" width="260" height="14" rx="3" fill="#f6f3ec" stroke="#ddd6c8" stroke-width="1.2" />
  <rect x="70" y="126" width="260" height="14" rx="3" fill="#f6f3ec" stroke="#ddd6c8" stroke-width="1.2" />
  <rect x="70" y="144" width="260" height="14" rx="3" fill="#f6f3ec" stroke="#ddd6c8" stroke-width="1.2" />
  <text x="200" y="186" font-size="11" fill="#56514a" text-anchor="middle">millions of rows ↓ one number</text>
  <text x="200" y="206" font-size="11" fill="#8b857a" text-anchor="middle">SUM(amount) GROUP BY region, quarter</text>
  <text x="600" y="30" font-size="13" fill="#1c1a17" text-anchor="middle" font-weight="700">Knowledge graph — traversal</text>
  <circle cx="600" cy="70" r="16" fill="#c8472b" />
  <text x="600" y="74" font-size="11" fill="#f6f3ec" text-anchor="middle">you</text>
  <circle cx="520" cy="120" r="14" fill="#1c1a17" />
  <circle cx="680" cy="120" r="14" fill="#1c1a17" />
  <circle cx="470" cy="176" r="12" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <circle cx="570" cy="176" r="12" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <circle cx="660" cy="176" r="12" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <circle cx="730" cy="176" r="12" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <line x1="600" y1="86" x2="520" y2="106" stroke="#cabfac" stroke-width="2" />
  <line x1="600" y1="86" x2="680" y2="106" stroke="#cabfac" stroke-width="2" />
  <line x1="520" y1="134" x2="470" y2="164" stroke="#ddd6c8" stroke-width="1.5" />
  <line x1="520" y1="134" x2="570" y2="164" stroke="#ddd6c8" stroke-width="1.5" />
  <line x1="680" y1="134" x2="660" y2="164" stroke="#ddd6c8" stroke-width="1.5" />
  <line x1="680" y1="134" x2="730" y2="164" stroke="#ddd6c8" stroke-width="1.5" />
  <text x="600" y="206" font-size="11" fill="#8b857a" text-anchor="middle">follow edges → exposure three hops out</text>
  <text x="400" y="262" font-size="12" fill="#8b857a" text-anchor="middle">choose by the shape of your hardest question, not by which technology is fashionable</text>
  <text x="400" y="288" font-size="12" fill="#8b857a" text-anchor="middle">"how much, by what" → warehouse ·  "what connects to what" → graph</text>
</svg>
<figcaption style="font-family:'IBM Plex Mono',monospace;font-size:0.75rem;color:#8b857a;margin-top:0.6rem;">Two engines, two query shapes: collapsing rows into a number, or walking edges into context.</figcaption>
</figure>

<h2 id="where-the-graph-genuinely-wins">Where the graph genuinely wins</h2>

<p>Four situations, and they share a signature — <em>the relationships are the
product</em>, not context for a measure:</p>

<ul>
  <li><strong>Fraud and risk rings</strong> — the suspicious thing isn’t any account, it’s the
shape of the connections between accounts.</li>
  <li><strong>Entity resolution across systems</strong> — the same customer under four
identifiers, discovered by how records relate rather than by matching strings.</li>
  <li><strong>Dependency and impact analysis</strong> — supply chains, and the
<a href="/essays/what-is-data-lineage/">data lineage</a> graph you already rely on, which
is a knowledge graph whether or not anyone calls it one.</li>
  <li><strong>Domains where connection <em>is</em> the science</strong> — drug interactions, genomics,
org and identity hierarchies.</li>
</ul>

<p>If your hardest questions are none of those, the honest answer is that your
warehouse isn’t the problem and a graph won’t fix it.</p>

<h2 id="the-ai-angle-without-the-hype">The AI angle, without the hype</h2>

<p>Graphs re-entered the conversation through retrieval. Vector search finds
passages that <em>look</em> semantically similar, which works well for
“find documents about the refund policy” and degrades badly on questions that
span several connected entities — the retriever has no way to know that these
five facts belong to one chain of reasoning.
<a href="https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/">GraphRAG</a>
approaches supply that structure explicitly, building an entity graph from the
corpus first and traversing it at query time — which is why they hold up on
multi-entity and whole-corpus questions where pure vector retrieval falls apart.</p>

<p>But notice what’s actually doing the work: not the graph database — the
<em>explicit relationships</em>. That’s the same asset an
<a href="/essays/ontology-vs-data-model/">ontology</a> provides, the same asset
<a href="/essays/conformed-dimensions/">conformed dimensions</a> provide inside a
warehouse, and the same reason a governed
<a href="/essays/what-is-a-semantic-layer/">semantic layer</a> improves text-to-SQL
accuracy. Machines reason better when meaning and structure are declared instead
of inferred. Graphs are one good way to declare them; they are not the only one,
and buying one won’t substitute for the modelling discipline underneath.</p>

<h2 id="the-pragmatic-architecture">The pragmatic architecture</h2>

<p>For nearly every organisation the answer is <strong>both, with clear roles</strong>: the
warehouse or <a href="/glossary/data-lakehouse/">lakehouse</a> remains the governed system
of record where metrics are defined and history is kept; the graph is built
<em>from</em> it as a derived layer serving relationship questions and AI grounding.
That ordering matters. A graph fed by an ungoverned warehouse inherits every
definitional argument you never settled — you end up with beautifully connected
wrong answers, which are harder to detect than plainly wrong ones, because the
traversal looks so convincing.</p>

<p>Get the meaning right first. Then choose the engine that matches the question.</p>]]></content><author><name>dataarchitect.studio</name></author><summary type="html"><![CDATA[A warehouse answers how much; a knowledge graph answers how things connect. They're different query shapes, not rivals — and most serious estates end up with both.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://dataarchitect.studio/assets/og-default.png" /><media:content medium="image" url="https://dataarchitect.studio/assets/og-default.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Ontology vs Data Model: Meaning vs Structure</title><link href="https://dataarchitect.studio/essays/ontology-vs-data-model/" rel="alternate" type="text/html" title="Ontology vs Data Model: Meaning vs Structure" /><published>2026-07-24T09:00:00+05:30</published><updated>2026-08-29T00:00:00+05:30</updated><id>https://dataarchitect.studio/essays/ontology-vs-data-model</id><content type="html" xml:base="https://dataarchitect.studio/essays/ontology-vs-data-model/"><![CDATA[<p>Here is the distinction, before the philosophy: <strong>a data model defines how data
is structured so a particular system can store and query it. An ontology defines
what concepts <em>mean</em> and how they relate, independently of any system that
stores them.</strong> The data model is committed to an implementation — these tables,
these keys, this warehouse. The ontology is deliberately not: it’s meant to be
reused by many applications, and to be formal enough that a machine can reason
over it rather than merely read it.</p>

<p>That distinction has been an academic footnote for twenty years. It stopped being
one the moment we started pointing language models at enterprise schemas and
asking them to interpret meaning they were never given.</p>

<h2 id="the-comparison-honestly">The comparison, honestly</h2>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>Data model</th>
      <th>Ontology</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Answers</strong></td>
      <td>How is this stored and queried?</td>
      <td>What does this mean, and how does it relate?</td>
    </tr>
    <tr>
      <td><strong>Scope</strong></td>
      <td>One system or application</td>
      <td>A domain, across systems</td>
    </tr>
    <tr>
      <td><strong>Optimized for</strong></td>
      <td>Storage, integrity, query performance</td>
      <td>Shared meaning, reuse, inference</td>
    </tr>
    <tr>
      <td><strong>Typical artifacts</strong></td>
      <td>Tables, columns, keys, constraints</td>
      <td>Classes, properties, relationships, axioms</td>
    </tr>
    <tr>
      <td><strong>Expressed in</strong></td>
      <td>DDL, ERDs, dbt models</td>
      <td>OWL/RDFS, SHACL, or a formal spec</td>
    </tr>
    <tr>
      <td><strong>Change driver</strong></td>
      <td>New query patterns, new sources</td>
      <td>New understanding of the domain</td>
    </tr>
    <tr>
      <td><strong>Reused across apps</strong></td>
      <td>Rarely — it’s built for one</td>
      <td>By design — that’s the point</td>
    </tr>
    <tr>
      <td><strong>Machine can infer from it</strong></td>
      <td>No (it can only execute queries)</td>
      <td>Yes (that’s what the axioms are for)</td>
    </tr>
    <tr>
      <td><strong>Validates data by</strong></td>
      <td>Constraints, in the database</td>
      <td>A separate shape language (SHACL)</td>
    </tr>
    <tr>
      <td><strong>Fails by</strong></td>
      <td>Being wrong for new questions</td>
      <td>Being too abstract to implement</td>
    </tr>
  </tbody>
</table>

<h2 id="the-same-problem-two-altitudes">The same problem, two altitudes</h2>

<p>Take the case every data architect has lived: three systems disagree about what
a customer is. Billing has accounts, the CRM has leads that sometimes become
accounts, the product database has users. The <em>data model</em> solution is
structural — reconcile them into one dimension with a
<a href="/glossary/surrogate-key/">surrogate key</a>, keep each source’s
<a href="/glossary/natural-key/">natural key</a> as an attribute, and you have something
queryable:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">-- Data model: a structure optimized for this warehouse's queries</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">dim_customer</span> <span class="p">(</span>
  <span class="n">customer_key</span>    <span class="nb">BIGINT</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span>   <span class="c1">-- surrogate, warehouse-owned</span>
  <span class="n">billing_acct_id</span> <span class="nb">TEXT</span><span class="p">,</span>                 <span class="c1">-- natural keys retained</span>
  <span class="n">crm_lead_id</span>     <span class="nb">TEXT</span><span class="p">,</span>
  <span class="n">product_user_id</span> <span class="nb">TEXT</span><span class="p">,</span>
  <span class="n">segment</span>         <span class="nb">TEXT</span><span class="p">,</span>
  <span class="n">valid_from</span>      <span class="nb">DATE</span><span class="p">,</span>  <span class="n">valid_to</span>  <span class="nb">DATE</span> <span class="c1">-- SCD Type 2 history</span>
<span class="p">);</span>
</code></pre></div></div>

<p>The ontology solution is semantic — it doesn’t tell you where to put the rows,
it tells you what the concepts <em>are</em> and how they relate, so any system can
implement it consistently:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Ontology (informally rendered, OWL-flavoured):

Class:    Party
Class:    CompletedOrder ⊑ Order
Property: placedOrder  Party → Order
Property: hasAccount   Party → Account

-- Defined classes: membership is DERIVED, not asserted
Customer ≡ Party ⊓ ∃ placedOrder.CompletedOrder
Prospect ≡ Party ⊓ ¬∃ placedOrder.CompletedOrder
-- note: disjointness now follows from the definitions.
-- It doesn't need declaring, and declaring it separately is how
-- people accidentally make their ontology inconsistent.
</code></pre></div></div>

<p>Those two <code class="language-plaintext highlighter-rouge">≡</code> lines are the tell. They aren’t columns, and they aren’t
constraints on a table — they’re <em>definitions</em>, and a reasoner given a Party with
one completed order will classify it as a Customer on its own. Nobody writes that
logic again in the next system. A data model can enforce a rule inside one
database; an ontology states what a thing <em>is</em>, once, for everywhere.</p>

<p>The subtlety that trips up most first ontologies is worth stating plainly,
because it’s the difference between a model that works and one that quietly
does nothing. <strong>OWL derives; it does not validate.</strong> It is <em>monotonic</em> — new
facts can only add conclusions, never retract them — so if you assert
<code class="language-plaintext highlighter-rouge">Customer disjointWith Prospect</code> and then hand the reasoner a Prospect who has
placed an order, you don’t get a helpful reclassification. You get a logical
inconsistency, and the reasoner stops being useful. And because OWL is
<em>open-world</em>, an axiom like “every Customer has at least one Account” does not
flag a Customer whose Account is missing from your data; it cheerfully infers
that one exists somewhere. If what you wanted was a check that fails, that’s a
different tool:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Shape (SHACL — validation, closed over your data):

CustomerShape
  targetClass:  Customer
  property hasAccount  { minCount: 1 }   -- FAILS if the data omits it
  property hasContact  { minCount: 1; class: Person }
</code></pre></div></div>

<p>Same vocabulary, opposite jobs: OWL says what follows, SHACL says what must be
present. Reach for the first when you want a machine to conclude something, the
second when you want it to complain — and the
<a href="https://www.w3.org/TR/owl2-primer/">OWL 2 primer</a> and
<a href="https://www.w3.org/TR/shacl/">SHACL specification</a> are the authoritative
references when a vendor’s “semantic layer” blurs the two.</p>

<figure style="margin:2rem auto;text-align:center;">
<svg viewBox="0 0 800 350" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto;font-family:'IBM Plex Mono',ui-monospace,monospace;" role="img" aria-labelledby="ontology-vs-data-model-t ontology-vs-data-model-d">
  <title id="ontology-vs-data-model-t">One ontology above many data models</title>
  <desc id="ontology-vs-data-model-d">A single ontology at the top defines Customer as a kind of Party, disjoint from Prospect, with a hasAccount property. Below it sit three separate implementations of that same meaning: a warehouse star schema with dim_customer, a CRM object model with Lead and Account, and an application's third-normal-form users table.</desc>
  <rect x="230" y="16" width="340" height="64" rx="6" fill="#c8472b" />
  <text x="400" y="42" font-size="14" fill="#f6f3ec" text-anchor="middle" font-weight="700">Ontology — meaning</text>
  <text x="400" y="63" font-size="11" fill="#f6f3ec" text-anchor="middle">Customer ⊑ Party · hasAccount · disjointWith Prospect</text>
  <text x="400" y="102" font-size="11" fill="#8b857a" text-anchor="middle">one definition, application-independent, machine-reasonable</text>
  <line x1="300" y1="80" x2="150" y2="150" stroke="#cabfac" stroke-width="2" />
  <line x1="400" y1="80" x2="400" y2="150" stroke="#cabfac" stroke-width="2" />
  <line x1="500" y1="80" x2="650" y2="150" stroke="#cabfac" stroke-width="2" />
  <rect x="50" y="150" width="200" height="72" rx="6" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <text x="150" y="177" font-size="12" fill="#1c1a17" text-anchor="middle" font-weight="700">warehouse model</text>
  <text x="150" y="198" font-size="10" fill="#56514a" text-anchor="middle">dim_customer, star schema</text>
  <rect x="300" y="150" width="200" height="72" rx="6" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <text x="400" y="177" font-size="12" fill="#1c1a17" text-anchor="middle" font-weight="700">CRM model</text>
  <text x="400" y="198" font-size="10" fill="#56514a" text-anchor="middle">Lead, Account objects</text>
  <rect x="550" y="150" width="200" height="72" rx="6" fill="#f6f3ec" stroke="#1c1a17" stroke-width="1.5" />
  <text x="650" y="177" font-size="12" fill="#1c1a17" text-anchor="middle" font-weight="700">app model</text>
  <text x="650" y="198" font-size="10" fill="#56514a" text-anchor="middle">users table, 3NF</text>
  <text x="400" y="256" font-size="12" fill="#1c1a17" text-anchor="middle">Data models — structure: three implementations of one meaning</text>
  <text x="400" y="296" font-size="12" fill="#8b857a" text-anchor="middle">the ontology is what they should all agree on; the models are how each system honours it</text>
  <text x="400" y="318" font-size="12" fill="#8b857a" text-anchor="middle">get the meaning wrong once and all three inherit the mistake</text>
</svg>
<figcaption style="font-family:'IBM Plex Mono',monospace;font-size:0.75rem;color:#8b857a;margin-top:0.6rem;">One ontology, many data models: meaning above, structure below.</figcaption>
</figure>

<h2 id="what-your-semantic-layer-already-is">What your semantic layer already is</h2>

<p>Here’s the part most ontology writing misses, and most data teams need to hear:
if you have a well-built <a href="/essays/what-is-a-semantic-layer/">semantic layer</a>,
you already own a lightweight ontology. Defining <em>revenue</em> once, in one governed
place, so every dashboard and notebook returns the same number, is an act of
ontology in everything but formalism —
<a href="/essays/conformed-dimensions/">conformed dimensions</a> are the same instinct
applied to dimensions, and a <a href="/essays/data-contracts-are-a-cultural-problem/">data contract</a>
is that agreement made enforceable.</p>

<p>What a true ontology adds beyond a semantic layer is threefold: <strong>formal
semantics</strong> (axioms a reasoner can act on), <strong>relationship modelling as a
first-class citizen</strong> (not just measures sliced by dimensions), and
<strong>application independence</strong> (meaning that outlives the warehouse it was first
implemented in). Each of those has a cost, which is why the honest answer for
most teams is <em>strengthen the semantic layer first, and reach for formal
ontology only when one of those three is genuinely the binding constraint.</em></p>

<h2 id="why-this-stopped-being-academic">Why this stopped being academic</h2>

<p>Ontologies had a long winter for a good reason: they were expensive to build,
hard to keep in sync with reality, and mostly consumed by other ontologists. What
changed is the reader. When the consumer of your schema was an analyst, tribal
knowledge filled the gaps — everyone knew that <code class="language-plaintext highlighter-rouge">cust_flg_2</code> meant enterprise
tier. When the consumer is a language model generating SQL, there is no tribal
knowledge; there is only what the model can read. This is measurable, not
rhetorical: the <a href="https://bird-bench.github.io/">BIRD text-to-SQL benchmark</a> was
built specifically to test queries over messy real-world schemas, and its
headline finding is that supplying external knowledge — the human explanation of
what a column actually means — moves execution accuracy substantially, while
accuracy still degrades sharply on questions spanning several entities and
relationships. That is exactly the territory where a data model is silent and an
ontology speaks.</p>

<p>That’s the real thesis, and it’s older than the technology: <strong>the discipline of
declaring meaning explicitly, before anyone queries anything, is the thing that
makes data trustworthy.</strong> Kimball called it declaring the
<a href="/essays/fact-table-grain/">grain</a> and conforming dimensions. The semantic layer
calls it defining a metric once. Ontologists call it formalising a domain. Same
instinct, three vocabularies, and the machines just made it non-optional.</p>

<h2 id="when-to-reach-for-which">When to reach for which</h2>

<p><strong>Build a data model</strong> — always. It’s not optional; something has to store the
rows, and structural discipline (<a href="/essays/fact-table-grain/">declared grain</a>,
<a href="/patterns/star-schema/">star schemas</a>, keys) is what keeps analytics honest.</p>

<p><strong>Build a semantic layer</strong> when more than one tool queries the same data and you
need one definition to win.</p>

<p><strong>Build a formal ontology</strong> when meaning must be shared across many systems or
organisations, when the <em>relationships</em> between entities are the analytical
product (fraud rings, supply chains, drug interactions), when regulation
requires machine-checkable definitions, or when you’re grounding AI systems that
must reason over connections rather than scan columns.</p>

<p>And beware the failure mode that gave ontologies their reputation: a beautiful,
formally correct model of a domain that no system implements and no query
touches. A data model can be wrong for tomorrow’s questions; an ontology can be
too abstract to ever be right about today’s. The strongest architectures use
both, at their proper altitudes — meaning declared once above, structure
optimised deliberately below.</p>]]></content><author><name>dataarchitect.studio</name></author><summary type="html"><![CDATA[A data model shapes data for one system; an ontology defines what concepts mean, independent of any system — and machines now read your schema.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://dataarchitect.studio/assets/og-default.png" /><media:content medium="image" url="https://dataarchitect.studio/assets/og-default.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>