What does an Iceberg catalog actually do?

It holds the authoritative pointer to each table's current metadata file and swaps that pointer atomically on every commit — which is what makes transactions work. On top of that mechanical job, catalogs layer the governance: who may read, write, or commit to which table, from which engine, with what audit trail.

Is an Iceberg catalog the same as a data catalog like Atlan or Collibra?

No — unfortunate naming collision. An Iceberg catalog is technical infrastructure: it's in the write path of every transaction. A data catalog is a discovery and documentation layer for humans. You'll likely have both, and the data catalog will read from the Iceberg catalog.

Can I switch Iceberg catalogs later?

Yes, but it's the painful migration in a lakehouse. The data files don't move, but every engine's connection config, all access-control policies, and any audit or lineage history live in the catalog. That switching cost — not the table format — is where lock-in lives in 2026, so choose deliberately.

Which Iceberg catalog should I default to?

Match your platform gravity: Unity if you're Databricks-centred, Glue if you're deep in AWS, Polaris (or Snowflake's managed Open Catalog) for a neutral, multi-engine REST catalog, Nessie if you specifically want git-like branching of your whole catalog. Neutral-by-default: Polaris.

Unity Catalog vs Glue Catalog — how do they compare?

They're the two gravity choices, and the trade is governance depth versus cloud convenience. Glue is IAM-native, serverless, and near-zero setup inside AWS, but its governance is thinner and it deepens your commitment to one cloud. Unity brings much richer governance — lineage, fine-grained policies, audit in one place — priced in Databricks commitment. Teams running Databricks on AWS commonly use both: Unity as the governing catalog, with Glue federated in for legacy tables.

How to Choose an Iceberg Catalog: Unity vs Polaris vs Glue vs Nessie

The table format decision got easy in 2026 — which means the hard decision moved. Every Iceberg table has exactly one authoritative record of “which metadata file is current,” and the component that holds it — the catalog — is now where governance, auditability, and the real lock-in of a lakehouse live. The main contenders: Unity Catalog (Databricks), Apache Polaris (open-sourced by Snowflake), AWS Glue, and Nessie. The decision rule, up front: follow your platform gravity — Unity for Databricks estates, Glue for deep-AWS shops — and when you want genuine engine neutrality, default to Polaris. The rest of this essay is why, and what each choice costs.

What a catalog actually is

Strip away the marketing and an Iceberg catalog does one mechanical thing: for every table, it stores a pointer to the current metadata file, and it swaps that pointer atomically when a commit happens. That single atomic swap is what gives an open table format ACID guarantees on dumb object storage. Everything else a catalog offers — access control, credential vending, auditing, lineage — is governance layered onto its position in the write path.

That position is why the choice matters: every engine, every pipeline, every commit goes through it. (Don’t confuse it with a data catalog like Atlan or Collibra — same word, different layer: one is transaction infrastructure, the other is documentation for humans.)

Unity vs Polaris vs Glue vs Nessie, compared

	Unity Catalog	Apache Polaris	AWS Glue	Nessie
Steward	Databricks (OSS core)	Apache Software Foundation	AWS	Dremio-backed OSS
Formats	Delta, Iceberg, Hudi	Iceberg only	Iceberg + others	Iceberg
Iceberg REST spec	Yes, endpoint	Yes — reference implementation	Yes	Yes
Access control	Strong, platform-integrated	Granular, engine-neutral	IAM-based	Basic
Managed offering	Databricks	Snowflake Open Catalog	AWS	Dremio
Distinctive strength	Lineage + governance in one platform	Neutrality	Zero-effort if you’re on AWS	Git-like branches and tags
Gravity	Databricks	None — that’s the point	AWS	Dremio / advanced teams

The structural fact underneath the table: the Iceberg REST catalog specification is what every engine now standardises against, so any spec-faithful catalog can serve Spark, Trino, Flink, and friends. The differences are in governance depth, ecosystem gravity, and who you’re comfortable depending on.

What connecting looks like

The catalog is one config block per engine — which is also the honest preview of a future migration:

# Spark: attach a REST catalog (Polaris, Unity's endpoint, Glue — same shape).
# Set at session start, not in SQL: catalog config is Spark configuration, and
# anything passed through SET lands in the Spark UI, event logs, and query history.
spark.sql.extensions = org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.lake = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.lake.type = rest
spark.sql.catalog.lake.uri = https://catalog.example.com/api/catalog
spark.sql.catalog.lake.warehouse = my_warehouse          # which warehouse this catalog serves
spark.sql.catalog.lake.rest.auth.type = oauth2           # token exchange, not a static secret
spark.sql.catalog.lake.oauth2-server-uri = https://idp.example.com/oauth/token
# The client secret is injected from a secret manager at submit time —
# it never appears in a notebook cell, a repo, or a log line.
spark.sql.catalog.lake.token = ${ICEBERG_CATALOG_TOKEN}

The spark.sql.extensions line is the one people omit and then spend an afternoon on: without it, MERGE, CALL, and time-travel syntax simply don’t parse. Note also where the credential isn’t — a catalog that is your security boundary is a poor place to start by leaking its own credential into query history.

With that config in place, the SQL is vendor-neutral:

-- Identical against any spec-faithful catalog answering that URI:
CREATE TABLE lake.sales.orders (
  order_id BIGINT, order_ts TIMESTAMP, amount DECIMAL(12,2)
) USING iceberg
PARTITIONED BY (days(order_ts));

Identical SQL, whichever catalog answers that URI. The switching pain is never the tables — it’s re-plumbing every engine’s config and rebuilding every access policy and audit trail that accumulated in the old catalog.

The Iceberg REST catalog spec defines both this wire protocol and the OAuth2 token exchange above; the open specification and the REST catalog OpenAPI definition are the authoritative reference when a vendor’s docs and your engine’s behaviour disagree.

The catalog decision follows gravity — with neutrality as the deliberate exception.

The honest trade-offs

Unity buys the deepest integrated governance — lineage, policies, audit in one place — priced in Databricks gravity. Glue is the lowest-effort answer inside AWS and the least interesting outside it. Polaris is the neutral choice: ASF-governed, REST-native, granular permissions, no engine agenda — but younger, and you’ll assemble more of the surrounding tooling yourself. Nessie gives you catalog-level branches and tags (test a whole pipeline against a branch of production data, then merge) — a genuinely distinctive capability that most teams don’t need yet.

Two of these pairings come up often enough to deserve their own treatment: Unity Catalog vs AWS Glue, where the answer for Databricks-on-AWS estates is usually both, and Apache Polaris vs Nessie, where the merger announced in 2024 still has not happened despite most guidance assuming it did.

Whichever you pick, pick it deliberately. In a lakehouse built on open formats, the catalog is the one component you can’t casually swap — it’s exactly where the next five years of governance decisions will accumulate.

What a catalog actually is

Unity vs Polaris vs Glue vs Nessie, compared

What connecting looks like

The honest trade-offs

Common questions