We finished the Parquet-to-Iceberg conversion in a single sprint. The migration took the better part of a year. If you only read the vendor decks, you would expect those two spans to match; the distance between them is the whole post.
The open table format won, and it deserved to. Files in object storage, an Iceberg-style table layer that gives you snapshots and schema evolution and ACID over those files, a catalog that tracks where the current metadata lives, and any engine you like reading on top. Decoupled storage and compute, no single vendor holding your data hostage, the warehouse experience without the warehouse lock-in. I moved a multi-petabyte estate off a proprietary warehouse onto exactly this, across a few hundred tables feeding analysts, ML pipelines, and a real-time-ish reporting layer. I would do it again.
But the year is the half worth talking about, and the marketing skips it. The table format itself is the easy half: writing old Parquet into Iceberg tables is close to mechanical. Operating those tables in production is where the real labor lives, and almost none of it appears in the getting-started guide.
The diagram is the trap
Every pitch draws the architecture the same way.
Everyone draws the bottom and the top. The box off to the right, running on a schedule, is what decides whether the whole system is good or miserable to live with.
The clean version is real enough. Data files sit in object storage, the table format keeps manifests that record which files belong to which snapshot, and a catalog points each engine at the current metadata pointer. Spark, Trino, your warehouse engine of the week, all of them read the same tables without copying anything, which is as elegant as the diagram promises.
What the diagram leaves off, every single time, is that box on the side: the maintenance jobs. Compaction, snapshot expiry, orphan-file cleanup, manifest rewriting. A pitch treats them as a footnote; in production they are a standing service with an on-call rotation, and if you do not staff them on day one, you will meet them around day ninety, when queries that used to take seconds start taking minutes for reasons nobody can see.
The small-file problem will find you
Streaming and frequent writes are how the lakehouse earns its keep over the old batch warehouse, and also how you generate ten thousand tiny files a day in a single partition.
Every commit that writes a few rows writes new files. A Kafka-fed ingestion job landing micro-batches every minute is, from the table’s point of view, a small-file generator with a nice name. Object storage shrugs; the query engine does not, because planning a scan means opening and reading metadata for every one of those files. A thousand 2 MB files cost vastly more to plan and read than ten 200 MB files holding the same rows. Same data, a query an order of magnitude slower, and a bill that tracks the query.
The lakehouse does not fix any of this for you. It hands you the tools (compaction that rewrites small files into right-sized ones, and bin-packing on write) and then expects you to schedule them, tune them, and pay for the compute they burn. Compaction is itself a heavy write job, so now you are scheduling a job whose entire purpose is to clean up after your other jobs, and the two compete for the same cluster and occasionally the same table. We learned to run compaction on its own isolated compute, off-peak, partition by partition, with concurrency limits so it never collided with a writer mid-commit. No guide mentioned any of that, and every bit of it was load-bearing.
Schema evolution does what it says, not what you meant
The format’s headline feature is that schema evolution is safe. Add a column, rename one, reorder them, and old data still reads correctly, because the format tracks fields by a stable ID rather than by position. That is genuinely good, and genuinely better than the Hive-table world it replaced, where one careless column add could shift every value a position to the left.
The gotcha is that “the format won’t corrupt your data” is not the same promise as “your downstream consumers won’t break.” The format will happily let an upstream team rename a column with a clean, ID-tracked, fully reversible operation, and that rename will sail straight into a dozen dashboards and three ML feature pipelines that were selecting it by name. The table is fine. Everything reading the table is on fire. (We found this one the way you always find these, on a Monday, from a director, not a test.)
Partition evolution is the sharper edge. You can change a table’s partitioning without rewriting history, which sounds like a free lunch and is not one. Old data stays partitioned one way, new data lands partitioned another, and a query spanning the boundary has to reason about both layouts. It still works; it is just slower and stranger than anyone expects going in.
Governance is the project the deck forgets entirely
Governance surprised even me, and I went in expecting surprises.
In the old warehouse, access control lived in one place. You granted on a schema, a table, a column, sometimes a row, and the warehouse enforced it on every query, because every query went through the warehouse: one front door, one bouncer.
The lakehouse removes that front door by design. That is the whole point: many engines, direct access to the files. A permission you defined for Trino does not automatically bind Spark, and a clever user holding raw object-storage credentials can skip the table layer entirely and read the Parquet sitting under your nicest row filter. The security boundary you assumed was a table has become a bucket. If your governance model assumed the engine enforces access, the lakehouse deleted that assumption without telling you.
You get this back, but you have to build it, and the building is where the months go. The catalog becomes your enforcement point, which makes choosing a catalog a security decision and not just a metadata one. We locked the object-storage layer down so that neither humans nor rogue jobs could read raw files directly, routed every real access through the catalog and a governed engine, and rebuilt column masking and row filtering as policies the catalog enforced rather than properties of a single warehouse. Lineage and audit go on top of that, because in a regulated business “who read this and when” is a question you answer under oath, not in a retro. None of that ships with the lakehouse; what ships is the empty slot where it belongs.
The catalog is the decision you will live with longest
Everything above routes through one choice you make early and revisit constantly: which catalog. Hive Metastore because everything speaks it, and you inherit its age and its scaling limits. A newer transactional catalog with proper commit semantics, branching, and finer governance, and you take on a younger system at the center of everything. A vendor’s managed catalog, which is excellent and is also the lock-in you told the CFO you were escaping.
None of the three is a clean answer. The catalog is where an “open” architecture goes to negotiate with reality: the format is a spec anyone can read, while the catalog is the live service that decides what “current” means and who is allowed to ask. It is a commitment you will be running for five years.
The lakehouse won, and it should have: decoupled storage and compute, no proprietary vault, real engine choice, every word of it worth the move. The format was a weekend; the maintenance, the schema discipline, the governance rebuild, and the catalog we argued about for a year are what you actually sign up to operate. I do not mean any of that as a warning against making the move. I mean it as a warning against budgeting for the sprint and getting handed the year.