Apache Iceberg™ is an open table format that provides data warehouse-grade features, such as atomic commits, partition pruning, schema evolution, and snapshot-based time travel. It works on cloud object stores like S3, Azure Blob Storage, Google Cloud Storage, and HDFS.
Its real strength, however, is portability: a single Iceberg catalogue can be queried by Databricks Spark, Snowflake, BigQuery, Trino/Starburst, AWS Athena/Glue, and even the new Microsoft Fabric runtime, allowing teams to choose the best engine for each workload while sharing one copy of the data.
Interest in the format surged after Databricks agreed to buy Tabular for roughly two billion dollars, a deal widely interpreted as an “Iceberg arms race” response aimed at Snowflake. That hype is deserved, but every technology has limitations, sometimes not discussed enough during the peak hype phase. The next sections outline the practical limitations I’ve observed and those highlighted during Data Council 2025.
Iceberg Implementation Limitations in 2025
Designed for Big Data first, Small Data usability needs improvement
Apache Iceberg began life at Netflix as a table format for petabyte-scale, slow-moving data sets, and the spec still introduces itself that way. Yet most organisations today operate at far smaller volumes: MotherDuck’s “Big Data is Dead” survey found the median analytic warehouse is below 100 GB, while critical datasets are often just a few gigabytes. Design choices that are optimal at Netflix scale, extra indirection layers, and deeply nested metadata translate into friction when your biggest table fits on a laptop.
Much like Kubernetes under the hood, Iceberg exposes powerful but low-level primitives. Its own creator, Ryan Blue, has warned about this in his talk “Why You Shouldn’t Care About Iceberg”. You should care about higher-level abstractions such as SQL or Spark; Iceberg is for database developers.
One practical rule of thumb: if the whole table still fits in memory in a single Parquet file or Apache Arrow, keep it that way. The Iceberg should be a database vendor problem, not a data engineer team’s problem.
Metadata overhead after writes amplification
A one-row update in Iceberg writes a new Parquet (or delete-file), a manifest that lists it, and a manifest-list file that lists the manifest, plus a fresh table-metadata JSON. Four blob storage writes each can take hundreds of milliseconds. So what can take a fraction of a millisecond on Postgres (modern NVM SSD) takes seconds in Iceberg.
This worsens as you perform more of them or updates across partitions. Your Iceberg query performance might be unusable. This is called file explosion, and you need to maintain your table through compactions to receive decent performance. The current Iceberg implementations usually avoid default partial compaction or vacuum as seen in many databases.
Features to make it less painful for small-scale users got lower priority. Ryan Blue mentions a single-file commit as an aspirational feature for Iceberg Spec V4, but as of May 2025, work is purely conceptual today.
Compaction and maintenance tax
To restore performance, you must routinely compact both data and metadata files. The reference implementation assumes Spark; on AWS, the recommended path is a managed Spark or Athena OPTIMIZE job, both billed separately. This adds another complexity, delay, and cost. It is worthwhile at the petabyte scale but questionable at the gigabyte scale.
A fragmented and incomplete ecosystem
The Iceberg spec is language-agnostic, but in practice, most production code still runs through the original Java stack. Newer ports in Python (PyIceberg), Rust, Go and even an early C++ effort are catching up, yet they are several releases behind the JVM reference implementation and miss ancillary tooling such as compaction services.
A similar long-tail exists one layer down: many engines can read Apache Parquet™, but few implement the newer encodings, so writers fall back to “safe” but sub-optimal layouts.
Iceberg itself is also evolving. Features introduced in format-v2 (row-level deletes, row-level IDs) and the just-announced format-v3 (queryable row-lineage, manifest skipping) are optional, so a table that validates in Spark may fail elsewhere at query time. In large organisations the slowest engine usually wins this standoff, effectively downgrading the entire lake to “common subset” mode. Most enterprise data warehouses charge on compute, so they have less incentive and urgency to fix that.
Typical gaps today at most vocal companies claiming Iceberg support.
Many popular query engines lack write support
ClickHouse exposes Iceberg only as a read-only engine and is still working on insert support. DuckDB’s Iceberg extension is similar, writing is limited to nightly builds as of May 20, 2025. Even those who implement it do that poorly, so you are left with leveraging Spark, Flink or Trino.
Merge-on-Read deletes
DuckDB recently fixed equality/positional delete handling, but older builds error out. ClickHouse cannot yet apply row-based delete files at all.
Efficiency gaps
Parquet with Iceberg is several times less performant in many query engines than native formats. Two to three times slower is common across ClickHouse and DuckDB. Even in most favourable tests, Snowflake has a 20% Iceberg penalty vs. native tables.
However, to get the best performance, query engines leverage Puffin files, which store extra statistics in metadata. Unsurprisingly, those files are vendor-specific, and they are not implemented in many open-source implementations. So, Iceberg fully managed by Snowflake or Databricks will be way more efficient, and by design, there is an efficiency gap for using an alternative implementation.
Unstructured data, wide tables
Generally, Iceberg excels when the schema is well-defined. Though many real-world use cases are unstructured, there are gaps to be closed. Basic primitives such as VARIANT are just being introduced. So, no luck if you are putting JSONs in your database and hoping the database will optimize that for you. In contrast, warehouse systems such as Snowflake have treated schemaless data as first-class for a decade, as described in the 2016 SIGMOD paper, where the authors described optimistic conversions and semi-structured data.
Wide tables, ones with many sparingly used columns, are also a poor fit for Iceberg. Their metadata model and gathering statistics per column do not work if you have too many of them (the default is statistics for 100 columns) and require manual tuning.
This is a challenge, particularly for observability and security workloads. The OpenTelemetry schema is quite broad, though it has many rarely used columns. Many products allow defining their own fields, which adds another bucket of challenges.
Governance and security are mostly out of scope
Data warehouses and data lakes are the crown jewels in many enterprises with sensitive data. Iceberg has no place to implement policies such as deny salary column to contractors. Iceberg will faithfully surface every byte it reads; blocking or redacting data is somebody else’s job.
Column masks, row filters and dynamic views have to be implemented in higher layers. Though Iceberg does not even provide a way to store security rules or controls.
In theory, Iceberg could introduce a security spec, and every query engine could adopt it. Though we are at least years away from that reality. Ironically, some of the enterprises might be better off sending all data to Snowflake or Databricks. Just rely on their security measures, and even if you overpay, avoiding a single data leakage is worth that price.
Iceberg’s hard design limits
All of the previous limitations were due to implementation. There might be a blocker in 2025
for some use cases, though, they can be addressed. In 2025, many data platforms already have a native format with superior performance and capabilities to Iceberg. I hope Iceberg Spec and Implementation eventually evolve, and many of them will disappear or be less important.
But data lake architecture and Iceberg also adhere to the laws of physics, so let me list some of the items that are unlikely ever to be addressed. The quote from table spec “...format that is designed to manage a large, slow-changing collection…”.
Limited write concurrency
Apache Iceberg’s optimistic-concurrency model guarantees serializable isolation, but it also means every writer must win an atomic metadata-file swap; conflicting commits are aborted and retried. In practice, you can sustain a few commits per minute, but not thousands per second — so OLTP-style workloads still need something like Postgres or MySQL, and you CDC the result into Iceberg later (e.g. using Mooncake).
For example, Adobe engineers experienced a hard limit of 15 transactions per minute, while a small-scale OLTP database can easily support thousands of transactions per second.
Limited real-time capabilities
Some use cases need real-time or near-real-time capabilities. They have low tolerance for delay, though they have to support out-of-order data. For example, monitoring and troubleshooting live production, each minute of outage matters and has a huge revenue impact. The Site Reliability Engineers need quick feedback between actions and their impact on the system.
Although Apache Iceberg is a rock-solid foundation for “cold” observability and security data, it struggles with high-velocity streams. Fresh rows are invisible until the writer finishes uploading Parquet files and atomically swaps the table-metadata pointer in the catalogue. This prioritizes atomic batch transactions over immediate visibility.
For these use cases, I recommend specialised data platforms such as Hydrolix, InfluxDB, Astra or ClickHouse.
Egress economics
Though sending data to most of the cloud is free, sending it out is typically very costly. Storing 1 TB in S3 for a month costs about $23, but moving that same terabyte out of AWS still starts at roughly $0.09 / GB, i.e. $90 — three months of storage. GCP and Azure are in the same ball-park; Only niche providers got free or affordable egress, such as Oracle Cloud, Linode and Cloudflare R2.
However, egress could be the dominant cost for all major cloud users. Because of these costs, many multi-cloud projects were doomed by economics. Though this seems to have nothing to do with Iceberg, it fundamentally incentivises picking specific vendors and their region, and sticking to it. It promotes some standardization, especially among hosted offerings. Certainly, it forced Snowflake or Databricks to operate out of many regions (or just use us-east-1 so you can blame the outage on your provider).
To some degree, Iceberg and replication on data storage may be good ways to control those costs. For example, Snowflake shipped Egress Cost Optimiser that minimizes paying multiple times that price.
However, it is unlikely that major clouds will give up their cost powers unless they are forced by regulators (e.g. European Data Act forces them to waive egress during one-time migration out).
Final thoughts
Iceberg is awesome. It is part of a broader trend of the reconstructed database. There are a plethora of great reasons why you should use it. It continues to innovate and will get better. Plenty of good articles were written about it.
However, like any technology, it has limitations: it’s best suited for sharing large datasets of slowly moving data across query engines, but for some workloads (such as real-time engines) you might be better off choosing a different solution today.
I would like to thank Suman Karumuri, Xiaodan Zhuang (Founder of Greptime) for providing feedback on the draft.
