Apache Iceberg is an open table format for large analytic datasets stored in object storage systems like Amazon S3, Google Cloud Storage, and Azure Data Lake Storage. Originally developed at Netflix and contributed to the Apache Software Foundation, the project reached top-level ASF status in 2020. Rather than prescribing a specific query engine, Iceberg defines a specification for how table metadata, schema history, partitioning, and data files (Parquet, ORC, or Avro) are organized and versioned together on disk. Think of it as a contract between storage and compute: any engine that implements the spec can read and write the same table safely.
Organizations care about Iceberg because the older Hive table format, which dominated data lakes for over a decade, lacks transactional guarantees, makes concurrent writes dangerous, and requires expensive data rewrites whenever a schema changes. Iceberg eliminates these problems by treating every write as an atomic snapshot commit, giving teams the ability to run multiple engines against the same dataset without data corruption, query tables as they existed at any past point in time, and evolve schemas freely without touching underlying files. For engineering teams managing high-volume telemetry, log analytics, or observability pipelines, those capabilities translate directly into reduced operational risk and lower infrastructure cost.
Key Takeaways
- Apache Iceberg is a table format specification, not a query engine, so it works alongside existing engines like Spark, Flink, and Trino rather than replacing them.
- Every write operation creates an immutable snapshot, enabling time travel, incremental reads, and safe concurrent access through optimistic concurrency control.
- Hidden partitioning lets engineers define partition transforms once at the table level, removing the need to hard-code partition predicates in every query.
Hive-based tables store partition metadata in a central metastore and rely on directory listings to discover files at query time. At scale, those directory listings become slow, concurrent writers overwrite each other without warning, and changing a column type often means rewriting entire partitions. These limitations forced teams to choose between flexibility and reliability, and most chose neither — they built brittle pipelines that broke whenever upstream schemas changed.
Iceberg replaces the flat directory model with a tree of immutable metadata objects: a catalog entry points to a table metadata file, which points to a manifest list, which points to manifest files that each reference a set of data files along with column-level statistics including min/max values and null counts. Query planners use those statistics to prune files before reading a single byte of actual data, which cuts scan costs dramatically on large-scale log and telemetry workloads.
Iceberg supports adding, dropping, renaming, reordering, and widening columns without rewriting data files. It tracks each column by a stable numeric ID rather than by name, so renaming a column does not break existing files or downstream readers. Partition layouts can also change over time — a table initially partitioned by day can migrate to hourly partitioning for newer data — and Iceberg handles both layouts transparently within the same table.
Apache Spark, Apache Flink, Trino, Dremio, Hive, and others all ship first-class Iceberg integrations. Cloud providers have followed: AWS Athena, EMR, Glue, and S3 Tables support Iceberg natively, Google BigQuery supports Iceberg via BigLake, and Snowflake offers native Iceberg Tables. This breadth of support means a team can ingest with Flink, transform with Spark, and serve ad-hoc queries through Trino without ever duplicating data or converting formats.
Alongside Delta Lake and Apache Hudi, Iceberg is one of three dominant open table formats enabling ACID semantics on data lakes — a pattern the industry commonly calls the "lakehouse" architecture. For teams evaluating data lake strategies, Iceberg's engine-agnostic specification and broad cloud adoption make it the lowest-friction starting point.
Adopting Iceberg introduces operational considerations that teams should plan for before writing production data.
Metadata overhead at extreme scale. When a table accumulates millions of small files — common in streaming ingestion scenarios with frequent micro-batch commits — the manifest files themselves multiply. Query planners must read many manifests during planning, which adds latency. Regular compaction jobs that merge small data files into larger ones and bin-pack manifests are not optional at scale; they are routine maintenance.
Catalog management. Iceberg requires a catalog service to store the pointer from a table name to its current metadata file. Options include the Hive Metastore, AWS Glue, Project Nessie, a JDBC database, or a REST catalog. Choosing the wrong catalog for your scale or access pattern creates a bottleneck, and migrating catalogs later is non-trivial.
Snapshot expiration and storage costs. Every commit adds a new snapshot. Without a scheduled expire_snapshots maintenance procedure, historical metadata and orphaned data files accumulate indefinitely, inflating storage bills. Teams new to Iceberg frequently discover this only after costs have already grown.
Row-level delete performance. The Iceberg v2 spec adds row-level deletes via merge-on-read delete files, enabling UPDATE, DELETE, and MERGE statements. Uncompacted delete files accumulate over time and degrade read performance until a rewrite job merges them into the base data files.
Start by picking a catalog that matches your existing infrastructure — AWS Glue if you run on AWS, a REST catalog or Nessie for cloud-agnostic setups. Define your first table using a supported engine (Spark is the most documented path), and write a small batch of Parquet files to verify end-to-end reads and writes. From there, layer in hidden partitioning by choosing transforms that reflect your most common query filters, typically hour or day for time-series data. Schedule compaction and snapshot expiration jobs from the beginning rather than retrofitting them later. For streaming ingestion, configure Flink's Iceberg sink with a checkpoint interval that balances latency against file count. Keep an eye on manifest count metrics as data volume grows; a spike in planning time is almost always a compaction signal.
For context on how open table formats like Iceberg fit into broader analytics architectures, the log analytics guide and the observability architecture overview cover complementary patterns worth reading alongside this article.
What is the difference between Apache Iceberg and a query engine like Trino or Spark?
Iceberg is a table format specification that defines how metadata and data files are laid out and versioned in object storage. Trino and Spark are execution engines that read and write those files; they implement the Iceberg spec through catalog connectors. You always need an engine to query an Iceberg table — Iceberg itself does not execute queries.
How does time travel work, and how far back can you go?
Each write operation creates an immutable snapshot with a timestamp and ID. You can query any snapshot by passing its ID or a timestamp to your engine, and Iceberg resolves the correct metadata automatically. You can only travel as far back as unexpired snapshots allow; once a snapshot expires through the maintenance process, data files referenced only by that snapshot become eligible for deletion.
Does Iceberg support real-time streaming ingestion?
Yes. Apache Flink ships a native Iceberg sink that commits micro-batches as snapshots on each checkpoint interval, typically ranging from seconds to a few minutes. Downstream consumers can perform incremental reads by requesting only snapshots newer than a given ID, which makes Iceberg viable for near-real-time observability pipelines. For sub-second latency requirements, teams usually pair Iceberg with a hot-path streaming store and use Iceberg as the durable archival tier.
Can multiple engines write to the same Iceberg table simultaneously?
Yes, through optimistic concurrency control. Each writer reads the current metadata, prepares its commit, and then attempts an atomic swap of the metadata pointer. If another writer committed first, the slower writer retries. This model avoids distributed locks while still guaranteeing that only one snapshot becomes the committed table state at any moment.