Incremental processing with Iceberg tables

Incremental computation is an efficient method of processing an input to generate an output. Using incremental computation allows you to process only the data that changed since the last run rather than reprocessing the whole input every time.

This page provides a conceptual overview of working with Iceberg tables incrementally. For an API guide to Python transforms, see Incremental Python transforms and the accompanying Incremental code examples. For guidance on changelogs specifically, see Iceberg changelogs and CDC pipelines.

Incremental build support with Iceberg

The following table summarizes Foundry's support for incremental build computation with Iceberg:

Incremental indexing of Ontology object types

Object types backed by a copy-on-write Iceberg table are indexed incrementally at row level. Row edits and deletions in the backing table propagate to the Ontology as object edits and deletions, so deleting a single row updates only the corresponding object rather than re-indexing the entire table.

Because Iceberg copy-on-write applies row-level changes by rewriting whole data files, the amount of data read during an incremental index build scales with the size of the rewritten files rather than the number of changed rows. Changes that affect rows spread across many files are more expensive to index than the same number of changes concentrated in a few files.

Reading and writing changes

Reading changes

When reading an Iceberg table incrementally, Foundry retrieves the changes in a given snapshot range according to the read mode you request.

In append-only read mode, the default, Foundry can resolve incrementally when the snapshot range contains only appends or replaces (compactions). If an overwrite or delete snapshot is present, incremental checks fail and Foundry either falls back to a full non-incremental read or fails the job, based on your require_incremental setting.

In changelog read mode, Foundry can resolve incrementally across appends, overwrites, deletes, and replaces. This mode enables changelog-based reads, such as Iceberg's changelog view ↗ for change-data-capture style pipelines.

Writing changes

Iceberg tables support the standard Spark SQL write statements ↗.

The following table summarizes the statements most relevant to writing changes incrementally:

You can use these open-source APIs directly when writing incrementally inside Foundry; for examples, see Advanced Iceberg APIs. Foundry also provides convenience wrappers for INSERT INTO (via append-only incremental) and MERGE INTO (via Foundry's apply_changelog). You can see more details on using the Foundry APIs in Incremental Python transforms.

Copy-on-write and merge-on-read

Iceberg offers two strategies for applying row-level changes. The choice is made by the table's producer but constrains its consumers, making it a pipeline-level decision.

Whether your table uses copy-on-write or merge-on-read is governed by the write.delete.mode, write.update.mode, and write.merge.mode table properties. If these properties are not set explicitly, your table inherits the Iceberg default of copy-on-write. These properties only take effect when a row-level change is applied. A merge-on-read table that has only ever been appended to contains no delete files and behaves like a copy-on-write table.

The below diagram illustrates the difference between copy-on-write and merge-on-read when applying a single row edit:

Merge-on-read tables benefit from regular compaction using the Rewrite data files maintenance task, which applies outstanding delete files and consolidates the results into new data files. Read amplification therefore lasts only until the next compaction.