Changelog API technical primer

A changelog describes rows inserted, updated, or deleted between Iceberg snapshots ↗. In Python transforms, Foundry's source.changelog() API reads these changes in two modes:

CallResult
source.changelog(["id"])Identifier changelog: latest changes for each identifier, including ties
source.changelog()Net changes changelog: net inserts and deletes, preserving duplicate row counts

Foundry's reader, source.changelog(), and writer, output.apply_changelog(), perform the validation described on this page. Calling Iceberg's native create_changelog_view or MERGE INTO directly does not run these checks.

Changelog read range

A transform reads changes after the input snapshot processed in its previous build, through the input snapshot selected for its current build. For example, if the previous build processed V0 and the current build reads through V3, the changelog includes V1, V2, and V3. It excludes the previously processed V0. If the input is still at V0, the changelog is empty and Foundry does not validate identifiers.

Compaction and rewrites of unchanged rows do not produce logical changes.

Iceberg changelog scans do not currently support the delete files produced by merge-on-read. Keep source tables on copy-on-write.

Identifier changelog (has identifier columns)

Pass identifier columns to read the latest changes for each key:

Copied!
1 changelog_df = source.changelog(["id"])

Use the name of each top-level identifier column, including any dots or backticks in the name.

Source identifier uniqueness

With the default enforce_primary_key=True, Foundry checks that each identifier present in the changelog has zero or one source row in the starting snapshot and after each subsequent snapshot. Duplicate or negative row counts raise an error before the dataframe is returned. Foundry counts changes within a snapshot together.

Identifier validation scans the starting snapshot's identifier columns, when present, and processes changes across the requested range before returning the dataframe. This can increase read time even when few rows change.

The optional enforce_primary_key=False parameter skips this additional snapshot scan and row-count validation. It does not disable Iceberg's own checks or the writer validation in output.apply_changelog().

Ensure primary key uniqueness before disabling validation

Disabling primary key enforcement without external uniqueness guarantees can silently produce incorrect changelogs. A temporary duplicate followed by a deletion can produce a changelog that incorrectly deletes an identifier. A row with that identifier still remains in the source. Keep validation enabled unless the source already guarantees uniqueness at the start of the read range and after every snapshot within it. See Primary key enforcement before disabling validation.

Sequential updates to the same key are valid. The row count must stay zero or one at every snapshot boundary.

With a starting snapshot, the read requires that snapshot and the history connecting it to the ending snapshot. The read fails if that history is unavailable or the starting snapshot is outside the ending snapshot's ancestry.

Without a starting snapshot, counting starts at zero and assumes the history reaches table creation. The reader does not verify that assumption. Expired history can therefore produce an incomplete changelog.

For each key, Foundry then keeps changes from the latest snapshot that changed it. If several changes remain for the key in that snapshot, Foundry returns them all so the writer can reject ambiguous actions.

With primary key enforcement enabled, the reader produces the following results:

Source writesResult
Update id1 from val1 to val2, then to val3 in a later snapshotUPDATE_AFTER(id1, val3)
Append (id1, val1), then append (id1, val2) in a later snapshotError: duplicate source key
Append (id1, val1) and (id1, val2) in the same snapshotError: duplicate source key

The first result replaces the output value for id1 or inserts it if absent. Both append examples fail before latest-change selection, regardless of physical row order.

For example, suppose a row has one copy at V0, two identical copies at V1, and one copy at V2. With primary key enforcement enabled, reading V0 to V2 fails because V1 contains a duplicate key, even though both endpoint snapshots are unique.

A net changes changelog preserves row counts when reading, but applying them requires different handling when the source contains duplicates.

How updates are represented

Foundry uses Iceberg's create_changelog_view ↗ procedure with identifier columns. Iceberg removes identical delete and insert pairs caused by file rewrites, then pairs remaining deletes and inserts into UPDATE_BEFORE and UPDATE_AFTER rows. Foundry removes UPDATE_BEFORE and keeps changes at each key's highest _change_ordinal.

Changes across snapshotsReturned changes
Update a row, or insert it and then update itFinal UPDATE_AFTER
Insert or update a row, then delete itFinal DELETE
Change a key to a previously unused valueDELETE for the old key and INSERT for the new key

The columns passed to changelog() define the keys for this read. Iceberg's identifier fields in table metadata ↗ do not enforce uniqueness.

Net changes changelog (no identifier columns)

Create a net changes changelog by omitting identifiers. This mode preserves row counts, including duplicates:

Copied!
1 changelog_df = source.changelog()

A net changes changelog matches rows using all data columns and cancels matching inserts and deletes across the snapshot range. If a value changes from val1 to val2, then to val3, the result is DELETE(val1) and INSERT(val3). The intermediate val2 changes cancel.

This mode uses Iceberg's net_changes option.

A net changes changelog is not a general replacement for an identifier changelog passed to apply_changelog. An ordinary update produces a DELETE and an INSERT for the same key, which the writer rejects. Convert these changes to the intended output action. Selecting one row by _change_ordinal alone can choose the delete because both changes have the same ordinal.

The writer also cannot apply changes to duplicate row counts. If two identical rows become one, the net changes changelog returns one DELETE. Even when every data column is an output identifier, the writer rejects that delete because it matches two output rows. Replicating duplicate rows requires separate count handling or a full snapshot replacement.

Changelog columns

In both modes, source.changelog() returns a dataframe containing the source columns plus these metadata columns:

ColumnTypeMeaning
_change_typestringIdentifier changelog: INSERT, UPDATE_AFTER, or DELETE
Net changes changelog: INSERT or DELETE
_change_ordinalintZero-based order of the snapshot within the requested range
_commit_snapshot_idlongSnapshot that produced the change

Several rows can share an ordinal. It identifies a snapshot within the range, not an individual row or an order between rows in that snapshot. Ordinals restart for each read, so do not order changes from separate reads by _change_ordinal alone.

Example changelog evolution

The following examples use consecutive snapshots V0 through V3 and omit _commit_snapshot_id.

Example table evolution across snapshots

Each snapshot makes the following changes:

SnapshotWriteTable after the write
V0Insert (id1, val1)(id1, val1)
V1Update id1 to val2(id1, val2)
V2Update id1 to val3 and insert (id2, val2)(id1, val3), (id2, val2)
V3Change (id2, val2) to (id1, val3), leaving the existing id1 row unchangedTwo copies of (id1, val3)

Each starting snapshot in the following examples has already been processed. Reading V0 to V3 therefore excludes V0 and includes V1, V2, and V3, with ordinals 0, 1, and 2. Reading V1 to V2 includes only V2, so its ordinal is 0.

Identifier changelog example evolution

These examples use the default enforce_primary_key=True. Ranges ending at V1 or V2 keep the latest changes for id1 and id2. The final example deliberately introduces a duplicate source key at V3, so that identifier read fails.

Identifier changelog example: V0 to V1

idvalue_change_type_change_ordinal
id1val2UPDATE_AFTER0

Identifier changelog example: V1 to V2

idvalue_change_type_change_ordinal
id1val3UPDATE_AFTER0
id2val2INSERT0

Identifier changelog example: V0 to V2

idvalue_change_type_change_ordinal
id1val3UPDATE_AFTER1
id2val2INSERT1

Identifier changelog example: V0 to V3

The reader raises an error because V3 contains two rows with key id1. It returns no identifier changelog.

Net changes changelog example evolution

A net changes changelog preserves the difference in row counts between the start and end snapshots.

Net changes changelog example: V0 to V1

idvalue_change_type_change_ordinal
id1val1DELETE0
id1val2INSERT0

Net changes changelog example: V1 to V2

idvalue_change_type_change_ordinal
id1val2DELETE0
id1val3INSERT0
id2val2INSERT0

Net changes changelog example: V0 to V2

The following table shows the changes before netting. The matching insert and delete of (id1, val2) cancel, so they appear struck through:

idvalue_change_type_change_ordinal
id1val1DELETE0
id1val2INSERT0
id1val2DELETE1
id1val3INSERT1
id2val2INSERT1

The returned net changes changelog is:

idvalue_change_type_change_ordinal
id1val1DELETE0
id1val3INSERT1
id2val2INSERT1

Net changes changelog example: V0 to V3

Assume both V2 rows share a file. The V3 rewrite deletes (id1, val3) once and inserts it twice. The net changes changelog cancels that delete against the earlier insert from V2, leaving both V3 inserts. The insert and delete of (id2, val2) also cancel:

idvalue_change_type_change_ordinal
id1val1DELETE0
id1val2INSERT0
id1val2DELETE1
id1val3INSERT1
id2val2INSERT1
id1val3DELETE2
id2val2DELETE2
id1val3INSERT2
id1val3INSERT2

The returned net changes changelog is:

idvalue_change_type_change_ordinal
id1val1DELETE0
id1val3INSERT2
id1val3INSERT2

Both inserts have ordinal 2. They preserve the two copies at V3; the ordinal does not identify which copy came first.

Applying changes to an output table

The identifiers passed to source.changelog() group source changes. The identifiers passed to output.apply_changelog() match those changes to output rows. You must pass at least one output identifier column:

Copied!
1 output.apply_changelog(changelog_df, ["id"])
_change_typeOutput action
INSERT, UPDATE_AFTER, or UPDATEUpdate a matching row or insert it if absent
DELETEDelete a matching row; leave the output unchanged if absent
UPDATE_BEFOREIgnore the row

The writer accepts only these exact _change_type values. Unknown values, lowercase variants, and nulls raise an error.

Output identifier uniqueness

Supply at most one actionable row per output key, including when creating the table. Resolve duplicate keys introduced by joins or other transformations according to the intended output before writing. Do not select an arbitrary tied row. A new table contains only the inserted or updated rows.

The output must also have unique keys, including after the initial full snapshot write. If an actionable changelog row matches multiple existing output rows, the writer raises an error before merging. The writer does not check duplicate output keys that have no corresponding action. Null key values match other null key values. This validation adds a scan and join of output keys before the write.

Schema requirements

For an existing output, missing or extra changelog data columns raise an error. This comparison excludes the changelog metadata columns, but the writer requires the output's payload columns even when every row is a DELETE. Align the columns, or explicitly evolve or replace the output before applying changes.

Identifier types must match the output types or allow an Iceberg type promotion: int to long, float to double, or increased decimal precision with the same scale. Other conversions raise an error. If a conversion is intentional, cast the changelog identifiers to the output types before calling apply_changelog; the writer then checks the converted keys for duplicate actions.

Stable output identifiers

When transforming identifier changes, the mapping from source keys to output keys must remain unique and stable when non-key source values change. For example, reading by id and writing by destination_id leaves a stale output row if destination_id changes from A to B: the reader returns only the new row, so the writer upserts B without deleting A. This also applies when a column keeps its name but its values are derived from a mutable field. To change output keys, explicitly emit old-key deletes using retained before-images, or replace the output from the current source snapshot. The identifier reader does not return before-images.

For the full workflow, see change data capture (CDC) with changelogs and the filtered-output example.