A changelog describes rows inserted, updated, or deleted between Iceberg snapshots ↗. In Python transforms, Foundry's source.changelog() API reads these changes in two modes:
| Call | Result |
|---|---|
source.changelog(["id"]) | Identifier changelog: latest changes for each identifier, including ties |
source.changelog() | Net changes changelog: net inserts and deletes, preserving duplicate row counts |
Foundry's reader, source.changelog(), and writer, output.apply_changelog(), perform the validation described on this page. Calling Iceberg's native create_changelog_view or MERGE INTO directly does not run these checks.
A transform reads changes after the input snapshot processed in its previous build, through the input snapshot selected for its current build. For example, if the previous build processed V0 and the current build reads through V3, the changelog includes V1, V2, and V3. It excludes the previously processed V0. If the input is still at V0, the changelog is empty and Foundry does not validate identifiers.
Compaction and rewrites of unchanged rows do not produce logical changes.
Iceberg changelog scans do not currently support the delete files produced by merge-on-read. Keep source tables on copy-on-write.
Pass identifier columns to read the latest changes for each key:
Copied!1changelog_df = source.changelog(["id"])
Use the name of each top-level identifier column, including any dots or backticks in the name.
With the default enforce_primary_key=True, Foundry checks that each identifier present in the changelog has zero or one source row in the starting snapshot and after each subsequent snapshot. Duplicate or negative row counts raise an error before the dataframe is returned. Foundry counts changes within a snapshot together.
Identifier validation scans the starting snapshot's identifier columns, when present, and processes changes across the requested range before returning the dataframe. This can increase read time even when few rows change.
The optional enforce_primary_key=False parameter skips this additional snapshot scan and row-count validation. It does not disable Iceberg's own checks or the writer validation in output.apply_changelog().
Disabling primary key enforcement without external uniqueness guarantees can silently produce incorrect changelogs. A temporary duplicate followed by a deletion can produce a changelog that incorrectly deletes an identifier. A row with that identifier still remains in the source. Keep validation enabled unless the source already guarantees uniqueness at the start of the read range and after every snapshot within it. See Primary key enforcement before disabling validation.
Sequential updates to the same key are valid. The row count must stay zero or one at every snapshot boundary.
With a starting snapshot, the read requires that snapshot and the history connecting it to the ending snapshot. The read fails if that history is unavailable or the starting snapshot is outside the ending snapshot's ancestry.
Without a starting snapshot, counting starts at zero and assumes the history reaches table creation. The reader does not verify that assumption. Expired history can therefore produce an incomplete changelog.
For each key, Foundry then keeps changes from the latest snapshot that changed it. If several changes remain for the key in that snapshot, Foundry returns them all so the writer can reject ambiguous actions.
With primary key enforcement enabled, the reader produces the following results:
| Source writes | Result |
|---|---|
Update id1 from val1 to val2, then to val3 in a later snapshot | UPDATE_AFTER(id1, val3) |
Append (id1, val1), then append (id1, val2) in a later snapshot | Error: duplicate source key |
Append (id1, val1) and (id1, val2) in the same snapshot | Error: duplicate source key |
The first result replaces the output value for id1 or inserts it if absent. Both append examples fail before latest-change selection, regardless of physical row order.
For example, suppose a row has one copy at V0, two identical copies at V1, and one copy at V2. With primary key enforcement enabled, reading V0 to V2 fails because V1 contains a duplicate key, even though both endpoint snapshots are unique.
A net changes changelog preserves row counts when reading, but applying them requires different handling when the source contains duplicates.
Foundry uses Iceberg's create_changelog_view ↗ procedure with identifier columns. Iceberg removes identical delete and insert pairs caused by file rewrites, then pairs remaining deletes and inserts into UPDATE_BEFORE and UPDATE_AFTER rows. Foundry removes UPDATE_BEFORE and keeps changes at each key's highest _change_ordinal.
| Changes across snapshots | Returned changes |
|---|---|
| Update a row, or insert it and then update it | Final UPDATE_AFTER |
| Insert or update a row, then delete it | Final DELETE |
| Change a key to a previously unused value | DELETE for the old key and INSERT for the new key |
The columns passed to changelog() define the keys for this read. Iceberg's identifier fields in table metadata ↗ do not enforce uniqueness.
Create a net changes changelog by omitting identifiers. This mode preserves row counts, including duplicates:
Copied!1changelog_df = source.changelog()
A net changes changelog matches rows using all data columns and cancels matching inserts and deletes across the snapshot range. If a value changes from val1 to val2, then to val3, the result is DELETE(val1) and INSERT(val3). The intermediate val2 changes cancel.
This mode uses Iceberg's net_changes option.
A net changes changelog is not a general replacement for an identifier changelog passed to apply_changelog. An ordinary update produces a DELETE and an INSERT for the same key, which the writer rejects. Convert these changes to the intended output action. Selecting one row by _change_ordinal alone can choose the delete because both changes have the same ordinal.
The writer also cannot apply changes to duplicate row counts. If two identical rows become one, the net changes changelog returns one DELETE. Even when every data column is an output identifier, the writer rejects that delete because it matches two output rows. Replicating duplicate rows requires separate count handling or a full snapshot replacement.
In both modes, source.changelog() returns a dataframe containing the source columns plus these metadata columns:
| Column | Type | Meaning |
|---|---|---|
_change_type | string | Identifier changelog: INSERT, UPDATE_AFTER, or DELETENet changes changelog: INSERT or DELETE |
_change_ordinal | int | Zero-based order of the snapshot within the requested range |
_commit_snapshot_id | long | Snapshot that produced the change |
Several rows can share an ordinal. It identifies a snapshot within the range, not an individual row or an order between rows in that snapshot. Ordinals restart for each read, so do not order changes from separate reads by _change_ordinal alone.
The following examples use consecutive snapshots V0 through V3 and omit _commit_snapshot_id.
Each snapshot makes the following changes:
| Snapshot | Write | Table after the write |
|---|---|---|
V0 | Insert (id1, val1) | (id1, val1) |
V1 | Update id1 to val2 | (id1, val2) |
V2 | Update id1 to val3 and insert (id2, val2) | (id1, val3), (id2, val2) |
V3 | Change (id2, val2) to (id1, val3), leaving the existing id1 row unchanged | Two copies of (id1, val3) |
Each starting snapshot in the following examples has already been processed. Reading V0 to V3 therefore excludes V0 and includes V1, V2, and V3, with ordinals 0, 1, and 2. Reading V1 to V2 includes only V2, so its ordinal is 0.
These examples use the default enforce_primary_key=True. Ranges ending at V1 or V2 keep the latest changes for id1 and id2. The final example deliberately introduces a duplicate source key at V3, so that identifier read fails.
id | value | _change_type | _change_ordinal |
|---|---|---|---|
id1 | val2 | UPDATE_AFTER | 0 |
id | value | _change_type | _change_ordinal |
|---|---|---|---|
id1 | val3 | UPDATE_AFTER | 0 |
id2 | val2 | INSERT | 0 |
id | value | _change_type | _change_ordinal |
|---|---|---|---|
id1 | val3 | UPDATE_AFTER | 1 |
id2 | val2 | INSERT | 1 |
The reader raises an error because V3 contains two rows with key id1. It returns no identifier changelog.
A net changes changelog preserves the difference in row counts between the start and end snapshots.
id | value | _change_type | _change_ordinal |
|---|---|---|---|
id1 | val1 | DELETE | 0 |
id1 | val2 | INSERT | 0 |
id | value | _change_type | _change_ordinal |
|---|---|---|---|
id1 | val2 | DELETE | 0 |
id1 | val3 | INSERT | 0 |
id2 | val2 | INSERT | 0 |
The following table shows the changes before netting. The matching insert and delete of (id1, val2) cancel, so they appear struck through:
id | value | _change_type | _change_ordinal |
|---|---|---|---|
id1 | val1 | DELETE | 0 |
id1 | val2 | INSERT | 0 |
id1 | val2 | DELETE | 1 |
id1 | val3 | INSERT | 1 |
id2 | val2 | INSERT | 1 |
The returned net changes changelog is:
id | value | _change_type | _change_ordinal |
|---|---|---|---|
id1 | val1 | DELETE | 0 |
id1 | val3 | INSERT | 1 |
id2 | val2 | INSERT | 1 |
Assume both V2 rows share a file. The V3 rewrite deletes (id1, val3) once and inserts it twice. The net changes changelog cancels that delete against the earlier insert from V2, leaving both V3 inserts. The insert and delete of (id2, val2) also cancel:
id | value | _change_type | _change_ordinal |
|---|---|---|---|
id1 | val1 | DELETE | 0 |
id1 | val2 | INSERT | 0 |
id1 | val2 | DELETE | 1 |
id1 | val3 | INSERT | 1 |
id2 | val2 | INSERT | 1 |
id1 | val3 | DELETE | 2 |
id2 | val2 | DELETE | 2 |
id1 | val3 | INSERT | 2 |
id1 | val3 | INSERT | 2 |
The returned net changes changelog is:
id | value | _change_type | _change_ordinal |
|---|---|---|---|
id1 | val1 | DELETE | 0 |
id1 | val3 | INSERT | 2 |
id1 | val3 | INSERT | 2 |
Both inserts have ordinal 2. They preserve the two copies at V3; the ordinal does not identify which copy came first.
The identifiers passed to source.changelog() group source changes. The identifiers passed to output.apply_changelog() match those changes to output rows. You must pass at least one output identifier column:
Copied!1output.apply_changelog(changelog_df, ["id"])
_change_type | Output action |
|---|---|
INSERT, UPDATE_AFTER, or UPDATE | Update a matching row or insert it if absent |
DELETE | Delete a matching row; leave the output unchanged if absent |
UPDATE_BEFORE | Ignore the row |
The writer accepts only these exact _change_type values. Unknown values, lowercase variants, and nulls raise an error.
Supply at most one actionable row per output key, including when creating the table. Resolve duplicate keys introduced by joins or other transformations according to the intended output before writing. Do not select an arbitrary tied row. A new table contains only the inserted or updated rows.
The output must also have unique keys, including after the initial full snapshot write. If an actionable changelog row matches multiple existing output rows, the writer raises an error before merging. The writer does not check duplicate output keys that have no corresponding action. Null key values match other null key values. This validation adds a scan and join of output keys before the write.
For an existing output, missing or extra changelog data columns raise an error. This comparison excludes the changelog metadata columns, but the writer requires the output's payload columns even when every row is a DELETE. Align the columns, or explicitly evolve or replace the output before applying changes.
Identifier types must match the output types or allow an Iceberg type promotion: int to long, float to double, or increased decimal precision with the same scale. Other conversions raise an error. If a conversion is intentional, cast the changelog identifiers to the output types before calling apply_changelog; the writer then checks the converted keys for duplicate actions.
When transforming identifier changes, the mapping from source keys to output keys must remain unique and stable when non-key source values change. For example, reading by id and writing by destination_id leaves a stale output row if destination_id changes from A to B: the reader returns only the new row, so the writer upserts B without deleting A. This also applies when a column keeps its name but its values are derived from a mutable field. To change output keys, explicitly emit old-key deletes using retained before-images, or replace the output from the current source snapshot. The identifier reader does not return before-images.
For the full workflow, see change data capture (CDC) with changelogs and the filtered-output example.