Iceberg table properties

Iceberg tables include a key-value map of table properties in their metadata. The initial map of table properties is set at table creation, and can be modified subsequently using ALTER TABLE ... SET TBLPROPERTIES. The Iceberg specification reserves a set of key properties and Foundry also provides a set of keys that are used to track Foundry-specific metadata, such as table RIDs.

Foundry-specific properties

Foundry-specific properties include:

Iceberg specification properties

The Iceberg specification also reserves a set of core Iceberg table properties ↗.

Foundry's Iceberg REST catalog includes the following constraints on certain core Iceberg table properties:

Where Foundry provides a default value, it takes precedence over the open source Iceberg specification default. Any property not explicitly set uses the Iceberg specification default value.

Set specific table properties in transforms

You can set specific table properties manually, for example, by running an ALTER TABLE statement in the Iceberg maintenance page, or by specifying properties in your transform code.

Below is an example code snippet setting table properties in transforms:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import polars as pl
from transforms.api import transform
from transforms.tables import IcebergOutput, TableOutput


@transform.using(output=TableOutput("/path/table"))
def compute(output: IcebergOutput) -> None:
    df_custom = pl.DataFrame({"phrase": ["Hello", "World"]})
    output.write_table(df_custom)

    tbl = output.iceberg().table()
    with tbl.transaction() as txn:
        txn.set_properties({
            "write.delete.mode": "merge-on-read",
            "write.update.mode": "merge-on-read",
            "write.merge.mode": "merge-on-read",
            "history.expire.max-snapshot-age-ms": "0",
        })

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from transforms.api import TransformContext
from transforms.api import transform
from transforms.tables import (
    TableOutput,
    TableTransformOutput,
)


@transform(
    output=TableOutput("/path/table")
)
def compute(ctx: TransformContext, output: TableTransformOutput):

    spark = ctx.spark_session

    df_custom = spark.createDataFrame([["Hello"], ["World"]], schema=["phrase"])
    output.write_dataframe(df_custom)

    identifier = output.identifier
    spark.sql(f"""
    ALTER TABLE {identifier}
    SET TBLPROPERTIES (
        "write.delete.mode": "merge-on-read",
        "write.update.mode": "merge-on-read",
        "write.merge.mode": "merge-on-read",
        'history.expire.max-snapshot-age-ms' = '0'
    )
    """)