Iceberg format version support

The Apache Iceberg table format specification ↗ is versioned, with each version extending the previous, though not all tools and clients may support the latest versions.

Foundry supports reading and writing Iceberg tables in format versions 2 and 3.

Format version defaults

The format version used by default depends on whether client-side Iceberg table encryption is enabled.

Foundry defaults to v2 for tables where client-side encryption is disabled, due to limited client support for the v3 specification.

Foundry defaults to v3 for tables where client-side encryption is enabled, as Iceberg table encryption ↗ is a v3 feature.

Override format versions

You can optionally override the default format version by specifying the version explicitly using the format-version table property.

For example, you can run:

Copied!
1
2
3
4
5
6
7
8
9
10
11
@transform.spark.using(
    output=TableOutput("/path/table_name"),
)
def compute(ctx: TransformContext, output: TableTransformOutput) -> None:

    df_custom = ctx.spark_session.createDataFrame([["Hello"], ["World"]], schema=["phrase"])

    df_custom.writeTo(output.identifier)\
    .using("iceberg") \
    .tableProperty("format-version", "3") \
    .createOrReplace()

Copied!
1
2
ALTER TABLE `ri.tables.main.table.00000002...`
SET TBLPROPERTIES ('format-version' = '3');

Copied!
1
2
3
4
5
catalog.create_table(
    "path.table",
    schema=schema,
    properties={"format-version": "3"}
)