Resource usage for Iceberg tables

Working with Iceberg tables uses Foundry compute and storage. You can review this usage and its associated cost in the Resource Management application.

Usage reported in Resource Management

Foundry compute

Just as with Foundry datasets, compute usage on Foundry Iceberg tables is determined by the Foundry application that performs the work. For example:

• Python transforms and Pipeline Builder jobs use Foundry compute while reading inputs, executing transformation logic, and writing outputs.
• SQL queries use compute to run queries against Iceberg tables and build Iceberg outputs.
• Data Connection syncs use compute while extracting data from an external source and writing it to an Iceberg table.

Foundry measures compute in compute-seconds. The allocated virtual CPUs and memory, the duration of the operation, the amount of data processed, and the complexity of the operation can all affect compute usage.

Iceberg storage

For an Iceberg table using Foundry-managed storage, Resource Management currently reports the total size of the live data and delete files referenced by the latest snapshot on the main branch.

As with Foundry datasets, storage is measured over time and reported in gigabyte-months. For example, storing 100 GB for an entire month generates 100 gigabyte-months of usage.

Note that the reported size does not currently include:

• Files exclusively referenced by historical snapshots or other branches.
• Iceberg metadata files, such as manifests and manifest lists.
• Orphaned files not referenced by any snapshots.

As a result, the storage reported in Resource Management can be lower than the table's complete physical storage footprint.

Storage is not reported for bring-your-own-bucket Iceberg tables.