Resource usage for Iceberg tables

Working with Iceberg tables uses Foundry compute and storage. You can review this usage and its associated cost in the Resource Management application.

Usage reported in Resource Management

Foundry compute

Just as with Foundry datasets, compute usage on Foundry Iceberg tables is determined by the Foundry application that performs the work. For example:

• Python transforms and Pipeline Builder jobs use Foundry compute while reading inputs, executing transformation logic, and writing outputs.
• SQL queries use compute to run queries against Iceberg tables and build Iceberg outputs.
• Data Connection syncs use compute while extracting data from an external source and writing it to an Iceberg table.

Foundry measures compute in compute-seconds. The allocated virtual CPUs and memory, the duration of the operation, the amount of data processed, and the complexity of the operation can all affect compute usage.

Iceberg storage

For an Iceberg table using Foundry-managed storage, Resource Management reports the total size of all data and delete files referenced by the table's current metadata. This includes files referenced by historical snapshots and by other branches and tags, as well as the latest snapshot on the main branch.

As with Foundry datasets, storage is measured over time and reported in gigabyte-months. For example, storing 100 GB for an entire month generates 100 gigabyte-months of usage.

The reported size does not include:

• Iceberg metadata files, such as table metadata files, manifest lists, and manifests.
• Orphaned files not referenced by any snapshot.

Storage is not reported for bring-your-own-bucket Iceberg tables.

Manage storage usage

Use snapshot expiration from the table's Maintenance tab to limit retained history and reduce reported storage.

Changes from snapshot expiration may take some time to appear in Resource Management.