Build data pipelines with Iceberg tables

You can use Foundry Iceberg tables as inputs and outputs in Python transforms and Pipeline Builder. This page provides a high-level overview of the available workflows.

Choose a pipeline authoring experience

The following table summarizes the main options for building pipelines with Foundry-managed Iceberg tables:

All Foundry-managed Iceberg tables must use server-side encryption. The final column refers to the client-side encryption option and whether the given pipeline type currently supports reading and writing tables with this additional layer of Iceberg table encryption ↗.

Note that Foundry's APIs for single-node Python transforms support reading and writing client-side encrypted Iceberg tables, but filter pushdown is not currently supported in this scenario.

Build pipelines with Python transforms

Python transforms offer the ability to create Spark or single-node pipelines using a variety of open-source libraries and compute engines.

For details on working with Iceberg in Python transforms, see:

• Transforms with Iceberg
• Code examples
• Advanced Iceberg APIs
• Incremental transforms
• Incremental code examples

Build pipelines with Pipeline Builder

Pipeline Builder provides a no-code graphical interface for constructing data pipelines.

You can use Iceberg tables as inputs and outputs in Pipeline Builder:

1. Open Pipeline Builder and create a standard or faster batch pipeline.
2. Select Add Foundry data, then select an existing Iceberg table.
3. Add and connect the transforms required for your workflow.
4. In the pipeline outputs panel, select Add pipeline output.
5. Select Foundry Iceberg table, then configure the output name and location.
6. Preview the result, then deploy the pipeline.

To process an input incrementally, use a standard Spark pipeline and set the Iceberg input's computation mode to Incremental.