• Documentation
    • Search documentation
    karat

    +

    K

    API Reference ↗
    Data connectivity & integrationIceberg tablesIceberg storage architecture & settingsBYOB storage setup

    Configuring bring-your-own-bucket storage for Iceberg tables

    This guide describes how to configure customer-managed storage buckets for use with Foundry Iceberg tables. These steps are only required if you are using bring-your-own-bucket (BYOB) storage. If you are using Foundry-managed storage, no additional configuration is needed.

    Foundry currently supports BYOB storage on AWS (S3) and Azure (ADLS). Google Cloud Storage is not currently supported.

    Step 1: Create your storage bucket

    Provision your storage bucket ideally in the same region as your Foundry instance. While not required, this is recommended for optimal performance.

    Configure appropriate network access on the storage account to permit connectivity from Foundry.

    AWS S3

    1. Provision an S3 bucket.

    2. Create an IAM role with the following permissions on the S3 bucket and the KMS key used to encrypt it:

      PermissionResource
      s3:DeleteObjectS3 bucket
      s3:GetObjectS3 bucket
      s3:ListBucketS3 bucket
      s3:PutObjectS3 bucket
      kms:DecryptKMS key
      kms:EncryptKMS key
      kms:GenerateDataKeyKMS key
      sts:GetFederationToken—

    3. Create an IAM user or OIDC identity provider that can assume the role you created. You will use the IAM user's credentials or the OIDC provider's tokens when configuring the Data Connection source. See the S3 source documentation for more detail on supported authentication mechanisms.

    Azure ABFS

    1. Provision a storage account and container.

    2. Provision client credentials for authentication. See the ABFS source documentation documentation for more detail on supported authentication mechanisms.

    3. Grant the service principal access to the storage location:

      • Assign the Storage Blob Data Contributor ↗ role on the container where data will be stored.
      • Ensure the service principal has at least Delegator permissions at the storage account level. Container-level permissions alone are not sufficient for Foundry.

    Step 2: Create a Data Connection source

    BYOB sources and credentials are highly privileged. Configure them with restrictive access settings, such as by placing them in an administrator-only project. This limits who can access the credentials and prevents unauthorized modifications that could disrupt access to your BYOB Iceberg data.

    Once your bucket is provisioned, create a Data Connection source to connect Foundry to your storage:

    1. In Data Connection, create a new source (either S3 or ABFS) using the credentials you provisioned.
    2. Optional: Specify a base path prefix for your Iceberg tables by appending it to the source URL (for example, s3://bucket-name/base-path/). If no base path is provided in the source, Foundry will set foundry-iceberg as the Iceberg base path. Do not modify the base path after configuring the source as an Iceberg storage location, as this may disrupt access to existing tables.
    3. Foundry supports any authentication mechanism that Data Connection supports, including access key and secret or OIDC.
    4. Enable the following settings on the source:
      • Allow this source to be imported into code repositories.
      • Enable exports to this source
    Security consideration

    Do not grant credentials to the storage bucket directly to Iceberg clients or other tools. Instead, leverage credential vending through the Foundry Iceberg catalog to provide scoped, short-lived access. See Access delegation & credential vending for more information.

    Step 3: Add the bucket in Control Panel

    After creating your Data Connection source, add the bucket to your Iceberg storage configuration in Control Panel. See Configuring storage locations for instructions.