Amazon S3

Connect Foundry to AWS S3 to read and sync data between S3 and Foundry.

Supported capabilities

Setup

1. Open the Data Connection application and select + New Source in the upper right corner of the screen.
2. Select S3 from the available connector types.
3. Choose to use a direct connection over the Internet or to connect through an intermediary agent.
4. Follow the additional configuration prompts to continue the setup of your connector using the information in the sections below.

Learn more about setting up a connector in Foundry.

Connection details

Required read and sync permissions for S3

The following AWS permission is required for interactive exploration of the S3 bucket:

Copied!
1
2
3
4
5
{
    "Action": ["s3:ListBucket"],
    "Resource": ["arn:aws:s3:::path/to/bucket"],
    "Effect": "Allow",
}

The following AWS permission is required for batch syncs, virtual tables and media syncs from S3:

Copied!
1
2
3
4
5
{
    "Action": ["s3:GetObject"],
    "Resource": ["arn:aws:s3:::path/to/bucket/*"],
    "Effect": "Allow",
}

See official AWS documentation on Policies and Permissions in Amazon S3 ↗ for more details on how to configure bucket policies in S3.

S3 proxy configuration (agent-based connections)

When connecting to S3 using a data connection agent, you can define proxy settings in two ways:

• Source Config: Define each proxy setting directly in the source configuration, as outlined in the table below.
• Agent's System Properties: As a fallback, you can configure the proxy settings within the agent's system properties. To achieve this, include the appropriate JVM arguments in the advanced configuration settings for the agent (for example, -Dhttps.proxyHost=example.proxy.com).

STS role configuration

STS role configuration allows you to make use of AWS Security Token Service ↗ to assume a role when reading from S3.

Cloud identity configuration

Cloud identity authentication allows Foundry to access resources in your AWS instance. Cloud identities are configured and managed at the enrollment level in Control Panel. Learn how to configure cloud identities.

When using cloud identity authentication, the role ARN will be displayed in the credentials section. After selecting the Cloud identity credential option, you must also configure the following:

1. Configure an Identity and Access Management (IAM) role in the target Amazon AWS account.
2. Grant the IAM role access to the S3 bucket to which you wish to connect. You can generally do this with a bucket policy ↗.
3. In the S3 source configuration details, add the IAM role under the Security Token Service (STS) role ↗ configuration. The cloud identity IAM role in Foundry will attempt to assume the AWS Account IAM role ↗ when accessing S3.
4. Configure a corresponding trust policy ↗ to allow the cloud identity IAM role to assume the target AWS account IAM role.

Virtual tables

This section provides additional details around using virtual tables from an S3 source. This section is not applicable when syncing to Foundry datasets.

When registering virtual tables, remember the following source configuration requirements:

• You must set up the source as a direct connection. Virtual tables do not support use of intermediary agents.
• Ensure that bi-directional connectivity and allowlisting is established as described under the Network Connectivity section in Connection details.
• If using virtual tables in Code Repositories, refer to the Virtual Tables documentation for details of additional source configuration required.
• If your bucket contains ., you must enable path-style access and set up the appropriate egress policy.

See the Connection Details section above for more details.

Delta

To enable incremental support for pipelines backed by virtual tables, ensure that Change Data Feed ↗ is enabled on the source Delta table. The current and added read modes in Python Transforms are supported. The _change_type, _commit_version and _commit_timestamp columns will be made available in Python Transforms.

Iceberg

An Iceberg catalog is required to load virtual tables backed by an Apache Iceberg table. To learn more about Iceberg catalogs, see the Apache Iceberg documentation ↗. All Iceberg tables registered on a source must use the same Iceberg catalog.

By default, tables will be created using Iceberg metadata files in S3. A warehousePath indicating the location of these metadata files must be provided when registering a table.

AWS Glue ↗ can be used as an Iceberg catalog when tables are stored in S3. To learn more about this integration, see the AWS Glue documentation ↗. The credentials configured on the source must have access to your AWS Glue Data Catalog. AWS Glue can be configured in the Connection Details tab on the source. All Iceberg tables registered on this source will automatically use AWS Glue as the catalog. Tables should be registered using database_name.table_name naming pattern.

Unity Catalog ↗ can be used as an Iceberg catalog when using Delta Universal Format (UniForm) in Databricks. To learn more about this integration, see the Databricks documentation ↗. As with AWS Glue, the catalog can be configured in the Connection Details tab on the source. You will need to provide the endpoint and a personal access token to connect to Unity Catalog. Tables should be registered using catalog_name.schema_name.table_name naming pattern.

Incremental support relies on Iceberg Incremental Reads ↗ and is currently append-only. The current and added read modes in Python Transforms are supported.

Parquet

Virtual tables using Parquet rely on schema inference. At most 100 files will be used to determine the schema.

Export data to S3

To export to S3, first enable exports for your S3 connector. Then, create a new export.

Required export permissions for S3

The following AWS permission is required to export data to S3:

Copied!
1
2
3
4
5
{
    "Action": ["s3:PutObject"],
    "Resource": ["arn:aws:s3:::path/to/bucket/*"],
    "Effect": "Allow",
}

See official AWS documentation on Policies and Permissions in Amazon S3 ↗ for more details on how to configure bucket policies in S3.

Export configuration options