Set up a source

A source is any sort of external data that you connect to Foundry. For example, a source could be a Postgres database, an S3 bucket, a filesystem on a Linux server, an SAP instance, or a REST API on the internet.

At a high level, below are the steps required to connect a source to Foundry. Note that Step 1 through Step 3 may require you to change or validate configurations within your existing architecture:

1. Ensure there is a valid network connection between the source and Foundry.
2. Provision valid credentials for Foundry to authenticate against the source.
3. If the source will be accessed via an agent, then ensure the agent has the appropriate drivers to access the source.
4. Finally, create the source in Data Connection.

Once you have this source connection established, you can configure syncs to bring specific sets of data into Foundry. Syncs can be entirely configured through the Data Connection UI, so the source setup is the final task that may require configurations to be updated in your organization's environment before you can access your data within Foundry.

Configure network access

To connect Foundry to a source, first validate network connections. A source using an agent needs two valid network connections: (1) from the agent host to the source and (2) from the agent host to Foundry. A direct connection requires a single valid network connection: from Foundry to the source.

You will need to make sure there is a connection between an agent that has been set up within your network and the source. The agent acts as a single point of validated entry to Foundry from your network and will handle the process of reading and sending data on to the Foundry VPC; for each new source you will only need to ensure there is a valid connection between the agent and the new source.

The steps required to establish this connection will vary depending on your organizational network settings. Regardless of your specific setup, the goal is for the agent to have the ability to connect to the source. This could involve the configuration of egress settings on the agent host, ingress settings on the source, firewall rules across the network, proxy settings on the agent, adding source system certificates to the agent truststore, and so on.

If you need to configure proxy settings for the Agent to reach the Source, you can do this via the Data Connection interface.

Provision credentials

In the majority of cases, Foundry will require authorized credentials (such as a username and password) to access sources. It is best practice to use a service account specifically for Foundry. Note that only one set of credentials is required even if you have multiple agents—the credentials are associated with the specific source resource in Foundry and will be used by any agents that you assign to that source.

Provision a service account for the source following any internal guidelines and processes that your organization has for establishing service accounts. Note the credentials before proceeding to the next step.

Add drivers, if needed

JDBC sources may require adding drivers needed to connect to your source. For agent-based sources, you will need to add the driver within the agent settings page for the agents you intend to use for the source connection.

Learn more about adding drivers.

Create the source in Data Connection

Once the above steps are done, you can proceed with creating the source in Data Connection:

• After logging in, navigate to Data Connection using the sidebar.
• Select the Sources tab.
• Select New source in the top-right.
• Select the source type corresponding to your data source.
• Select Via an agent, then select Continue in the bottom right.

Save the source in a Project

Next, name your source and choose a Project to place it in. We generally recommend creating a new Project for each source, as this provides the most natural way to permission datasets derived from this source.

You can read more about source permission best practices or consult the full guidance for how to structure data pipelines end-to-end in Foundry.

Select Create source and continue in the bottom-right.

Configure source

Add details about how to connect to your source. These will depend on the source type you are using and typically consist of basic credentials such as connection URLs, cloud provider regions, and so on.

Add credentials

Add the credentials you provisioned previously to allow the source to connect to your data.

Save and continue

Select Save in the bottom-right to complete setting up your source. Once your source is fully set up, you can proceed to set up a sync to bring data into Foundry.

Troubleshooting

If the source does not work properly, check the following:

1. Ensure the agent is healthy (if using an agent).
2. Log in to the agent's host and confirm that a connection to the source can be established independently of Foundry.
3. Check log files on the agent's host, as suggested for sync failures.