A Foundry source represents a connection between Foundry and an external system. Some example Foundry source types include a Postgres database, an S3 bucket, a filesystem on a Linux server, an SAP instance, or a REST API over the Internet.
At a high level, below are the steps required to connect a source to Foundry. Note that Step 1 through Step 3 may require you to change or validate configurations within your existing architecture:
Once you have this source connection established, you can configure syncs to bring specific sets of data into Foundry. Syncs can be entirely configured through the Data Connection UI, so the source setup is the final task that may require configurations to be updated in your organization's environment before you can access your data within Foundry.
To connect Foundry to an external system, first validate network paths:
You will need to set up an agent if the external system you are connecting to is hosted on a separate network from Foundry's network.
If you decide to use a Foundry worker source with agent proxy policies, the agent will be used as a networking proxy only and the compute will happen in Foundry. If you decide to use an agent worker source, the agent will serve both as networking proxy and a compute engine.
Ensure there is a connection between an agent set up within your network and the external system. The agent acts as a single point of validated entry to Foundry from your network and will handle the process of reading and sending data on to the Foundry instance. For each new system, you will only need to confirm there is a valid connection between the agent and the new system.
You will not need to establish direct network egress from the external system to Foundry, as traffic only flows from the agent to Foundry and from the agent to the external system. Learn more about the architecture of data connection.
The steps required to establish this connection will vary depending on your organizational network settings. Regardless of your specific setup, the goal is for the agent to have the ability to connect to the external system. This could involve the configuration of egress settings on the agent host, ingress settings on the external system, firewall rules across the network, proxy settings on the agent, adding source system certificates to the agent truststore, and so on.
If you need to configure proxy settings for the agent to reach the external system, you can do this through Data Connection.
You must have the Information security officer
role on your enrollment to configure network egress. If you do not have permissions to configure egress, contact your Palantir administrator to request access.
You can find the Information security officer
role in the Enrollment permissions section of Control Panel. A user must have the Enrollment administrator
role to view this section.
Foundry worker sources additionally require network egress policies to route the traffic.
To configure a network policy, navigate to the Network egress section in Control Panel. Pick a direct connection policy if the external system you are connecting to is hosted in the same network than Foundry. Pick an agent proxy egress policy if the source you are connecting to is in a separate network from Foundry.
If you are unable to view the Network egress section, contact your Palantir administrator to set up the network policy.
For most cases, Foundry will require authorized credentials (such as a username and password) to access external systems. We recommend using a dedicated service account with credentials scoped specifically for the required access in Foundry.
Provision a service account for the source following any internal guidelines and processes that your organization has for establishing service accounts. Note the credentials before proceeding to the next step.
Once the above steps are done, you can proceed with creating the source in Data Connection:
Next, name your source and choose a Project to place it in. We generally recommend creating a new Project for each source, as this provides the most natural way to permission datasets derived from this source.
You can read more about source permission best practices or consult the full guidance for how to structure data pipelines end-to-end in Foundry.
Select Create source and continue in the bottom-right.
On the next setup page, select the network policy you configured earlier by choosing Use existing policy and searching for the policy name.
Add details about how to connect to your source. These will depend on the source type you are using and typically consist of basic credentials such as connection URLs, cloud provider regions, and so on.
JDBC sources may require you to uplod JDBC drivers, then specifying which Java class from the driver should be used.
External systems using self-signed certificates require you to upload certificates to ensure the connection can be trusted.
For Foundry worker connections, add certificates on to the source itself using the following steps:
Learn how to configure certificates on an agent when using agent work connections..
Add the credentials you provisioned previously to allow the source to connect to your data.
Select Save in the bottom-right to complete setting up your source. Once your source is fully set up, you can proceed to set up a sync to bring data into Foundry.
Preview the source by selecting Preview in the right panel to confirm the connection has been successfully established. If the source does not work properly, refer to our sync troubleshooting documentation. If you are using an agent worker, review our agent troubleshooting documentation as well.