Core concepts

This page describes the core concepts used throughout Data Connection.

Sources

A Source represents a single connection, including any configuration necessary to specify the target system and the credentials required to successfully authenticate. Sources must be configured with a particular runtime depending on the networking between the Palantir platform and the target system. The runtime also defines where any capabilities used with the source will be run.

A source is set up based on a particular connector (also referred to as a source type). A broad range of connectors are available in the Palantir platform, designed to support the most common data systems across organizations. Depending on the connector and runtime selected, different capabilities may be available.

For systems without a dedicated connector, the generic connector or REST API source may be used with code-based connectivity options such as external transforms, external functions, and compute modules.

Credentials

A credential is a secret value that is required to access a particular system; that is, credentials are used for authentication. Credentials can be passwords, tokens, API keys, or other secret values. In the Palantir platform, all credentials are encrypted and stored securely. Depending on the runtime, secrets may be stored locally on a Data Connection agent, or directly in the platform.

Some sources are able to authenticate without storing any secrets, such as when using OpenID connect, outbound applications, or a cloud identity.

Runtimes

Sources must be configured with a runtime. The runtime defines the networking configuration and where capabilities are executed.

Palantir allows you to use three different runtimes to connect to your systems. In general, if the system to which you're connecting can accept inbound connections from the network where your Foundry instance is hosted, you should use a direct connection runtime. If this is not possible, and the source type you are using supports an agent proxy, this is the preferred agent-based option. If neither of the other runtimes are available, you should fall back to using an agent worker.

Direct connection

Direct connections enable users to connect to data sources accessible over the Internet without needing to set up an agent. This is the preferred source connection method if the data source is accessible over the Internet; it avoids the operational overhead of setting up and maintaining agents and offers high uptime and performance. Learn how to set up a direct connection.

Agents

An agent is a piece of software provided by Palantir that runs on a host within your network. The agent connects to your source systems, and can also communicate with Foundry. An agent is required in order to use the agent proxy and agent worker runtimes. The same agent may be used as an agent proxy or agent worker, which is determined when using the agent with a particular source.

Learn more about how to set up an agent by following this tutorial.

Agent proxy runtime

The agent proxy runtime is used to connect to data sources not accessible over the Internet. The agent acts as an inverting network proxy, forwarding network traffic originating in Foundry into the network where the agent is deployed, and relaying traffic back to Foundry. This allows capabilities in Foundry to work almost exactly the same as when using a direct connection but without requiring you to allow inbound network traffic to your systems originating from Foundry's IP addresses.

For high availability, multiple agents can be configured with non-overlapping maintenance windows to ensure there is always an active agent to proxy connections to the target systems reachable through the agent proxy. Learn how to set up an agent proxy runtime.

Agent worker runtime

The agent worker runtime is used to connect to data sources not accessible over the Internet. An agent worker should only be used when the desired connector does not support the agent proxy runtime. Agent worker runtimes are associated with a single or multiple agents that store the source configuration and credentials locally in an encrypted format, and run source capabilities on the agent itself. Learn how to set up a source with an agent worker runtime.

Capabilities

Sources may support a variety of capabilities, where each capability represents some functionality that can run over the source connection. There are a wide range of supported capabilities for bringing data into Foundry, pushing data out of Foundry, virtualizing data stored outside of Foundry, and making interactive requests to other systems.

A summary of available capabilities is included in the following table. For more information about capabilities supported for a specific connector, refer to that connector's documentation page.

Additional capabilities are being developed, and capability coverage is regularly updated in the documentation for specific connectors.

Supported capabilities for specific connectors are also displayed on the new source page in the Data Connection application. It is possible to search both by connector name and by capability. The example below shows the results of a search for sources that support a "virtual" option.

Batch syncs

Batch syncs read data from an external system and write it into a Foundry dataset. A batch sync defines what data should be read and which dataset to output into in Foundry. Batch syncs can be configured to sync data incrementally and allow syncing data both with and without a corresponding schema. Learn how to set up a sync.

In general, there are two main types of batch syncs:

• File batch syncs: Allows syncing files without a schema directly into a Foundry dataset. These files may then be accessed in downstream transforms; for example, using file-based transforms. The most common systems that support file batch syncs are filesystems and blob stores such as S3, ABFS, Google Cloud Storage, SMB, and Sharepoint online. Learn more about file batch syncs.
• Table batch syncs: Allows syncing data with a schema into a Foundry dataset. Part of the sync definition in this case also includes how to translate types between the external system schema and the supported Foundry schema options. The most common systems that support table batch syncs are databases and SaaS providers such as Microsoft SQL Server, Postgres, SAP, Salesforce, and Netsuite.

Streaming syncs

Streaming syncs provide the ability to stream data from systems that provide low latency data feeds. Data is delivered into a streaming dataset. Some examples of systems that support streaming syncs include Kafka, Amazon Kinesis, and Google Pub/Sub.

Learn more about streaming syncs.

Change data capture syncs

Change data capture (CDC) syncs are similar to streaming syncs, with additional changelog metadata automatically propagated to the streaming dataset where data is delivered. This type of sync is normally used for databases that support some form of low-latency replication. Learn more about change data capture syncs.

Media syncs

Media syncs allow importing media data into a media set. Media sets provide better tooling than standard datasets for ingesting, transforming, and consuming media data throughout Foundry. When dealing with PDFs, images, videos, and other media, we recommend using media sets over datasets. Learn more about media syncs.

Hyperauto

HyperAuto is a specialized capability that can dynamically discover the schema of your SAP system and automate syncs, pipelines, and creation of a corresponding ontology within Foundry. HyperAuto is currently only supported for SAP. Learn more about HyperAuto.

File exports

File exports are the opposite of file batch syncs. When doing a file export, data is taken directly from the underlying files contained within a Foundry dataset, which are written as-is to a filesystem location in the target system. Learn more about file exports.

Table exports

Table exports are the opposite of table batch syncs. When performing a table export, data is exported as rows from a Foundry dataset with a schema, which are then written to a table in the target system. Learn more about table exports.

Streaming exports

Streaming exports are the opposite of streaming syncs. When doing a streaming export, data is exported from a Foundry stream, and records are written to the specified streaming queue or topic in the target system. Learn more about streaming exports.

Webhooks

Webhooks represent a request to a source system outside of Foundry. Webhook requests can be flexibly defined in Data Connection to enable a broad range of connections to external systems. Learn more about webhooks.

Virtual tables

Virtual tables represent the ability to register tabular data from an external system into a virtual table resource in Foundry.

In addition to registering individual virtual tables, this capability also allows for dynamic discovery and automatic registration of all tables found in an external system.

Learn more about virtual tables.

Virtual media

Virtual media works similarly to media syncs, allowing media from an external system to be used in a media set but without copying the data into Foundry. Instead, media files contained in an external system can be registered as virtual media items in a specific media set.

Learn more about virtual media.

Exploration

The interactive exploration capability allows you to see what data is contained in an external system before performing syncs, exports, or other capabilities that interact with that system.

Exploration is most commonly used to check that a connection is working as intended and that the correct permissions and credentials are being used to connect.

Use in code

The ability to use connections from code is intended to allow developers to extend and customize connections from Foundry to other systems. Palantir's general principle is that anything possible in the platform using dedicated connectors and point-and-click configuration options should also be achievable by writing custom code. At any point, developers should be able to switch to code-based connectivity for more granular control over the functionality or performance of workflows that perform external connections.

Any connector may be used in code; in most cases, we recommend using either the REST API source or generic connector when connecting from code.

Other concepts

Data connection also includes a variety of other concepts relevant for specific workflows. Some concepts were previously used and are now sunset, but are retained here for reference.

Syncs

Historically, the term Sync was used in a generic way to refer to bringing data into Foundry. Syncs are now separated into the more specific capabilities listed above. More details are available for each capability, such as batch syncs, streaming syncs, change data capture syncs, media syncs, and so on.

Tasks [Sunset]

The plugin framework used to implement connectors allows custom extensions called tasks. Tasks represent a unit of functionality configured by providing YAML and implemented in Java as part of the Data Connection plugin. Palantir has stopped developing new tasks, and all officially supported capabilities have migrated away from using tasks.

Tasks are currently considered Sunset according to the product life cycle. Tasks will eventually be fully deprecated, and we strongly recommend using code-based connectivity options anywhere tasks may have previously been required.