Databricks

Connect Foundry to Databricks to read and sync data between Databricks and Foundry.

Supported capabilities

Capability	Status
Exploration	🟢 Generally available
Bulk import	🟢 Generally available
Incremental	🟢 Generally available
Virtual tables	🟡 Beta

Setup

Open the Data Connection application and select + New Source in the upper right corner of the screen.
Select Databricks from the available connector types.
Choose to use a direct connection over the Internet or to connect through an intermediary agent.
Follow the additional configuration prompts to continue the setup of your connector using the information in the sections below.

Learn more about setting up a connector in Foundry.

Configuration options

The following configuration options are available for the Databricks connector:

Option	Required?	Description
`Hostname`	Yes	The Databricks compute resource’s Server Hostname value.
`HTTP Path`	Yes	The Databricks compute resource’s HTTP Path value.

Please refer to the official Databricks documentation ↗ for information on how to obtain these values.

Authentication

You can authenticate with Databricks in the following ways:

Personal access token: Uses a personal access token to authenticate as a Databricks user. Learn more in the official Databricks documentation ↗.
OAuth machine-to-machine (M2M): Uses a client ID and secret to authenticate as a Databricks service principal. Learn more in the official Databricks documentation ↗.
Basic: Uses the username and password to authenticate as a Databricks user. Basic authentication is legacy and not recommended in production. Learn more in the official Databricks documentation ↗.

Networking

The Databricks connector requires network access to the Hostname provided in Configuration options on port 443. If you are using a direct connection through the internet, ensure that an egress policy exists. For agent runtimes, the server running the agent must have network access to this domain.

Virtual tables

This section provides additional details around using virtual tables with a Databricks source. This section is not applicable when syncing to Foundry datasets.

Virtual tables backed by Databricks external tables ↗ can also be configured using the Foundry connector that corresponds to their cloud object storage location. Currently DELTA, PARQUET and AVRO file formats are supported.

If the cloud object storage connector is configured to use Unity Catalog ↗, auto-registration can be used to register all external tables in Unity Catalog that are stored within the configured storage location. Currently Amazon S3 and Azure Blob Filesystem connectors support Unity Catalog.

Manual registration of individual tables in Unity Catalog is coming soon. Refer to the Virtual Tables documentation for more information on manual vs auto-registration of tables.

Virtual tables capability	Status
Source formats	🟡 Beta: tables, views, and materialized views
Manual registration	🟡 Beta
Automatic registration	🟡 Beta
Pushdown compute	🟡 Beta; available via the Databricks connector ↗
Incremental	🔴 Not available

When using virtual tables, remember the following source configuration requirements:

You must set up the source as a direct connection. Virtual tables do not support use of intermediary agents.
Ensure that bi-directional connectivity and allowlisting is established as described in the Networking section of this documentation.
If using virtual tables in Code Repositories, refer to the Virtual Tables documentation for details of additional source configuration required.
You must specify a warehouse in the connection details, using the HTTP path field. Refer to the official Databricks documentation ↗ for more information on how to get connection details for a Databricks compute resource.
The credentials provided must have usage privileges on the warehouse.

See the Connection Details section above for more details.