Microsoft SQL Server

Connect Foundry to Microsoft SQL Server to read and sync data between SQL Server databases and Foundry.

Supported capabilities

CapabilityStatus
Exploration🟢 Generally available
Batch syncs🟢 Generally available
Incremental🟢 Generally available
Change data capture syncs🟢 Generally available

Setup

  1. Open the Data Connection application and select + New Source in the upper right corner of the screen.
  2. Select MS SQL Server from the available connector types.
  3. Choose to use a direct connection over the Internet or to connect through an agent runtime.
  4. Follow the additional configuration prompts to continue the setup of your connector using the information in the sections below.

Learn more about setting up a connector in Foundry.

Authentication

You can authenticate with SQL Server in the following ways:

  1. Username and password: Provide a username and password. We recommend the use of service credentials rather than individual user credentials.
  2. Active Directory Msi*: This option will use the authentication=ActiveDirectoryMSI JDBC setting. An msiClientId may optionally be provided.
  3. Active Directory Password*: This option will use the authentication=ActiveDirectoryPassword JDBC setting. A username and password for an Active Directory user must be provided to use this setting.
  4. Active Directory Service Principal*: This option will use the authentication=ActiveDirectoryServicePrincipal JDBC setting. A principal ID (sometimes referred to as an application or client ID) must be specified, along with a secret for that principal ID.

For more information on these authentication modes, see the official documentation ↗. For all authentication options, ensure that the provided user and role has the necessary privileges on the target database, as well as permission to read from or write to the target table(s).

* Note that Azure Active Directory is now called Microsoft Entra ID ↗; however, the JDBC options on the SQL Server driver published by Microsoft retain the original names referring to Active Directory.

Networking

The Microsoft SQL Server connector requires network access to the SQL Server instance that you wish to connect to. SQL Server connections will normally use a hostname to connect on port 1433.

To enable direct connections out of Foundry to SQL Server, the appropriate egress policies must be added when setting up the source in the Data Connection application.

For SQL Server instances hosted on a cloud service like Azure SQL or AWS RDS, you must add an egress policy for the hostname retrieved from your cloud provider’s console.

If Azure redirect mode is used, then egress policies for all resolved IP addresses must also be added. The resolved IP may occasionally change, and you must update the egress policies to allow the new IP. If your instance of SQL Server is hosted on Azure, for example, then you can find more information on public IP addresses for Azure SQL instances in the Azure SQL documentation ↗.

  • To find the hostname for your Azure SQL instance, navigate to the Settings > Properties page in the Azure portal, and look for the Server Name field. An example hostname-based egress policy for Azure SQL: <your-database-name>.database.windows.net (port 1433)
  • To find the resolved IP, you can run nslookup <your-database-name>.database.windows.net from the command line. The final result will be the IP address that this hostname resolves to in Azure. The following is an example IPv4-based egress policy for Azure SQL: x.x.x.x (port 1433). Azure SQL does load balancing across multiple hosts, so you may need to run the nslookup command several times and add all of the resolved IP addresses.

If you are connecting to an Azure SQL instance from a Foundry instance also hosted in Azure, you will need to use the Proxy connection policy option. For traffic originating within Azure, the connection policy defaults to Redirect. Using the redirect option to connection for Azure-Azure connections would require configuring egress policies for all Azure SQL IP addresses on all ports in the range of 11000 to 11999. This is possible but not recommended as it is overly permissive. For details on Azure SQL connection policies, see the official Azure SQL documentation ↗.

If you are connecting with an agent runtime, you must ensure that the agent host has firewalls open to the host names, IP addresses, and ports required to connect to your MS SQL Server database.

Connection details

OptionRequired?Description
Host typeYesSpecify how Foundry should connect with your SQL Server database.

Option 1: Hostname
Provide a hostname. This is the recommended option for all SQL Server connections, and should always be used when connecting to an Azure SQL ↗ instance.

Option 2: IPv4
Provide an IPv4 address. If you normally connect using an IPv4 address, either within a corporate network or over the Internet, you can use this option.

Option 3: IPv6
Provide an IPv6 address. Use this option if you normally connect using an IPv6 address.
PortYesSpecify a port to use when connecting. The default port for most SQL Server instances will be 1433. For more information on ports, see the official documentation for the version of SQL Server you are connecting to.
Database nameYesThe name of the database you're connecting to within your instance of MS SQL Server.
AuthenticationYesConfigure using the Authentication guidance shown above.
Require encryptionYesDefaults to enabled. For more details, see Microsoft's documentation for the encrypt setting on the SQL Server JDBC driver:

Connection properties reference ↗
Encryption support examples ↗
Trust server certificateYesDefaults to disabled. For more details, see Microsoft's documentation for the trustServerCertificate setting on the SQL Server JDBC driver:

Connection properties reference ↗
Network ConnectivityYesYou must provide a runtime that is networked to your MS SQL Server instance. For instances behind a corporate firewall, you will normally need to use an agent runtime. For cloud-hosted instances, refer to the Networking section for more details.

Change data capture

The Microsoft SQL Server source supports change data capture syncs.

To enable change data capture for Microsoft SQL Server, you must run a command like the one below to enable CDC on the database.

USE <database>
GO
EXEC sys.sp_cdc_enable_db
GO

Then, run another command on each table that should be recording changelogs:

EXEC sys.sp_cdc_enable_table
    @source_schema = N'<schema>'
  , @source_name = N'<table_name>'
  , @role_name = NULL
  , @capture_instance = NULL
  , @supports_net_changes = 0
  , @filegroup_name = N'PRIMARY';
GO

Once change data capture is enabled for the table(s) you wish to sync to Foundry, you can navigate to the Overview page and select + Create CDC sync to start creating a new change data capture sync.

The exploration runtime must be working in order to create a change data capture sync. If the runtime is still initializing, you may need to wait a few seconds and refresh the page to proceed with creating a change data capture sync.

For more information on these commands and using change data capture (CDC) with Microsoft SQL Server, see the official documentation ↗ for the version of SQL Server in use.