Agent configuration reference

The following sections describe the various configurations available when creating, monitoring, and managing agents within your network. The following workflows will be explained:

• Agent configuration reference
  • Agent permissions
  • Proxy configuration
    • Agent Manager proxy configuration
    • Bootstrapper proxy configuration
    • Source proxy configuration
  • Agent metrics and health monitoring
    • Metrics
      • "Time until next expiration" metrics
    • Health monitoring
  • Automatic upgrade windows
    • Upgrade windows
    • Stagger upgrades
    • Schedule dataset syncs
  • Move an agent to a new directory
  • Move an agent to a new host
  • Install on a Windows host

Learn more about agents and how to set them up in the Data Connection application.

Agent permissions

Permission to create agents is administered in Control Panel. To create an agent, you must have either the Organization administrator or Data flows administrator role assigned to you. It is also possible to create a custom role and assign the Create agent workflow to that role. Organization roles are managed on the Organization permissions page in Control Panel.

Learn more about managing roles in Control Panel.

In addition to the Organization-level role, you must be an Editor or Owner of the Project in which you want to save a newly-created agent.

Certificates

Agents communicate with both Foundry and your internal network. This means that agents need to have the correct certificates in their truststores for these connections to be established.

There are two situations that may require additional certificates to be configured on an agent:

• Certificates to allow agent communication with Foundry
• Certificates to allow agent communication with your systems

In both cases, certificates may be added from the agent management page in the Data Connection application.

Add certificates to an agent

To add a certificate to an agent, navigate to the agent details page for your agent in Data Connection, then select Manage agent certificates.

From here, choose the certificate to be added, along with an alias if desired. Then, add the contents of the certificate. The certificate should be added as a string similar to the below example, including newlines but without a trailing newline character. The certificate shown below is the public certificate for https://palantir.com.

Note that you cannot enter a certificate chain; you must enter each certificate separately.

-----BEGIN CERTIFICATE-----
MIIGXjCCBUagAwIBAgIQASByQ6gv8Z6X7wEqsyBb1DANBgkqhkiG9w0BAQsFADBY
MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEuMCwGA1UE
AxMlR2xvYmFsU2lnbiBBdGxhcyBSMyBEViBUTFMgQ0EgMjAyNCBRMjAeFw0yNDA2
MTcxNjUwMTVaFw0yNTA3MTkxNjUwMTRaMBkxFzAVBgNVBAMMDioucGFsYW50aXIu
Y29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvBalUG3JFYaSiRO2
enRnEwdeGUgtal9isJfB1++LxcPwo/DP2dncK+ur7URID0TVWOqu+4vXE2mmC9jz
Kx0o/URrMoz70i6qF6/Oyq6CuOHjZINiAN0ovNBBEPNGbSVD3Xq/eWgI7PNQ8hfI
9BJ/3WVA17oSG3zEXiWP3+CiL3Wm1Gn38oOt4URBMA0hgLqyOoU3ooqYIK8Fz2K/
OxAJvq45z2lonMZFFzj5thO5dBBch26mNAacO4MvI9mhUrMZtYvGBRZoXrph4EmF
TJDo2UTYiST0Tq6ibNW+NTuv66DrqFvzOpZybNuZsS6VrisYQ4huPN9jVz7RNFhJ
aeJvbQIDAQABo4IDYTCCA10wGQYDVR0RBBIwEIIOKi5wYWxhbnRpci5jb20wDgYD
VR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNV
HQ4EFgQUuAbgKrz0fIAXR1I/89IpUMN57AgwVwYDVR0gBFAwTjAIBgZngQwBAgEw
QgYKKwYBBAGgMgoBAzA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxz
aWduLmNvbS9yZXBvc2l0b3J5LzAMBgNVHRMBAf8EAjAAMIGeBggrBgEFBQcBAQSB
kTCBjjBABggrBgEFBQcwAYY0aHR0cDovL29jc3AuZ2xvYmFsc2lnbi5jb20vY2Ev
Z3NhdGxhc3IzZHZ0bHNjYTIwMjRxMjBKBggrBgEFBQcwAoY+aHR0cDovL3NlY3Vy
ZS5nbG9iYWxzaWduLmNvbS9jYWNlcnQvZ3NhdGxhc3IzZHZ0bHNjYTIwMjRxMi5j
cnQwHwYDVR0jBBgwFoAUrw0C0MMbnlj47zdiLecDXZ5BSoowSAYDVR0fBEEwPzA9
oDugOYY3aHR0cDovL2NybC5nbG9iYWxzaWduLmNvbS9jYS9nc2F0bGFzcjNkdnRs
c2NhMjAyNHEyLmNybDCCAX0GCisGAQQB1nkCBAIEggFtBIIBaQFnAHUA5tIxY0B3
jMEQQQbXcbnOwdJA9paEhvu6hzId/R43jlAAAAGQJxs/FAAABAMARjBEAiATvomb
hMrUty8Vj703fTBSzY5qDxEMI473IigiGIiXugIgbj4/j24jloGdVoedM3jb6DFw
yAXkuZD3SMHBLsEvP9gAdgDd3Mo0ldfhFgXnlTL6x5/4PRxQ39sAOhQSdgosrLvI
KgAAAZAnGz7qAAAEAwBHMEUCIAHXbm9F2rwyxD36aHoGZRrnDtgg9UDRy5UtHK6D
OrmKAiEAjfomH4CGUrkBbwD8pzt9BbC6u6gPPveYiURxFIq//RUAdgB9WR4S4Xgq
exxhZ3xe/fjQh1wUoE6VnrkDL9kOjC55uAAAAZAnG0C6AAAEAwBHMEUCIQDDOg9s
KZqzbCu0mNBQMRAv6/2HkuLjZSGMxjq/F0R1/wIgKMHBSsNgeVED+LpTcIBYgp1q
SXgbwSizE6OD+1Ewol0wDQYJKoZIhvcNAQELBQADggEBAAr/tnc9dtTfwrczP7Ok
1+tLKmFRss4/1KQgLY8Tyy45Pag53ikn2n3tSPG1OpRTSmfPhPs9/UQRtMf7f2Gk
ObSXDVpPArtFBFDfZug+j22gVSYQr6zgFJu4Y9QD1GGtICqkmTScubfnjwdffTv6
5oNY4LbVGp5yctAd80OFUXspy+oVGsvv61a1pFO+s/NXleSrqDGL1oWcFW5Uj8GH
jnTM+Lt/HupqZ/ThVSkjMOug+hB875Yf8mWvadKYBX0Ga2s51cp8CI49FRswziY6
3oXPKfHHQybpIKhGosuSyzY8pL8UofHNp8gicAV80Vj6Mw+L8gWaAkCR6YnzQIyJ
9lc=
-----END CERTIFICATE-----

Certificates to allow agent communication with Foundry

Certificates for connecting to your Foundry instance are included in the default agent install bundle. Generally, this means that no additional certificates will be required for your agent host to communicate with Foundry.

If traffic from your agent to Foundry is intermediated with an explicit or transparent proxy, certificates for the proxy may need to be added to the agent as part of initial setup. Since missing certificates in this scenario will prevent the agent from being managed by Foundry, these certificates must be added to the agent before downloading the initial install bundle.

If there are certificate-related errors on agent startup that prevent the agent from communicating with Foundry, these will appear in a logfile called <bootvisor_directory>/var/service.log.

If you encounter an error when attempting to set up your agent for the first time or know in advance that additional certificates will be required, you should:

1. Skip through the agent setup page without attempting to download and install on your agent host.
2. Add necessary certificates from the agent management page as described in the section above, then download and follow the standard installation instructions.

If your additional certificate is for an explicit proxy, additional configuration may be required.

Once an agent successfully connects to Foundry, these certificates should not need to be changed unless your network configuration between the agent host and Foundry changes.

Certificates to allow agent communication with your systems

When using an agent as an agent worker runtime you may need to add certificates to allow connectivity to specific target systems. This is described in more detail in the agent worker configuration reference documentation.

Agents in TLS-inspected environments

If you perform TLS inspection on traffic on the host where your agent is installed, you will need to manually import your root CA to the Agent Manager's truststore before the Agent Manager can connect to Foundry for the first time.

To do this, follow the instructions below:

1. Obtain the public certificate for the root CA used to re-sign inspected traffic. Your network team should be able to provide this. Alternatively, you can use OpenSSL on the agent's host to inspect the certificate chain and find the root certificate: openssl s_client -showcerts -connect FOUNDRY_URL:443.
2. Use SSH to access the agent's host and place the certificate somewhere accessible to the user that runs the Agent Manager. One such location may be <agent-manager-install-directory>/var/security.
3. Navigate to the Agent Manager's Java installation: cd <agent-manager-install-directory>/jdk/amazon-corretto*.
4. Import the certificate to the Agent Manager's truststore: ./bin/keytool -import -alias <CERT_NAME> -file /path/to/certificate.pem -keystore ../../var/security/truststore.jks (when prompted, the password is truststore).
5. Start the Agent Manager. You should see Connected to Agent Manager, but no connection from agent on the agent detail page in Data Connection, confirming that the Agent Manager has successfully connected to Foundry.
6. To allow the agent to connect to Foundry, follow the instructions above to add the same certificate to the agent.

Proxy configuration

When configuring your agent for the first time, or when connecting to a remote source, you may need to configure a proxy depending on your organization's network configuration. A proxy may be used to manage communication from the agent to Foundry, or it may be needed to reach data sources within your network.

For your agent to use a proxy, you will need to configure the proxy in both the Agent and Bootstrapper configurations found in the Advanced tab of the Manage Configuration window in the agent configuration page within Data Connection. Make sure to provide the hostname without http:// at the beginning.

Agent Manager proxy configuration

The Agent Manager proxy configuration must be added to the <agent-manager-install-directory>/var/conf/runtime.yml file on the host where the agent was installed. Below is an example Agent Manager configuration snippet with a proxy-configuration block:

Copied!
1
2
3
4
5
6
7
8
9
service-discovery:
    services:
    magritte-coordinator:
        ...
        proxy-configuration:
            hostAndPort: proxy-host.com:3128
            credentials: # these are optional
                username: USERNAME
                password: PASSWORD

This proxy will be used by the Agent Manager to connect back to Foundry's magritte-coordinator. It will not be used for connections from the agent to sources.

Bootstrapper proxy configuration

Once you have configured the Agent Manager proxy you can then configure the bootstrapper proxy. To do this, navigate to the agent configuration page within Data Connection, toggle the Advanced option in the Manage Configuration section, and finally select the Bootstrapper tab. Below is an example bootstrapper configuration snippet with a proxyConfiguration block:

Copied!
1
2
3
4
5
6
7
coordinator:
    proxyConfiguration:
        host: HOST
        port: PORT
        credentials: # these are optional
            username: USERNAME
            password: PASSWORD

Once you have updated the configuration, you must save your changes and restart the agent for them to take effect.

This proxy will be used by the Bootstrapper for connecting back to Foundry's magritte-coordinator. It will not be used for connections from the agent to sources.

Source proxy configuration

For connecting from an agent to a data source through a proxy, configure the agent's JVM-level proxy on the Bootstrapper from the Bootstrapper tab in the Advanced section of the Manage Configuration page.

Use these JVM flags:

Copied!
1
2
3
4
agent:
  jvmArguments: >-
    -Dhttp.proxyHost=<PROXY URL> -Dhttp.proxyPort=<PROXY PORT>
    -Dhttps.proxyHost=<PROXY URL> -Dhttps.proxyPort=<PROXY PORT>

Agent metrics and health monitoring

Once you set up an agent in Data Connection, you can view metrics and monitor health to maintain performance.

Metrics

Navigate to the agent page in Data Connection, then select the Metrics tab. The metrics available for your agent include, but may not be limited to, the following:

• OS physical memory
• Disk space
• Usable disk space
• OS load normalized WRT cores
• CPU utilization
• Time until next expiration in agent keystore
• Time until next expiration in agent truststore
• Agent heap memory
• Percentage of heap used
• Agent uptime
• Agent threads
• Syncs/tasks upload since last agent restart
• Running syncs/tasks
• Agent syncs/tasks queued
• Syncs/tasks duration
• Agent last heartbeat time
• Agent Manager version stale time
• Agent version stale time

Hover over the metric cards for timestamped details, and select the top right corner of the card to expand a detailed view.

"Time until next expiration" metrics

The Time until next expiration in agent keystore and Time until next expiration in agent truststore metrics refer to the time until the earliest certificate expiration in the agent keystore and truststore, respectively. For example, if the agent's keystore has two certificates, one that expires in a week and one that expires in a month, the number would be 1w as that is the closest expiration date.

The agent keystore and truststore include certificates added by users as well as those automatically added by the Agent Manager. Agent Manager certificates are automatically upgraded.

If the certificate has already expired, the metric will show 000ms. If there are no certificates stored for the agent, the graph will be empty.

"Version stale time" metrics

The Agent Manager version stale time and Agent version stale time metrics refer to how out of date the agent and agent manager are relative to what is available on your environment.

The version out of date metrics are calculated as the number of days between when the agent or agent manager was last updated and the release date of the latest version available. The below example illustrates how these metrics and associated monitors are expected to behave:

In this example, the agent version stale time metric on the first two days is 0. The metric then jumps to 2 when the new version becomes available, and will then continue to increase until the next agent maintenance window, and finally drop back to 0 once an update completes successfully during a maintenance window.

Health monitoring

Health monitors allow you to configure alerts of varying severities (high, medium, or low) for any metric when certain conditions or thresholds are met.

You can monitor an agent's health by creating a monitoring view in the Data Health application. A monitoring view is a collection of monitoring rules that are of particular interest to a subscribed group of users.

You can view existing monitoring views by selecting the Monitoring views tab.

After selecting a specific monitoring view, you can configure the health monitors of an agent by selecting Manage monitors. From this page, you can create a new monitoring rule.

In the Create monitoring rules page, you can configure specific rules and alerts of varying severities.

Learn more about tracking data health with monitors and integrating monitors with PagerDuty.

Automatic upgrade windows

The Data Connection agent service is regularly updated with security, stability, and performance improvements. The best way to ensure that agents receive these important improvements in a timely manner is to configure upgrade windows for each agent in use. The sections below describe what happens during an upgrade window and provide best practice guidelines.

Upgrade windows

An agent upgrade window is a set of time intervals during which it is considered safe for the agent to be temporarily offline. These time intervals recur weekly and can be defined on the Maintenance Window page in the Agent settings tab for the given agent in the Data Connection application.

The Data Connection coordinator monitors agents and their respective upgrade windows; they will perform automatic upgrades of agents during these upgrade windows when new versions are available.

As part of the upgrade, the agent will be restarted. This will terminate any running jobs and briefly prevent new jobs from running on the agent.

Agent upgrade windows should be at least 60 minutes long. However, the actual upgrade should be relatively short; it should take approximately the same amount of time as a restart of the agent.

Stagger upgrades

To ensure minimal impact to data pipelines, we recommend assigning at least two agents to all Data Connection sources and to stagger upgrade windows for any given set of sources running on those agents. For example, one agent could have an upgrade window defined on Sundays, while the other has an upgrade window scheduled for Wednesdays. This ensures that during any given agent's upgrade window, jobs that are interrupted can be retried on the partner agent, and new jobs can continue to queue and run until the agent being upgraded is fully back online.

Schedule dataset syncs

When staggered upgrade windows cannot be used, it is important to schedule upgrade windows during periods of low (ideally zero) activity. In this case, dataset syncs should be scheduled such that they finish before the start of the upgrade window or start several minutes after the window (to account for the restart occurring towards the end of the window).

Move an agent to a new directory

Follow the steps below to move an agent to a new installation directory for the same machine.

1. In Data Connection, navigate to the Syncs page and ensure that no syncs are currently running.
   • Check if syncs are running by using the filter on the left of the page to view only those that currently have a Running status.
   • If syncs are running, either wait for them to complete or cancel them by selecting their status link (for example, Running 14 minutes ago), and then choosing Cancel build from the build page. Ensure that sync owners are appropriately notified if you need to cancel their syncs.
2. Navigate to the Agents page. Select the name of the agent you want to move.
3. In the Configuration panel, select Advanced. For each tab of the Advanced settings, change all references that use an absolute path. Look for anything that starts with / and modify these to what the new path will be.
   • Note: Kerberos settings require an absolute path.
4. Stop the agent in Foundry. To do so, select the dropdown arrow next to Restart Agent in the upper right corner of the screen, then select Stop (Unsafe).
   • Note: The Unsafe label is meant to warn that stopping the agent will interrupt any running syncs, which is why we took the precautions detailed in Step 1.
5. On the terminal of your machine, SSH into the agent.
6. Change to the administrative user for the agent. Depending on your configurations, this could require entering -- sudo -su palantir or -- sudo -su admin.
7. In your terminal, navigate to the agent installation directory using cd.
8. Stop the agent by running ./service/bin/init.sh stop.
9. If the agent was previously configured to autostart:
   1. For Linux, run ./service/bin/auto_restart.sh clear.
   2. For Windows, remove any scheduled tasks that were set up while following instructions on installing on a Windows host.
10. Wait a few minutes and check that there are no Java processes being run by the administrative user. If there are, stop them manually.
11. Optionally, remove the contents of ./var/data/binaries, ./var/data/cache, and ./var/data/processes folders to reduce the amount of data being transferred.
12. Move the agent to the new directory folder by entering mv <source directory> <new installation directory>.
13. Navigate to the new agent installation directory. Restart the agent by running ./service/bin/init.sh start.
14. If the agent was previously configured to auto-restart, re-enable it by following the auto-restart setup instructions.
15. Wait five minutes, then refresh the agent page in Foundry to ensure that the agent is connected and syncs are successful.

Move an agent to a new host

When moving an agent to a new host, make sure that the new host meets the operating system requirements and has the same operating system and architecture as the previous agent. The new host must also have the same firewall network configurations as the old host. For example, if the agent was previously running on a linux distribution running on an x86_64 architecture, the new host can have any distribution of linux, but must be running on the same architecture. The instructions below will not work to move an agent from Linux to Windows or to an ARM architecture. For these cases, get a new download link from the UI and proceed with a new install.

When re-installing the agent using a new download link from the UI, existing source credentials will not be able to be decrypted and must be re-entered manually. To preserve these credentials, consider the following steps to migrate the source secrets from the old agent to the new one:

1. Navigate to the directory <bootvisor dir>/var/data on the old agent.
2. Copy the files source-encryption-key and source-encryption-key.private to the same directory (<bootvisor dir>/var/data) on the new agent.
   • If these files are lost or overwritten on the old agent before copying them, all source credentials will be lost, and they will need to be manually re-added to the new agent.

If the agent relies on other files in the file system, ensure they are also moved to the new agent installation to maintain proper functionality.

Follow the steps below to move an agent to a new host:

1. Follow steps 1 through 11, excluding step 3 in the instructions for moving an agent to a new directory. This will stop the agent and prepare it to be transferred to a new host.
2. Use rsync or any other file transfer tool to copy the agent folder from the old host to a location on the new host.
3. Follow previously skipped step 3 from the instructions on moving an agent to a new directory. For each file or folder referenced by an absolute path, copy that resource to a location on the new host and update the configuration to point to the new path.
4. Follow steps 13 through 16 from the moving an agent to a new directory guide.

Install on a Windows host

Follow the steps in the guide to setting up a agent, but select Windows as your operating system on the download and configure agent step. Once you install your agent, you will need to set up a Windows scheduled task that restarts the Agent Manager on crash or machine restart, as follows:

1. Open Windows Task Scheduler as an Administrator.
2. Create a new scheduled task.
3. Under the General tab, perform the following:
   1. Set When running the task, use the following user account to SYSTEM.
   2. Make sure Run whether the user is logged in or not is selected.
   3. Make sure Do not store password... is NOT selected.
   4. Update Configure for to the correct operating system for the Windows virtual machine.
   5. Select Run with highest privileges.
4. Under the Triggers tab, create two triggers:
   1. Set up one trigger on an event and choose on startup.
      • Ensure the task is Enabled.
   2. Set up another trigger on a schedule:
      • Set the task to run daily.
      • Check the Repeat task every box, and choose 5 minutes for a duration of 1 day.
5. Under the Actions tab:
   1. Add a new Start a Program action.
   2. Copy the whole path to the magritte-bootvisor-win batch file in the box for Program/script.
      • The full path will be similar to C:\example\path\to\folder\containing\magritte\agentmagritte-bootvisor-win-{version}\service\bin\magritte-bootvisor-win.bat.
   3. Copy the whole path to the enclosing folder for the batch script into the box for Start in (optional).
      • The full path will be similar to C:\example\path\to\folder\containing\magritte\agent\magritte-bootvisor-win-{version}.
   4. There are no arguments, so leave Add arguments (optional) blank.
6. Select OK to save your task.
7. Attempt to manually run the task by right-clicking on the task and choosing Run.
8. You should see the agent start up successfully. This can take a minute or two to reflect in the interface in Foundry. If the agent does not start successfully, check the History tab of the scheduled task for errors. You can also verify the operation's success by using the Windows command prompt to change directory into the specific folder and then executing the associated .bat file.

Backups and caches

Agents may backup or cache files as part of installation, version upgrade, version downgrade, and ongoing usage. Normally, backups and cache may be safely ignored, and will be automatically deleted after 30 days. If backups and cached files are using up a higher than desired amount of disk space on the agent host, it is safe to delete them as long as this is performed outside of a maintenance window.

Backup and cache files will be found in the following sub-directories within the agent installation directory: