To cleanly uninstall an agent, refer to the documentation on reinstalling or upgrading the agent in the user interface. Before deleting the agent's directory, make sure to stop all related processes and copy any local settings, such as proxy configurations. Additionally, clear any cron jobs as outlined in the agent setup documentation.
Timestamp: February 13, 2024
The problem could be related to the incorrect interpretation of a column's datatype during the schema application in Foundry. To debug, download the Parquet files and use Python code to read the data with the applied schema. If the error messages mention columns names, you could exclude the problematic column from the schema to see if the rest load correctly.
Timestamp: February 13, 2024
To ensure that an S3 bucket only contains the latest exported data, you can use external transformations to call AWS APIs directly and implement custom logic for cleanup or pre/post-processing. This could involve deleting everything in the bucket before exporting, creating new directories, or moving things around. Additionally, you can also write a script to delete the contents of the S3 bucket before exporting from Foundry.
Timestamp: February 13, 2024
There is no parallelism between two agents. Syncs are scheduled on available healthy agents either randomly or based on whichever has fewer syncs in queue, which is configurable. Each agent can run a configurable number of syncs concurrently depending on allocated resources.
Timestamp: February 13, 2024
The solution is to reconfigure the source with the new details, which includes using the computer name in the URL.
Timestamp: February 13, 2024
The Service Now connector currently only supports batch syncs. To perform writes to Service Now, such as automatically creating tickets, you can build directly against their API using the REST API source type or external transforms.
Timestamp: February 14, 2024
No, Data Connection sources do not currently inherit Markings from agents.
Timestamp: February 13, 2024
Create an additional string column that is a string value of the timestamp, and perform incremental syncs on that string column instead of the original timestamp column.
Timestamp: February 20, 2024
No additional setup for AWS PrivateLink is required if both the Foundry instance and the customer's AWS VPC are in the same region, as AWS transfers data without exposing it to the internet.
Timestamp: February 13, 2024
The issue could be due to differences in the Java versions of the plugin and the bootvisor.
Timestamp: February 23, 2024
You will need to integrate with the Globus Python SDK using Python external transforms.
Timestamp: February 13, 2024
For JDBC exports, legacy export tasks using the JDBC connector are the only option available for now.
Timestamp: February 21, 2024
You must only use Palantir signed jars.
Timestamp: February 13, 2024
Convert datasets from Parquet to CSV in Foundry transforms, and then use file-based exports (Data Connection exports) to write the data to a file-based destination like S3 or streaming systems like Kafka.
Timestamp: February 13, 2024
Yes, it is possible to update the source type while keeping syncs intact. We recommend saving the existing configuration and reverting if something breaks. Additionally, try the update on a test source first before applying the changes to the actual source.
Timestamp: February 13, 2024
If soft-delete is enabled for the ABFS source, then you cannot use shared access signature and a blob SAS token to connect to ABFS. This is the allowed configuration from Azure.
Timestamp: April 16, 2024
Yes, if your tabular datasource has a JDBC driver, you can use the JDBC export task to export data.
Timestamp: April 25, 2024
You can connect to MS OneLake by using external transforms and leveraging the Python client provided by OneLake.
Timestamp: April 16, 2024
No, stored procedures on the database cannot be viewed or accessed directly on the Foundry side when connected through a data connector, but they can be executed via the "SQL Query" option when configuring a sync.
Timestamp: April 16, 2024
Agents need to download updated versions of bootstrapper / bootvisor / agent binaries and initial or updated versions of managed plugin binaries. Some of these are always downloaded, while others are only downloaded if a source of that type is assigned to the agent.
Timestamp: April 24, 2024
$start that is not showing up in the incremental section of the sync UI?Cast the binary type column $start to varchar(max) to avoid truncation and then use the column in the incremental section of the sync UI.
Timestamp: April 16, 2024
rewritePaths to rename files when exporting data to Azure, and why is it only exporting one file?You should use the new export functionality for file-based exports, which does not support rewritePaths. Instead, perform any necessary file renaming or data transformations upstream of the export process. This approach is recommended because legacy export tasks are more difficult to configure and debug.
Timestamp: April 16, 2024
To migrate an agent between two hosts, you should first shut down the agent properly on the old host using ./auto_restart.sh clear; ./init.sh stop to remove the cronjob and stop the bootstrapper. Then, copy the entire directory containing the agent to the new host using a tool like scp, assuming both hosts are up and can connect.
Timestamp: April 18, 2024
For importing data into Foundry, you should use syncs/extracts, which are supported for JDBC. For use cases where the SQL query changes dynamically, you should use external transforms to write the custom logic for data ingestion, rather than using it to change the sync configuration. This approach is preferred over data connection tasks, which are discouraged due to their limitations.
Timestamp: April 16, 2024
The recommended approach is to ingest the array-type columns as strings and then parse them in Pipeline Builder.
Timestamp: April 24, 2024
Owner permissions on the Project are required to create an agent.
Timestamp: April 16, 2024
The issue might be transient and can be fixed by refreshing the service a few times.
Timestamp: April 16, 2024
The SSL parameter needed for an Oracle JDBC driver connection is CONNECTION_PROPERTY_THIN_NET_ENCRYPTION_LEVEL.
Timestamp: May 23, 2024
The filter:
- type: sortByLastModified
order: DESCENDING
can be used to limit the number of files being ingested and guarantee the order in which the files are chosen.
Timestamp: April 24, 2024
AWS Fargate (Serverless ECS or EKS) is not recommended by Palantir as infrastructure for deploying the Data Connection Agents, primarily due to the lack of default volumes attached to them. Choosing to deploy agents in containers using these services is not officially supported.
Timestamp: September 5, 2024