Running Contour workflows on top of data in Foundry requires the use of Foundry compute, a resource measured in compute-seconds. This documentation details how Contour uses compute and provides information about investigating and managing compute usage in the product.
Each Contour board displaying data is backed by a query to the Contour backend. Foundry serves Contour queries by executing them on an auto-scaling Spark cluster. Contour query compute-seconds are exclusively measured based on the compute-seconds a query actually uses while it is executing. However, in interactive mode, compute resources are “always on” for all Contour analyses to provide faster responses; this means that each individual interactive query uses compute-seconds at a faster rate than batch compute.
The amount of resources that a query uses depends on many factors, such as the size and layout of the input datasets and the complexity of the query (such as the Contour boards used).
Contour uses Foundry’s parallelized compute backend to execute queries on Foundry data. Compute in Contour is therefore measured with the following formula.
Note that the default interactive_contour_usage_rate is 15. This is the rate at which Contour uses compute based on the always-on parallelized compute framework. If you have an enterprise contract with Palantir, contact your Palantir representative before proceeding with compute usage calculations.
total_compute_seconds = contour_usage_rate * (driver_compute_seconds + executor_compute_seconds)
Contour’s “always ready” execution system is designed to be reactive to ad-hoc queries made by end users. The cost of a Contour query in Foundry compute-seconds is based on the time required for executing the query on the Spark cluster and the number of cores used while the query is executed. The number of cores is derived by looking at the queries’ share of the Spark application where the query was scheduled.
In interactive mode, compute is used each time a Contour board is created or updated. This includes “cascading” compute that affects boards downstream from a given board in a Contour path. For example, the following actions will use compute in a Contour analysis:
Contour paths can be “saved as” datasets. This saves the underlying code generated by the Contour backend into a dataset job specification. This dataset build can then be run on a schedule or ad hoc. These dataset jobs are measured as batch jobs and do not incur the “always on” overhead that interactive Contour usage does.
Contour batch datasets are measured as generic parallelized batch computation, similar to dataset created and updated via Code Repositories and Code Workbook. These batch computations use compute at the same rate as as other batch computation.
Interactive usage from Contour is always attached to the analysis that it is being triggered from. For a given Contour analysis, all interactive compute is attached to the analysis resource itself.
Individual boards that run compute produce Spark metrics that can be viewed from the Builds application. This can be accessed from the View jobs section on the context menu of each board. Batch compute usage from Contour is attached to the dataset that the compute produces, similar to Code Repositories and Code Workbook. For both interactive and Batch compute, you can view your overall usage in the Resource Management Application.
Contour runs on the Spark computation framework, and is therefore affected by two main factors: data scale and query complexity (the number and type of operations run on the data). The more operations that must be performed on the data, the more compute the backend must use to calculate the result. However, since some operations are much more expensive than others, it is possible to add operations and reduce overall usage (e.g. by adding a filter before an expensive operation).
The number of operations on data is affected by the number of boards in a Contour path. Boards farther down in a path rely on the output from boards earlier in the path. Adding new boards into a path can change the data being sent through the path, requiring re-computation of boards and more compute.
Having more Contour users may incur more compute due to an increase in queries. However, Contour has two features to reduce compute usage by many users: checkpoints and caching.
Optimizing the analysis and input datasets can reduce cost and analysis time. Learn more about Contour optimization.
Contour operates under the same Spark measurement principles as other Spark-based products in the platform, such as Code Repositories. With cost-based routing, Contour will automatically optimize the size of the backend drivers and executors in order to efficiently meet the demands of the data size and query type.
Example Contour Driver Size:
num_vcpu: 2
gib_ram: 11
Example Contour Executor Size:
num_vcpu: 2
gib_ram: 12
num_executors: 3
Query Duration: 3 seconds
Interactive Contour Usage Rate: 15
driver_compute_seconds = interactive_usage_rate * driver_compute_seconds
= 15 * max(2vcpu, 11gib / 7.5gib) * 3sec
= 15 * 2 * 3 = 90 compute-seconds
executor_compute_seconds = interactive_usage_rate * executor_compute_seconds
= 15 * max(2vcpu, 12gib / 7.5gib) * 3sec * 3executors
= 15 * 2 * 3 * 3 = 270 compute-seconds
total_compute_seconds = driver_compute_seconds + executor_compute_seconds
= 90 compute-seconds + 270 compute-seconds = 360 compute-seconds