Spark UI [Beta]
Viewing the Spark UI in Foundry is a Beta feature and not be available on all enrollments. Contact Palantir Support to install the Spark UI. Currently, only Transforms jobs with containerized infrastructure are supported for viewing.
Spark has its own Web UI ↗ which complements Foundry's Spark details page with additional information, including:
- Executor lifecycle information, such as executor launch and shutdown.
- Larger samples of task and executor metrics, including peak memory usage.
- All Spark configs used during execution.
Viewing Spark UI
To view the Spark UI for a Transforms job, re-run the job as a debug job. You will see a Spark UI button; selecting this will open Spark's Web UI.
Spark events appear in the Spark UI after a delay of 1-2 minutes.
Spark UI in Foundry usage
Spark's Web UI is rich in detail but does not present information in a manner tailored for Foundry. Below, we provide advice on navigating Spark's Web UI for Foundry jobs.
SQL execution
Spark can break up SQL queries into a main query and one or more subqueries. In some cases, a subquery is more interesting than the main query. This is true for many dataset writes in Foundry.
When viewing a "Writing dataset ..." SQL execution in the Spark UI, you can find the query graph for the write linked under Sub Execution IDs.
Context warming
The Jobs tab in the Spark UI shows that Transforms jobs trigger an initial count job. The purpose of the count job is to request executor allocations early, while the runtime performs additional setup (including installing dependencies). This increases the likelihood of executors being available by the time the Transform is ready to run.
