Accelerate Spark with Velox

Spark acceleration is a technique that leverages low-level hardware optimizations to improve the performance of Spark jobs. By using platform-specific features, native acceleration aims to significantly reduce the time it takes to process large-scale data workloads, which can result in faster job execution and improved resource utilization.

Velox ↗ is a reusable, high-performance, low-level data processing library that provides a set of primitives for building high-performance data processing systems. It is designed to be used as a foundation for building higher-level data processing systems, and it is used in Foundry to accelerate Spark jobs.

Quick start

Spark acceleration can be used on any existing Spark pipeline. You do not need to make any changes to your logic.

To use native acceleration in your Python transform pipeline, you must complete the following:

1. Upgrade your Python repository to the latest version.
2. Configure an off-heap memory profile.
3. Enable the VELOX backend, as shown in the following code snippet:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
from transforms.api import configure, ComputeBackend, Input, Output, transform_df


@configure(
    ["EXECUTOR_MEMORY_MEDIUM", "EXECUTOR_MEMORY_OFFHEAP_FRACTION_HIGH"],
    backend=ComputeBackend.VELOX)
@transform_df(
    Output('/Project/folder/output'),
    source_df=Input('/Project/folder/input'),
)
def compute(source_df):
    ...

Coverage

The accelerator will selectively offload operators and expressions with implementations in Velox. User-defined functions (UDFs) and unsupported operators will continue to run in Spark. The coverage of Velox is continually expanding, so performance should improve over time.

View the full list of supported operators and expressions ↗

Configure memory for accelerated Spark

To optimize your natively accelerated Spark project, start by using the EXECUTOR_MEMORY_OFFHEAP_FRACTION_HIGH setting for off-heap memory. This memory is used by Velox, which handles some tasks outside the JVM. Observe the performance, and adjust the off-heap memory up or down as needed.