Advanced configuration

This section describes how advanced configuration options can be used in Java transforms.

Maximum build duration

It may be desirable to limit the run duration of a job to ensure data freshness or to limit costs. For example, if a job is interacting with an external service and becomes unresponsive, it is useful to have a limit on its run duration, as it may not complete. In Code Repositories, you can limit job duration by using the MaxAllowedDuration and Compute decorators, as shown below:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
package myproject.datasets;

import com.palantir.transforms.lang.java.api.*;
import java.time.Duration;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;

public final class FilterTransform {

   @MaxAllowedDuration(value = "PT2H")
   @Compute
   public void myComputeFunction(
           @Input("/examples/students_hair_eye_color") FoundryInput myInput,
           @Output("/examples/students_hair_eye_color_filtered") FoundryOutput myOutput) {
       Dataset<Row> inputDf = myInput.asDataFrame().read();
       myOutput.getDataFrameWriter(inputDf.filter("eye = 'Brown'")).write();
   }
}