Evaluations

An evaluation is a collection of metrics, images, plots, and tables that captures how a specific model version performs against a given set of test data. Evaluations are written to an evaluation set associated with the model, where they can be visualized and compared on the model page.

Integrate evaluations into model training code

Evaluations are created in Code Repositories. The @pm.transforms.evaluation decorator wires up an evaluation set for each ModelInput in the transform, and the ModelInput then provides a hook for creating and writing to an evaluation. When the build succeeds, the evaluation is automatically committed to the evaluation set; if the build fails, it is aborted.

The below code snippet demonstrates evaluation usage in Code Repositories:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
from transforms.api import transform, Input
import palantir_models as pm
from palantir_models.transforms import ModelInput

# The decorator indicates the evaluation set that will be written to on the model input(s).
@pm.transforms.evaluation("error_analysis")
@transform.using(
    input_data=Input("..."),
    model_input=ModelInput("..."),
)
def compute(input_data, model_input):
    evaluation = model_input.create_evaluation(name="my-evaluation")

    # log metrics
    evaluation.log_metric("mae", 14250.0)
    evaluation.log_metrics({"mae": 14250.0, "r2": 0.91})

    # log an image, plot, or table
    evaluation.log_image("residual_distribution", pillow_image)
    evaluation.log_plot("residuals_by_tier", fig)
    evaluation.log_table("error_by_price_tier", error_by_tier)

Learn more about creating and visualizing evaluations.