Experiments

An experiment is an artifact that represents a collection of metrics produced during a model training job. Experiments allow developers to log hyperparameters and metrics during a training job, visualize them on the model page, and compare between different model versions.

Why use experiments?

The model development process is inherently iterative, and it can be difficult to keep track of different attempts at producing a model. Experiments provide a lightweight python API for logging details related to those different attempts, including metrics and hyperparameters. Those metrics and hyperparameters can be visualized and compared between different model versions to provide a better understanding of how different parameters affect the model's performance. Below is an overview of how to create and write to experiments.

Create experiments

The ModelOutput class used to publish models from Jupyter® Code Workspaces and Code Repositories provides hooks for creating experiments.

Create experiments in Code Workspaces:

Copied!
1
2
3
4
from palantir_models.code_workspaces import ModelOutput
# `my-alias` is an alias to a model in the current workspace
model_output = ModelOutput("my-alias")
experiment = model_output.create_experiment(name="my-experiment")

Create experiments in Code Repositories:

Copied!
1
2
3
4
5
6
7
8
9
from transforms.api import configure, transform, Input
from palantir_models.transforms import ModelOutput

@transform(
    input_data=Input("..."),
    model_output=ModelOutput("..."),
)
def compute(input_data, model_output):
    experiment = model_output.create_experiment(name="my-experiment")

If any two experiments for a given model use the same name, they will be automatically deduplicated, allowing for the same code to be used multiple times without worrying about renaming the experiment.

Occasionally, model training code may fail due to network errors when writing to the experiment, or overflowing the maximum size of the series. While the Python client aims to handle these errors as gracefully as possible, there may be times where that is not possible. Clients can choose how errors should be handled, selecting from three different error handling variants:

• FAIL - Will instantly re-raise the error and the code will fail.
• WARN (default) - Will log a warning for the error, then suppress all future errors.
• SUPPRESS - Will not log anything.

The error handler mode can be set during experiment creation as shown below:

Copied!
1
2
from palantir_models.experiments import ErrorHandlerType
experiment = model_output.create_experiment(name="my-experiment", error_handler_type=ErrorHandlerType.FAIL)

Publish experiments

In order for experiments to be displayed in the model page, they must be published alongside a model version. Once published, experiments can be viewed in the model page.

Copied!
1
model_output.publish(model_adapter, experiment=experiment)

Learn more about visualizing experiments after publishing.

Log hyperparameters

Hyperparameters can be logged using the Experiment.log_param and Experiment.log_params functions. Hyperparameters are single key-value pairs that are used for storing static data associated with a model training job.

Copied!
1
2
3
4
5
6
experiment.log_param("learning_rate", 1e-3)
experiment.log_param("model_type", "CNN")
experiment.log_params({
    "batch_size": 12,
    "parallel": True
})

Experiments currently support logging hyperparameters of the following types:

• Boolean
• Date/Datetime
• Float
• Int
• String

Log metrics

Metrics can be logged using the Experiment.log_metric and Experiment.log_metrics functions. Metrics are logged to a series, where the series tracks each logged value in a time series.

When logging metrics, if the metric series has not been created, a new series will be created. Additionally, callers may pass a step parameter to set the step to log to.

Copied!
1
2
3
4
5
6
experiment.log_metric("train/acc", 1.5)
experiment.log_metric("test/acc", 15, step=1)
experiment.log_metrics({
    "train/acc": 5,
    "train/loss": 0.9
})

Limits

The below table lists limits related to experiments in Foundry.

To increase these limits, contact Palantir support.

Review the model experiments Python API reference for more information.