API: Model experiments

The model experiments Python API provides a set of methods for writing metrics and hyperparameters to experiments. This page contains information on available functions and classes in the palantir_models.experiments package.

palantir_models.experiments.Experiment

A class for tracking model training experiments. An experiment can be created from any ModelOutput class.

Copied!
1
model_output.create_experiment(name="experiment-name")

palantir_models.experiments.ErrorHandlerType

An enum for controlling how errors are handled when running experiment code. This specifically governs errors caused by non-user code such as network errors when writing metrics, or experiment size overflows.

The desired error handler type can be passed to ModelOutput.create_experiment when creating an experiment as shown below:

Copied!
1
2
from palantir_models.experiments import ErrorHandlerType
model_output.create_experiment(name="mnist-experiment", error_handler_type=ErrorHandlerType.WARN)

---

Experiment.log_param

palantir_models.experiments.Experiment.log_param(key: str, value: Union[bool, date, datetime, float, int, str])

• Logs the given parameter value to the given key.
• Parameters can be overwritten by logging to the same parameter key.

Copied!
1
experiment.log_param(key="learning_rate", value=1e-3)

Parameters

• key: str
  • The name of the parameter to write.
• value: Union[bool, date, datetime, float, int, str]
  • The value to write to the parameter.

---

Experiment.log_params

palantir_models.experiments.Experiment.log_params(parameter_map: Dict[str, Union[bool, date, datetime, float, int, str]])

• The batched version of log_param.

Copied!
1
2
3
4
experiment.log_params(parameter_map={
    "learning_rate": 1e-3,
    "model_type": "CNN"
})

Parameters

• parameter_map: Dict[str, Union[bool, date, datetime, float, int, str]]
  • A mapping of parameter name to parameter value.

---

Experiment.log_metric

palantir_models.experiments.Experiment.log_metric(metric_name: str, metric_value: float, step: Optional[int] = None)

• Logs the metric value in the series.
• If the series does not exist (has not been logged to yet), one will be created.
• If step is provided, an attempt to log at that step will be made. If the step has already been logged to, an error will be raised.
• If step is not provided, the current step of the series will be auto-incremented by 1.

Copied!
1
2
experiment.log_metric(metric_name="train/loss", metric_value=1.5)
experiment.log_metric(metric_name="train/loss", metric_value=0.9) # will auto-increment the step

Parameters

• metric_name: str
  • The name of the metric series to write the value to.
• metric_value: float
  • The value to write.
• step: Optional[int]
  • Optional step to write to. If the provided step is less than the current step, an error will be raised. If the step is not provided, the step will be auto-incremented.

---

Experiment.log_metrics

palantir_models.experiments.Experiment.log_metrics(values: Dict[str, float], step: Optional[int] = None)

• The batched version of log_metric.
• Logs all provided values to their respective series at the same step.
• If any series does not exist (has not been logged to yet), it will be created.
• If step is provided, an attempt to log at that step will be made. If the step has already been logged to on any series, an error will be raised.
• If step is not provided, the current step of the all the series will be independently auto-incremented by 1.

Copied!
1
2
3
4
experiment.log_metrics(values={
    "train/loss": 1.5,
    "train/acc", 0.7
})

Parameters

• values: Dict[str, float]
  • Mapping of metric name to metric value.
• step: Optional[int]
  • Optional step to write to. If the provided step is less than the current step of any series being written to, an error will be raised. If the step is not provided, the steps of each series will be independently auto-incremented.

---