transforms.api.transform_polars

To use the Polars library, you must add polars as a run dependency in your meta.yml file. For more information, refer to the documentation ↗.

The transform_polars decorator is a thin wrapper around the lightweight decorator. Using it results in the creation of a lightweight transform that lacks some features of a regular transform.

This works similarly to the transform_pandas() decorator, however, instead of pandas DataFrames, the user code is given and is expected to return Polars DataFrames.

Note that spark profiles cannot be used with lightweight transforms, meaning that they also cannot be used with @transform_polars.

Copied!
1
2
3
4
5
6
7
8
>>> @transform_polars(
...     Output('ri.main.foundry.dataset.out'),  # An unnamed Output spec
...     first_input=Input('ri.main.foundry.dataset.in1'),
...     second_input=Input('ri.main.foundry.dataset.in2'),
... )
... def my_compute_function(ctx, first_input, second_input):
...     # type: (polars.DataFrame, polars.DataFrame) -> polars.DataFrame
...     return first_input.join(second_input, on='id', how="inner")