Foundry connectors

Connectors for interacting with Foundry from the transforms APIs.

A Connector can be used interactively to construct TransformInput and TransformOutput objects and also to run a Transform.

FoundryConnector

class transforms.foundry.connectors.FoundryConnector(service_config, auth_header, filesystem_id=None, fallback_branches=None, resolver=None)

• Entry point for accessing Foundry services.
• The Foundry object manages interactions with Foundry services by providing APIs for manipulating datasets.

Parameters

• service_config (dict ↗)
  • A configuration dictionary conforming to the JSON spec in the Java class com.palantir.remoting.api.config.service.ServicesConfigBlock.
• auth_header (str ↗)
  • The authorization string to use when connecting to Foundry services.
• filesystem_id (str ↗, optional)
  • The backing filesystem to use.
• fallback_branches (List[str ↗], optional)
  • Fallback branches.
• resolver (Callable[[str ↗], str ↗], optional)
  • Function for resolving a dataset alias into a rid. Defaults to resolving the alias as a Project path.

---

input(alias=None, rid=None, branch=None, end_txrid=None, start_txrid=None, schema_version=None)

• Construct a TransformInput from the given parameters.
• The resource identifier used to construct the TransformInput will be resolved from the given alias unless the rid parameter is passed.

Parameters

• alias (str ↗, optional)
  • The alias of the dataset.
• rid (str ↗, optional)
  • The resource identifier of the dataset.
• branch (str ↗, optional)
  • The branch from which to read the dataset. If not set the branch is chosen as the first branch in the fallbacks list that exists in the Catalog.
• end_txrid (str ↗, optional)
  • The end transaction of the view, if not set, defaults to the latest transaction on the given branch.
• start_txrid (str ↗, optional)
  • The starting transaction of the view.
• schema_version (str ↗, optional)
  • The schema version to use when reading, if not set, defaults to the latest schema version on the given branch.

Returns

• An input object representing the requested dataset.

Return type

• transforms.api.TransformInput

Raises

• ValueError ↗
  • If either the alias or rid (but not both) is not specified.
• ValueError ↗
  • If a branch is not specified and a fallback branch cannot be found in the Catalog.

---

output(alias=None, rid=None, branch=None, txrid=None, filesystem_id=None)

• Construct a TransformOutput from the given alias or rid.
• The resource identifier used to construct the transforms.api.TransformOutput will be resolved from the given alias unless the rid parameter is passed.

Parameters

• alias (str ↗, optional)
  • The alias of the dataset.
• rid (str ↗, optional)
  • The resource identifier of the dataset.
• branch (str ↗, optional)
  • The branch to which to write the dataset. If not set the branch is chosen as the first branch in the fallbacks list.
• txrid (str ↗, optional)
  • The transaction into which data should be written.
• filesystem_id (str ↗, optional)
  • The filesystem in which to create the dataset if it doesn’t already exist.

Returns

• An output object representing the requested dataset.

Return type

• transforms.api.TransformOutput

Raises

• ValueError ↗
  • If either the alias or rid (but not both) is not specified.

---

run(transform)

• Run the given Transform using the latest inputs and outputs.

Parameters

• transform (transforms.api.Transform)
  • The transform to run.

---

auth_header

• str
  • The auth header used to contact Foundry.

---

fallback_branches

• List[str]
  • The fallback branches used to retrieve datasets.

---

spark_session

• pyspark.sql.SparkSession ↗
  • Understand the SparkSession created by FoundrySparkManager.