Add an Ontology output

You can choose to add an Ontology output to guide your pipeline integration towards clean, structured data that defines new elements of your global Ontology. Adding output guidance can help save time on compute checks and help inform the end-to-end creation of your workflow.

Pipeline Builder currently supports creating Ontology objects and links from both batch and streaming datasets. However, it is not possible to enable edits or configure many-to-many links for objects backed by streaming datasets in Pipeline Builder.

Add an object type output

There are two ways to add an object type output. You can create a blank object via the pipeline outputs panel, or create a populated object on the graph connected to a transform node.

With the pipeline outputs panel

Open the Pipeline outputs panel located on the right side of Pipeline Builder, then select Object type > Add to create a blank object template where you can edit the object name, icon, and properties.

Note that the Plural name and Object type ID will auto-populate from Name for convenience.

You can now populate your object with data by selecting Enter data manually or by dragging the output of an existing dataset or transform to this node's input.

With the graph

Select the transform node that you want to create the object from, then select Add output > New object type.

Enable edits on object types

You can enable edits on object types in Pipeline Builder or in Ontology Manager if the object was created in Pipeline Builder and is backed by a batch dataset.

To enable edits, hover on the object type in the right panel and select Edit. Toggle Allow edits to objects of this type. Save and deploy your pipeline for this to go into effect.

Add a link type output

Once object type outputs have been added, you can define link type outputs. Similar to object outputs, these can be created in the panel or on the graph.

With the pipeline outputs panel

In the Pipeline outputs panel, select Link type > Add. This opens a form in which you can define the left and right sides of your link type.

Legend

Link type ID: This is an auto-generated ID that uniquely represents this link.

Choose an object type output: Select an object type from the dropdown menu. If you do not have any object outputs in your pipeline, you will need to configure an object output type first.

Cardinality: Choose the cardinality of each side of the link type. Two types of cardinality are supported: one-to-many and many-to-many. One-to-one link types are currently not supported. In the examples below, assume that there are two object types that are related to each other through a cardinality: an Aircraft object type and a Flight object type.

• One-to-many cardinality: This indicates that one Aircraft can be linked to many Flights.
• Many-to-many cardinality: This indicates that one Aircraft can be linked to many Flights and one Flight can be linked to many Aircraft.
  • Note that when you select a many-to-many cardinality, you will see a prompt to Click and drag the output from a dataset or transform to use as the join table. This table should contain all combinations of links between the primary key of the first object type (Aircraft in this example) and the second object type (Flight in this example).
  • Below is an example of many-to-many cardinality, where a Class object can be linked to many Student objects and a Student object can be linked to many Class objects:

Primary key/column: Choose which property to use to create the left side of the link.

• In a one-to-many cardinality link type, the primary key will be determined by the object type.
• In a many-to-many cardinality link type, select which columns in the join table map to the primary keys of each of the linked object types.

Foreign key/column: Choose which property to use to create the right side of the link.

• In a one-to-many cardinality link type, the foreign key property of one object type must refer to the primary key property of the other object type.
• In a many-to-many cardinality link type, select which columns in the join table map to the primary keys of each of the linked object types.

Name: Fill in the display name for each side of the link type. A side of the link type represents the link to that object type. The display name for the Aircraft object type describes the link from Flight to Aircraft. In this example, you could choose the display name "Assigned Aircraft" since one Flight has one assigned Aircraft.

Plural display name: For sides of a link type with the cardinality many, you will also be prompted to fill in a plural display name so user applications can display the correct name when displaying the linked objects. In our example, the plural display name for the Flight object type will describe the link from Aircraft to Flight. We might choose the plural display name to be "Scheduled Flights" as one Aircraft has many scheduled Flights.

With the graph

Select the two object outputs that you want to create the link from, then right-click and select New link type. This will create a link with all the object information populated.

Alternatively, select a single object on the graph to create a New link type with one side populated.

Resolve Ontology validation errors

Users may make pipeline changes that result in validation errors in the Ontology. Some examples of errors include:

• Conflicting changes made to the same object type or link type in both Pipeline Builder and Ontology Manager.
• Changing the primary key of an object type that is referenced by a link type outside of the pipeline.
• Making breaking changes to an object type's schema such that a schema migration is required.

Any errors will show up on the right side panel under the Deploy this pipeline section. In the Errors tab, there will be a message describing what type of resolution needs to happen before deployment.

You must resolve these errors before you are able to deploy your pipeline.

When the Resolve button displays an arrow, selecting it will take you to Ontology Manager to resolve validation errors. Otherwise, you will be able to resolve the errors directly in Pipeline Builder.

Resolve merge conflicts in Ontology Manager

Follow the prompts in Ontology Manager to apply the merge conflict resolutions. If your changes affect potential edits, you can decide whether to drop edits or make your edits compatible with the latest version of the object. Otherwise, no additional specifications are needed for the merge conflict to be resolved, and you can select Apply.

Once changes are applied, the Ontology will be updated with the latest changes and you can select Go back to pipeline.

After resolving conflicts, you may occasionally encounter the error Ontology type outputs are out of date with the ontology when returning to your pipeline. This is because the pipeline has yet to pick up changes to your Ontology outputs made through the conflict resolution workflow above.

Selecting Resolve should fix this issue and allow you to deploy your pipeline.

Resolve conflicts in Pipeline Builder

Some Ontology validation errors do not require switching to Ontology Manager. In the example below, a schema migration is required, which happens when an object type schema in Pipeline Builder has been modified from the latest schema on the main branch of the Ontology. Select Resolve to choose the migration strategy directly in Pipeline Builder.

For impacted properties, you can choose whether to Drop edits from property or migrate your edits to make them compatible with the latest version of the property.

Once you have chosen the best approach, select Apply migrations.

Bring object type outputs back into Pipeline Builder

You can bring object types backed by a dataset output from your pipeline into Pipeline Builder following the steps below:

On the right side panel, select the cog icon next to Pipeline outputs.

Select Replace with objects. This will open the Replace datasets with ontology type outputs dialog.

Select the dataset outputs you want to convert back to the corresponding objects in your pipeline and select Import selected outputs.

Save your changes and deploy your pipeline.

[Deprecated] Disown Ontology type outputs in Pipeline Builder

Disowning Ontology type outputs in Pipeline Builder will transfer management of those Ontology types to Ontology Manager. This means that all future modifications to those Ontology types will only be possible in Ontology Manager. To disown Ontology types, select the object types you want to disown. Any link types that reference the selected object types will automatically transfer to Ontology Manager.

Select Disown selected object and link types.

In this pipeline, the object type output nodes will convert to dataset output nodes that back the object types in Ontology Manager. From the far right panel, navigate to the object type in Ontology Manager by selecting the icon next to the output.

Deploy your pipeline after disowning the selected objects.