Linked products
You can modularize a complete workflow into a set of linked products in a Marketplace store. This allows smaller products to be iterated upon and upgraded in isolation.
Using linked products allows you to ensure:
- Automatic mapping between one product’s content and another product’s inputs
- Semantically-versioned dependencies between products
- Less duplicate content upon installation, as overlapping content in multiple products can be separated into one upstream product for consumption downstream
For example, if your workflow contains data-cleaning pipelines and object types backed by the clean data, then the pipelines and object types may be packaged into separate products. A connection between the pipeline and Ontology products enables their simultaneous installation, automatically mapping the clean datasets content from the pipeline product to the required inputs of the Ontology product.
Packaging linked products
DevOps inspects the source entities that are packaged to determine the links between products in the same store. In the following notional example, there is a data pipeline which outputs a dataset and an object type which is backed by the dataset.
The pipeline and object type can be packaged as individual products, and DevOps will ensure that the output dataset of the Pipeline product meets the dataset input requirements of the Ontology product. This link is derived from the source entities.
For example, if your product contains a Workshop application that has a dependency on the "Airplane" object type, then DevOps will only link products which contain the exact same "Airplane" object type as content.
DevOps creates the product links when the downstream product is packaged. This is demonstrated in the image above with the Ontology product. In other words, DevOps looks for products which have content fulfilling inputs of the product draft being published. Additionally, only products in the same store are considered. This means, if the upstream product is repackaged to produce new outputs and these outputs can be consumed by the downstream product, then the downstream product must also be repackaged to regenerate the link to ensure an up-to-date workflow.
While packaging a product, you can see which other products can provide inputs to the packaging draft.
Each linked product can be opened to see the mapping between the two products.
Breaking changes
A breaking change to a product is one which would cause dependent workflows of the product’s content to no longer function correctly. For example, a breaking change may be the removal of content or a type change of a dataset column.
If a breaking change is made in the upstream product, it may break the link to downstream products. The product link is for a semver compatibility range and therefore breaking changes should be indicated with a major version increment. Major increments signal to users that there are backwards-incompatible changes in the product’s content.
If a product was mistakenly published with a minor version increment, you can repackage the downstream product to trigger linked product generation and resolve the incompatibility. This ensures the downstream product has a link to the most recent version of the upstream product.
The safest procedure to follow is to repackage any downstream products of a recently-published product, so that all linked product mappings are regenerated.
Installing linked products
When a user installs a product, inputs can be fulfilled by choosing another installation or draft of a linked product. To see which linked products are available, visit the Inputs page of the installation draft.
To see the linked installations and drafts, select View graph.
Linked drafts are shown as an installation group on the drafts homepage.
Linked installation drafts are installed together during a single job, with pre-existing installations serving as input without being affected. After starting the installation, you will be redirected to the Job page, where you can keep track of the job status.
Splitting up a product
Often, large products contain distinct stages of a workflow (e.g. data-cleaning and application). Such a product can be split into smaller products, to allow more rapid iteration and releases of each part.
By separating some of the large product's content into another product, you can install both products together using the automatic mapping between them. However, this can lead to duplicate resources in existing installations.
Installing the newly-created product will create new resources, instead of reusing the installed resources of the original, larger product.
Best practices for product linkage
The following are some tips to consider when linking products, to ensure that your product remain serviceable:
- Package products from upstream to downstream. This ensures the generated product links are most up-to-date.
- Indicate breaking product changes with a major version increment.
- Repackage downstream product consumers to regenerate the product link.