This content is also available at learn.palantir.com ↗ and is presented here for accessibility purposes.
Always be documenting. Whether it’s visually with the Data Lineage application or text-based using other Foundry resources such as Notepad; being rigorous and thorough in explaining your pipeline logic and dependencies will promote rapid troubleshooting and prevent maintenance headaches.
In this tutorial you:
Introduced a cleaning step in your pipeline that used Pipeline Builder to join your preprocessed files into a usable output.
Practiced using Pipeline Builder’s version control mechanisms.
Saved a Data Lineage representation of the Datasource stage of your pipeline.
Created a Notepad document to contain pipeline information about the Datasource pipeline.
Below is a list of supporting product documentation for this training:
Now that you’ve created a multi-node flow from raw to clean, you’ll work on generating a schedule to automatically run the transforms in sequence. Skillfully scheduling pipelines is an important part of pipeline monitoring, and in the next tutorial, you’ll use the Scheduler application in a recommended configuration and produce written documentation about your pipeline to facilitate troubleshooting and maintenance activities.