4. Scheduling Data Pipelines7. Key Takeaways

7 - Key Takeaways

This content is also available at learn.palantir.com ↗ and is presented here for accessibility purposes.

The Scheduler application brings automation and logic to your data pipelines. In this tutorial, you configured an elementary schedule and augmented your project with additional documentation to improve maintainability. Skillfully scheduling pipelines is an important part of pipeline monitoring, and we strongly recommend all data engineers working on data pipelines in Foundry be familiar with best practices for implementing schedules.

In this tutorial you:

  • Developed an event-driven schedule for the Datasource Project segment of the larger, more fully-featured pipeline you’ll continue to develop throughout this training track.

Below is a list of product documentation used in the course of this training:

The next several tutorials will show you ways you can bring your data engineering skills to bear in the Foundry platform to perform common data transformation tasks. The first of these will be accessing and parsing raw files (csv and json) to bring flight passenger data into your overall project.