4. Scheduling Data Pipelines6. Exercise Summary

6 - Exercise Summary

This content is also available at learn.palantir.com ↗ and is presented here for accessibility purposes.

✅ What you built

An event-driven schedule to execute your pipeline up to and including your clean output.

✅ What you learned

  • In Foundry, a pipeline Schedule is treated as a first-class artifact with its own RID and permissioning scheme.
  • The pipeline schedule manager is accessible inside the Data Lineage application.
  • Most pipelines should use the connecting build mode rather than full or single build modes.
  • Scheduled builds run when the condition defined by the trigger(s) is satisfied.
  • Triggers can be event-based, time-based, or a combination of the two.
  • Most data pipelines should use event-based triggers.
  • We recommend aborting the schedule on failure with a retry value of 3 and 1 minute in between failures.
  • Force build should only be used for data connection.
  • Data Connection syncs should use their own designated schedule separate from the rest of the pipeline.