5 - Add a Time-Based Check from the Data Health Application

📖 Task Introduction

Data timeliness is a chief concern for pipeline maintainers and data consumers alike. The Time Since Last Updated (TSLU) check helps ensure your pipeline is updating at the required cadence. For variety, this task will guide the installation of the TSLU check on the output of your Flight Alerts schedule build using the Data Health application rather than in Data Lineage as you did in the previous task. Either method is acceptable for adding health checks.

🔨 Task Instructions

1. In your Data Lineage graph, right-click the flight_alerts_clean node and click on the word Open (not on the ↗) in the fly-out menu to open the dataset in the Dataset application.
2. In the top left area of the Dataset app, notice the Health menu item with the 1 next to it. Click the menu item to open the Data Health app as a tab inside the Dataset app.
3. From the Add checks section at the bottom of the screen, choose the Time ▾ dropdown and select Time since last updated.
   • Recall the logic in your Flight Alerts Schedule, which executes when the raw input triggers receive new data (or flight_alerts_raw has new logic). If we had an actual source system with its own schedule, we'd have a better idea about how frequently we should be expecting new data to flow in from the external source. Since we don't, let's set a variable expectation based on deviations from the median TSLU.
4. Tick the second checkbox and set the values to be 1 deviation above the median (median calculated based on the last 10 check results).
5. Select the Add check group link and select your Flight Alerts Schedule group.
6. Add a note: “Consider revising when the source table update cadence is known.”
7. Save your health check.

📚 Recommended Reading (~5 min read)

By default, check creators are automatically subscribed to alerts on check failure and will receive email and/or in-app notifications depending on the configured preferences. Read this documentation entry to learn more about these settings. You should also read this page, which further unpacks the nuances of “watching” a check. Also, read this documentation page about time-based checks.

Note in the image below the ability to pause, snooze, edit, or delete each check individually.