This content is also available at learn.palantir.com ↗ and is presented here for accessibility purposes.
Data Health checks run after a build has completed using a variety of backend processes depending on the check type. Because they run separate from the transform logic after a build or job has completed, they cannot be used to fail a build. In other words, if you install a primary key uniqueness health check, you will be only be notified of the failure, but undesirable data may continue to propagate downstream.
Foundry’s Data Expectations library, by contrast, can be invoked in Pipeline Builder to create health checks that cause the job to fail if unmet and add a layer of "documentation" in your code pipeline about the expected shape and size of the data. So, if your encoded primary key data expectation fails, your job will fail and unexpected data will not propagate downstream. What’s more, encoded expectations will show up in the Data Health app alongside any standard ones you configure.
DATAENG 06: If you have not completed the previous course in this track, do so now.
In some cases, data health checks like the ones you applied in the previous tutorial may be sufficient to monitor your pipelines. A full monitoring and protection program should take advantage of the Data Expectations framework for greater granularity and control. In this brief tutorial, you’ll practice adding some of these checks to your pipeline and view them in the Data Health application.