Monitoring rules reference

Monitoring rules are configured on a per resource basis, with rules for the following resources:

• Agents
• Schedules
• Objects and links
• Streaming datasets
• Live deployments
• Time series syncs
• Geotemporal observations
• Automations

All monitoring rules contain a configurable field called Alert severity which is the severity granted to an alert when its condition is triggered. Monitoring views can be configured to only send out alerts that meet or exceed a certain severity.

Agent rules

Agent last heartbeat time

Alerts when the agent bootstrapper's last heartbeat is older than a set threshold.

We recommend setting this monitor value to 10 minutes.

Agent manager version stale time

Alerts when the agent bootstrapper version has not been upgraded since a set threshold.

We recommend setting this monitor value to 10 days.

Agent version stale time

Alerts when the agent version has not been upgraded since a set threshold.

We recommend setting this monitor value to 10 days.

High CPU utilization

Alerts when the agent CPU utilization exceeds a set threshold.

We recommend setting this monitor value to 80 (%).

JVM heap usage is close to the limit

Alerts when the JVM heap usage exceeds a set threshold.

We recommend setting this monitor value to 70 (%).

Low disk space

Alerts when the available disk space drops below a set threshold.

We recommend setting this monitor value to 10GB.

Time until earliest keystore certificate expires

Alerts when a certificate in the agent's keystore will expire within a set threshold.

** We recommend setting this monitor value to medium severity at less than 30 days and high severity at less than 10 days.**

Time until earliest truststore certificate expires

Alerts when a certificate in the agent's truststore will expire within a set threshold.

We recommend setting this monitor value to medium severity at less than 30 days and high severity at less than 10 days.

Queue size

Alerts when the number of jobs queued on an agent exceeds a set threshold.

We recommend setting this monitor value to 70 (jobs).

Schedule rules

Consecutive schedule failures

Alerts when the number of consecutive schedule failures meets or exceeds a set threshold. This does not count schedule runs that result in a cancelled build.

The default behavior for this monitor is to alert with medium severity at one failure and high severity at three failures, though these thresholds are highly dependent on the frequency and stability of the schedules that are included in the monitoring rule's scope.

Schedule duration

Alerts when a schedule is running longer than a set threshold.

Object and link rules

Changelog jobs failing on active pipeline

Alerts when the "changelog" job for the object or link's active pipeline is failing. This rule is non-configurable, alerting with medium severity at one changelog job failure and high severity at three changelog job failures.

Merge changes job failing on active pipeline

Alerts when the "merge changes" job for the object or link's active pipeline is failing.

The default behavior for this monitor is to alert with low severity at one failure, medium severity at three failures, and high severity at seven failures.

Sync jobs failing on active pipeline

Alerts when the "sync" job for the object or link's active pipeline is failing.

The default behavior for this monitor is to alert with low severity at one failure, medium severity at three failures, and high severity at seven failures.

Changelog jobs failing on replacement pipeline

Alerts when the "changelog" job for the object or link's replacement pipeline is failing. This rule is non-configurable, alerting with medium severity at one changelog job failure and high severity at three changelog job failures.

Merge changes job failing on replacement pipeline

Alerts when the "merge changes" job for the object or link's replacement pipeline is failing.

The default behavior for this monitor is to alert with low severity at one failure, medium severity at three failures, and high severity at seven failures.

Sync jobs failing on replacement pipeline

Alerts when the "sync" job for the object or link's replacement pipeline is failing.

The default behavior for this monitor is to alert with low severity at one failure, medium severity at three failures, and high severity at seven failures.

Scroll job failing on pipeline

Alerts when the "scroll" job for the object or link's active or replacement pipeline is failing. Scroll jobs are responsible for streaming data from the backing datasource to the object databases.

The default behavior for this monitor is to alert with low severity at one failure, medium severity at three failures, and high severity at seven failures, and these values are configurable.

Sync propagation delay

Alerts when a dataset backing the object has a transaction with a sync time that exceeds a set threshold.

Invalid stream records detected

Alerts when records in an input stream contain format violations. The scroll job ignores these records. This rule is non-configurable, alerting with critical severity when the number of ignored rows is greater than or equal to one.

Streaming dataset rules

Derived stream monitors

Last checkpoint duration

Alerts if the last checkpoint took more time than the configured threshold to complete.

Liveness: time since last successful checkpoint

Alerts if the stream has not completed a checkpoint since the configured threshold. The default threshold configuration is 2 minutes. This monitor encompasses streams that are not running as well as streams failing a checkpoint.

Total lag

Alerts if a stream's lag (total unprocessed upstream records) exceeds the set threshold.

This monitor indicates that streaming transforms are taking too long to run, or there is a problem with the streaming transforms infrastructure.

Total throughput

Alerts if a stream's throughput (records processed per checkpoints) falls below the set threshold.

This monitor indicates that streaming transforms are taking too long to run, or there is a problem with the streaming transforms infrastructure.

Ingest stream monitors

Records ingested over last 5 minutes / 30 minutes / 1 hour / 4 hours / 1 day

Alerts if the number of records ingested into the raw stream's live view over the selected time window was less than or equal to the configured threshold.

Live deployment rules

Live deployment heartbeat

Alerts when deployment has not emitted a heartbeat for more than one minute.

Time series sync rules

Points written by the time series sync over last 5 or 30 minutes

Alerts if the number of points written by the time series sync over the last 5 or 30 minute window was less than or equal to the configured threshold.

Geotemporal observation rules

Geotemporal observations sent over last 5 or 30 minutes

Alerts if the number of geotemporal observations sent over the last 5 or 30 minute window was less than or equal to the configured threshold.

Automation rules

Automation has no new events

Alerts if there has been no new evaluation since the configured threshold.

The latest event exceeded the automation's failure threshold

Alerts if the most recent execution had at least the configured number of failures.