Stream monitoring [Beta]

Stream monitoring enables alerting around your pipeline's health.

A pipeline is considered "healthy" when it is:

• Ingesting new records from source systems.
• Processing and writing records into user-facing Foundry applications.

You can monitor the health of a pipeline by setting ingest and output alerts that fire when the number of records ingested or output, over a time period, falls below a user-defined threshold.

For example, an ingest monitor could alert if your stream has ingested zero records over the last five minutes, while an output monitor could alert if your stream has output less than 1000 records over the last 30 minutes.

Creating monitors

1. Open the Data Health application.
2. Select the Monitoring Views tab.
3. Create or select an existing Monitoring View.
4. Navigate to the Manage Monitors tab.
5. Select Add new alert.
6. Select Add monitoring rules.

Configuring ingest monitors

Ingest monitors ensure that:

• Your source system is generating and sending records to Foundry.
• Foundry is writing records into the ingest stream's live view.

To configure this:

1. Follow the creating monitors instructions.
2. Select Streaming dataset as the resource type.
3. Add the ingest streams to monitor and select Next.
4. Select + Add in the Ingest stream monitors card.
5. Select Records ingested monitor with a 5 or 30 minute duration.
6. Set your threshold and select Next.
7. Review your monitor and select Save monitoring rules.

Examples:

• Records ingested with a five minute duration and threshold of zero: Alerts when your stream has written zero records to the live view over the last five minutes.
• Records ingested the 30 minute duration and threshold of 1000: Alerts when your stream has written less than or equal to 1000 records to the live view over the last 30 minutes.

Configuring output monitors

Output monitors ensure that your pipeline is:

• Processing ingested records.
• Writing those processed records into user-facing Foundry applications.

Time series

To monitor records written to time series, you will set alerts on the time series sync.

1. Follow the creating monitors instructions.
2. Select Time series sync as the resource type.
3. Add the time series syncs to monitor and select Next.
4. Select + Add in the monitors card.
5. Select Points written to Epoch with a 5 or 30 minute duration.
6. Set your threshold and select Next.
7. Review your monitor and select Save monitoring rules.

Examples:

• Points written to Time Series DB with a five minute duration and threshold of zero: Alerts when your time series sync has written zero records over the last five minutes.
• Points written to Time Series DB with a 30 minute duration and threshold of 1000: Alerts when your time series sync has written less than or equal to 1000 records over the last 30 minutes.

Geotemporal observations

To monitor geotemporal observations, you will set alerts on the backing observation dataset.

1. Follow the creating monitors instructions.
2. Select Geotemporal observations as the resource type.
3. Add the observation datasets to monitor and select Next.
4. Select + Add in the monitors card.
5. Select Geotemporal observations sent with a 5 or 30 minute duration.
6. Set your threshold and select Next.
7. Review your monitor and select Save monitoring rules.

Examples:

• Geotemporal observations sent with a five minute duration and threshold of zero: Alerts when your geotemporal sync has sent zero observations to geotime over the last five minutes.
• Geotemporal observations sent the 30 minute duration and threshold of 1000: Alerts when your geotemporal sync has sent less than or equal to 1000 observations to geotime over the last 30 minutes.

Viewing metrics

To view the metrics underlying a monitor, select the monitor rule in row in the monitoring view's Manage monitors tab.

Metrics are only available for streaming or time series monitors with a single target scope.

Monitor notifications

You can configure notifications through the monitoring view's Manage subscriptions tab.

To set email alerts:

1. Select Add users to group.
2. Search and select a user.
3. Select the severities to get notified for.

To set PagerDuty alerts:

1. Select Create new PagerDuty integration.
2. Set the Integration Name.
3. Create an integration in PagerDuty.
4. Copy and paste the integration key into the Integration Key.
5. Select the Severity of alerts.

Firing alerts

You can view firing alerts in the Troubleshoot alerts tab of your monitoring view.