Streaming resource guide

This page lists the resources you may need to reference when implementing an end-to-end streaming workflow.

Data Connection supports syncing data from a wide variety of streaming platforms into Foundry streaming datasets, which can then be used in streaming pipelines. Streaming syncs enable data to flow into Foundry with low latency and high throughput to support real time decision-making processes.

There are two ways to sync data from streams into Foundry:

• Data Connection supports pulling records from streaming platforms into Foundry. As with batch syncs, data is read from a stream and synced to Foundry using only unidirectional connections using the agent architecture.
• If desired, Foundry enables pushing records from a stream directly into a Foundry stream via the stream proxy.

Foundry can connect to many sources of streaming data, including:

• Apache Kafka
• ActiveMQ
• Amazon Kinesis
• Amazon SNS
• Amazon SQS
• Google Pub/Sub
• IBM MQ
• RabbitMQ
• MQTT (beta support)
• Solace

This page lists the resources you may need to reference when implementing an end-to-end streaming workflow.

1. Core concepts

We recommend reviewing the following introductory concept page to understand what streams are, how they are stored, and how they are processed.

• Streams
• Flink in Foundry streaming

2. Overview

These pages will offer a broader scope of the various points to consider when determining if streaming is right for your use case.

• Streaming pipelines overview
• Comparison: Streaming vs. batch
• Performance considerations
• Streaming compute usage
• Streaming profiles

3. Connect to data sources

You will need to complete one of the following workflows to connect your external data sources to Foundry for streaming. We recommend reviewing both options to understand possible benefits and limitations for your use case.

• Set up a streaming sync
• Push data into a stream

4. Transform your streaming data

You can use Pipeline Builder to transform your live data. Outputs of your Pipeline Builder transforms will still be streaming datasets that you can use in real time throughout Foundry.

• Create a streaming pipeline with Pipeline Builder
• Integrate your stream with the Ontology

5. Monitor streaming pipelines [Beta]

Set up alerting around your pipeline's health.

• Stream monitoring

6. Development tools

Here, you can find tools to improve development of streaming pipelines.

• Reset stream