This page lists the resources you may need to reference when implementing an end-to-end streaming workflow.
Data Connection supports syncing data from a wide variety of streaming platforms into Foundry streaming datasets, which can then be used in streaming pipelines. Streaming syncs enable data to flow into Foundry with low latency and high throughput to support real time decision-making processes.
There are two ways to sync data from streams into Foundry:
Foundry can connect to many sources of streaming data, including:
This page lists the resources you may need to reference when implementing an end-to-end streaming workflow.
We recommend reviewing the following introductory concept page to understand what streams are, how they are stored, and how they are processed.
These pages will offer a broader scope of the various points to consider when determining if streaming is right for your use case.
You will need to complete one of the following workflows to connect your external data sources to Foundry for streaming. We recommend reviewing both options to understand possible benefits and limitations for your use case.
You can use Pipeline Builder to transform your live data. Outputs of your Pipeline Builder transforms will still be streaming datasets that you can use in real time throughout Foundry.
Set up alerting around your pipeline's health.
Here, you can find tools to improve development of streaming pipelines.