Media set syncs

This page discusses how to set up a media set source and sync into Foundry via Data Connection.

The following source types support media syncs:

• Amazon S3
• OneLake and Azure Blob Filesystem (ABFS)

A growing list of sources support media syncs. However, if your desired file-based source is not yet supported, you can ingest your files within a dataset and convert them into media sets via Python transforms. For example, to ingest files from SharePoint Online into a media set, you can use the SharePoint Online connector to sync files into a Foundry dataset, then create a Python transform that reads those files and writes them into a media set. The SMB connector documentation includes a similar example.

Set up a media set source and sync

1. Find a supported source by navigating to the Source page via + New Source. Then, search for Media Sync to find all supported sources.

2. Ensure you have permissions to import any necessary network policies and then set up supported source using the appropriate instructions below:

• Amazon S3
• OneLake and Azure Blob Filesystem (ABFS)

3. In the Overview page of the source, find the Media set syncs section to create a media set sync.

4. Set up the media set sync by selecting the desired media file types. See supported media set schemas.

5. Create the desired build schedule for your media sync ingest. You can edit the schedule after the initial configuration.

6. Select the relevant subfolder within your source. If your media files are at the root path, there is no need to add a subfolder configuration.

7. Set up your sync filters. Available sync filters include Exclude files already synced, Path matches, File size limit, and Ignore items not matching schema.

8. Choose Save media set sync when you have selected your initial configuration.

9. Select Run to trigger your first sync and view your media sync.

Once you have set up your media set sync, learn how to leverage your media set with transforms in Pipeline Builder.