Use media sets with Python transforms

You can use media sets in Python transforms for PDF text extraction, optical character recognition (OCR), image tiling, metadata parsing, and more. The following sections explain how to set up media sets in your Python repository and how to read to and write from media sets with Python transforms.

Add a dependency on transforms-media in your code repository.

You must use the @transform decorator when working with media sets. Media set inputs and outputs can be passed in using transforms.mediasets.MediaSetInput and transforms.mediasets.MediaSetOutput specifications. During a build, these specifications are resolved into transforms.mediasets.MediaSetInputParam and transforms.mediasets.MediaSetOutputParam objects, respectively. These MediaSetInputParam and MediaSetOutputParam objects provide access to the media set within the compute function. Any number of media set inputs or outputs can be used in combination with any other valid transform inputs and outputs (such as tabular datasets). For example:

Copied!
1
2
3
4
5
6
7
8
from transforms.api import transform
from transforms.mediasets import MediaSetInput, MediaSetOutput
@transform(
    images=MediaSetInput('/examples/images'),
    output_images=MediaSetOutput('/examples/output_images')
)
def translate_images(images, output_images):
    ...

Read from media sets

You can access individual media items either by the file path or RID:

Copied!
1
2
3
4
5
6
7
8
9
10
from transforms.api import transform
from transforms.mediasets import MediaSetInput, MediaSetOutput
@transform(
    images=MediaSetInput('/examples/images'),
    output_images=MediaSetOutput('/examples/output_images')
)
def translate_images(images, output_images):
    image1 = images.get_media_item_by_path("image1")
    image2 = images.get_media_item("ri.mio.main.media-item.123")
    ...

However, you will likely want to transform all the items in your media set. To do this, you must first pull the items into a dataframe using a listing method. In the example below, we list all items in the input media set and write the resulting dataframe to a tabular output:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
from transforms.api import transform, Output
from transforms.mediasets import MediaSetInput
@transform(
    images=MediaSetInput('/examples/images'),
    listing_output=Output('/examples/listed_images')
)
def translate_images(ctx, images, listing_output):
    media_items_listing = images.list_media_items_by_path_with_media_reference(ctx)

    # You can perform regular PySpark transformations on media_items_listing

    column_typeclasses = {'mediaReference': [{'kind': 'reference', 'name': 'media_reference'}]}
    listing_output.write_dataframe(media_items_listing, column_typeclasses=column_typeclasses)

If multiple items in the media set are at a particular path, only the most recent will be included in the listing. The listing will have the following schema:

+--------------------------+-----------+-------------------+
|        mediaItemRid      |    path   |  mediaReference  |
+--------------------------+-----------+-------------------+
| ri.mio.main.media-item.1 | item1.jpg |  {{reference1}}   |
| ri.mio.main.media-item.2 | item2.jpg |  {{reference2}}   |
| ri.mio.main.media-item.3 | item3.jpg |  {{reference3}}   |
+--------------------------+-----------+-------------------+

Note that the above example only shows the top three rows of the listing.

By setting the typeclass of the mediaReference column, we allow the column to be read as a media reference.

Calls to get_media_item(), get_media_item_by_path(), and so on return a Python file-like stream object. All options accepted by io.open() ↗ are also supported. Note that items are read as streams, meaning that random access is not supported.

You can also return metadata about individual media items without downloading the full item. The metadata will include information such as the dimensions for images, length for audio, and more. For a full reference of available metadata, see the appendix below. The example below adds a column to the media item listing with the metadata for each image.

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
from transforms.api import transform, Output
from transforms.mediasets import MediaSetInput
from pyspark.sql import functions as F
from pyspark.sql.types import StringType
from conjure_python_client import ConjureEncoder
@transform(
    images=MediaSetInput('/examples/images'),
    listing_output_with_metadata=Output('/examples/listed_images_with_metadata')
)
def translate_images(ctx, images, listing_output_with_metadata):

    def get_metadata(media_item_rid):
        metadata = images.get_media_item_metadata(media_item_rid)
        return ConjureEncoder().default(metadata)

    metadata_udf = F.udf(get_metadata, StringType())

    media_items_listing = images.list_media_items_by_path_with_media_reference(ctx)
    listing_with_metadata = media_items_listing.withColumn('metadata', metadata_udf(F.col('mediaItemRid')))
    column_typeclasses = {'mediaReference': [{'kind': 'reference', 'name': 'media_reference'}]}
    listing_output_with_metadata.write_dataframe(listing_with_metadata, column_typeclasses=column_typeclasses)

Media sets support a certain number of built-in transformations out of the box. See the appendix below for the API and list of supported transformations. Calls to these transformations will also return a Python file-like stream object. To use these built-in transformations, call the appropriate method on the media set input. For example:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
@transform(
    images=MediaSetInput('/examples/images'),
    image_text_output=Output('/examples/listed_images_with_text')
)
def translate_images(ctx, images, image_text_output):

    def get_ocr_on_image(media_item_rid):
        return images.transform_image_to_text_ocr_output_text(media_item_rid).read().decode('utf-8')

    ocr_on_image_udf = F.udf(get_ocr_on_image, StringType())

    media_items_listing = images.list_media_items_by_path_with_media_reference(ctx)
    listing_with_ocr = media_items_listing.withColumn('text', ocr_on_image_udf(F.col('mediaItemRid')))
    column_typeclasses = {'mediaReference': [{'kind': 'reference', 'name': 'media_reference'}]}
    image_text_output.write_dataframe(listing_with_ocr, column_typeclasses=column_typeclasses)

Write to media sets

Media sets can be used as outputs to transformations by using the MediaSetOutput specification. Unlike regular datasets, media sets must already exist before being used as an output. This can be done through the Foundry filesystem interface by navigating to the folder where you want the media set to exist.

To upload an item, call the put_media_item() endpoint on the output media set. This endpoint accepts any file-like object and a path which will be used to identify the item in the output media set. The following is a basic example:

Copied!
1
2
3
4
5
6
7
8
9
from transforms.api import transform
from transforms.mediasets import MediaSetInput, MediaSetOutput
@transform(
    images=MediaSetInput('/examples/images'),
    output_images=MediaSetOutput('/examples/output_images')
)
def upload_images(images, output_images):
    with images.get_media_item_by_path("image1.jpg") as input_image:
        output_images.put_media_item(input_image, "copied_image1.jpg")

When copying items from one media set to another, you can use the fast_copy_media_item() method on the output. This is a faster and more efficient option than downloading and re-uploading the media item:

Copied!
1
2
3
4
5
6
7
@transform(
    images=MediaSetInput('/examples/images'),
    output_images=MediaSetOutput('/examples/output_images')
)
def upload_images(images, output_images):
    origin_media_item_rid = images.get_media_item_rid_by_path("image1.jpg")
    output_images.fast_copy_media_item(images, origin_media_item_rid, "fast_copied_image1.jpg")

Items can be uploaded to media sets in user-defined functions (UDFs) for higher parallelism. In the example below, we transform the PDFs in the input media set into JPEGs using the built-in PDF to JPEG transformation and upload those JPEGs to a new output media set. We then write out a tabular dataset containing the media references of those uploaded JPEGs:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from transforms.api import transform, Output
from transforms.mediasets import MediaSetInput, MediaSetOutput
from pyspark.sql import functions as F
from pyspark.sql.types import StringType
@transform(
    pdfs=MediaSetInput('/examples/PDFs'),
    output_images=MediaSetOutput('/examples/JPEGs'),
    output_references=Output('/examples/JPEG listing')
)
def upload_images(ctx, pdfs, output_images, output_references):

    def upload_jpg(media_item_rid, path):
        with pdfs.transform_document_to_jpg(media_item_rid, 0) as jpeg:
            response = output_images.put_media_item(jpeg, path)
        return response.media_item_rid

    upload_udf = F.udf(upload_jpg, StringType())

    listed_pdfs = pdfs.list_media_items_by_path(ctx)
    media_reference_template = output_images.media_reference_template()
    uploaded_jpegs = listed_pdfs\
        .withColumn('uploaded_media_item_rid', upload_udf(F.col('mediaItemRid'), F.col('path')))\
        .select('path', 'uploaded_media_item_rid')\
        .withColumn("mediaReference", F.format_string(media_reference_template,
                                                      'uploaded_media_item_rid'))

    column_typeclasses = {'mediaReference': [{'kind': 'reference', 'name': 'media_reference'}]}
    output_references.write_dataframe(uploaded_jpegs, column_typeclasses=column_typeclasses)

Upload from a filesystem (Catalog) dataset

The Python media set SDK has built-in tooling to upload the files from a conventional dataset in the Palantir filesystem (known as the Catalog) into a media set. For example:

Copied!
1
2
3
4
5
6
7
8
from transforms.api import transform, Input
from transforms.mediasets import MediaSetOutput
@transform(
    pdfs_dataset=Input('/examples/PDFs'),
    pdfs_media_set=MediaSetOutput('/examples/PDFs mediaset')
)
def upload_to_media_set(pdfs_dataset, pdfs_media_set):
    pdfs_media_set.put_dataset_files(pdfs_dataset, ignore_items_not_matching_schema=False)

This transform will upload all items from the dataset into the media set. If any items do not match the schema of the media set (for example, if there is a JPEG in the dataset), then the build will fail. By setting ignore_items_not_matching_schema=True any such mismatches will instead be ignored.

Files can alternatively be uploaded one by one. For example:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
from transforms.api import transform, Input, Output, incremental
from transforms.mediasets import MediaSetInput, MediaSetOutput
import os

@transform(
    output_mediaset=MediaSetOutput("/path/mediaset_output", should_snapshot=False),
    input_dataset=Input("/path/dataset_of_raw_files"),
)
def compute(input_dataset, output_mediaset):
    all_files = list(source_df.filesystem().ls())
    for current_file in all_files:
        with input_dataset.filesystem().open(current_file.path, 'rb') as f:
            filename = os.path.basename(current_file.path)
            output_mediaset.put_media_item(f, filename)

Reference: Built-in transformations

Transform a PDF document to JPEG

Transform an individual page of a PDF document into a JPEG and return it.

• Operates on: PDF documents

• Returns: JPEG image

• Parameters:

  • media_item_rid: The RID of the media item to be transformed.
  • page_number: The zero-indexed page number.
  • Height (optional): The desired height of the output image, in pixels.
  • Width (optional): The desired width of the output image, in pixels.

• Example:

Copied!
1
input_pdfs.transform_document_to_jpg("ri.mio.main.media-item.1", 0)

Transform a PDF document to PNG

Transform an individual page of a PDF document into a PNG and return it.

• Operates on: PDF documents

• Returns: PNG image

• Parameters:

  • media_item_rid: The RID of the media item to be transformed.
  • page_number: The zero-indexed page number.
  • Height (optional): The desired height of the output image, in pixels.
  • Width (optional): The desired width of the output image, in pixels.

• Example:

Copied!
1
input_pdfs.transform_document_to_png("ri.mio.main.media-item.1", 0)

Transform a PDF document to text by page with Optical Character Recognition (OCR)

OCR PDF into text output by page.

• Operates on: PDF documents

• Returns: Unstructured text in utf-8 encoding

• Parameters:

  • media_item_rid: The RID of the media item to be transformed.
  • page_number: The zero-indexed page number.
  • Language (optional): Language code, defaulted to ENG.

• Example:

Copied!
1
2
raw_output = input_pdfs.transform_document_to_text_ocr_output_text("ri.mio.main.media-item.1", 0)
doc_text_ocr = raw_output.read().decode("utf-8")

Transform a PDF document to hOCR XML by page with OCR

OCR PDF into hOCR XML by page. Learn more about hOCR ↗.

• Operates on: PDF documents

• Returns: hOCR xml in utf-8 encoding

• Parameters:

  • media_item_rid: The RID of the media item to be transformed.
  • page_number: The zero-indexed page number.
  • Language (optional): Language code, defaulted to ENG.

• Example:

Copied!
1
input_pdfs.transform_document_to_text_ocr_output_hocr("ri.mio.main.media-item.1", 0)

Transform a PDF document to extract raw text

Extract field from the PDF and return it. This is a parsing method that does not require image processing unlike with OCR.

• Operates on: PDF documents

• Returns: Unstructured text in UTF-8 encoding

• Parameters:

  • media_item_rid: The RID of the media item to be transformed.
  • page_number: The zero-indexed page number.

• Example:

Copied!
1
2
raw_output = input_pdfs.transform_document_to_text_raw("ri.mio.main.media-item.1", 0)
doc_text_extraction = raw_output.read().decode("utf-8")

Transform a PDF document to extract form fields

Extract all form fields from the whole PDF and return it.

• Operates on: PDF documents

• Returns: JSON

• Parameters:

  • media_item_rid: The RID of the media item to be transformed.

• Example:

Copied!
1
input_pdfs.transform_document_to_text_extract_field("ri.mio.main.media-item.1")

Transform a PDF document to extract table of contents

Extract field from the PDF and return it.

• Operates on: PDF documents

• Returns: JSON

• Parameters:

  • media_item_rid: The RID of the media item to be transformed.

• Example:

Copied!
1
input_pdfs.transform_document_to_text_extract_table_of_contents("ri.mio.main.media-item.1")

Transform image into hOCR XML

OCR image into hOCR XML. Learn more about hOCR ↗.

• Operates on: Image

• Returns: JSON

• Parameters:

  • media_item_rid: The RID of the media item to be transformed.

• Example:

Copied!
1
input_pdfs.transform_image_to_text_ocr_output_hocr("ri.mio.main.media-item.1")

Transform image into text

OCR image into text.

• Operates on: Image

• Returns: Unstructured text in utf-8 encoding

• Parameters:

  • media_item_rid: The RID of the media item to be transformed.

• Example:

Copied!
1
input_images.transform_image_to_text_ocr_output_text("ri.mio.main.media-item.1")

Transcribe audio to text

Transcribe audio file with speech into text.

• Operates on: Audio

• Returns: Unstructured plain text file with transcription

• Parameters:

  • media_item_rid: The RID of the media item to be transformed.
  • Language (optional): Language code to transcribe audio to. If left empty, the language will be inferred from the beginning of the audio file. Use either the set 1 or the ISO language name: https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes

• Examples:

Copied!
1
2
input_audio_files.transcribe("ri.mio.main.media-item.1")
input_audio_files.transcribe("ri.mio.main.media-item.1", TranscriptionLanguage.ARABIC)