BigQuery compute pushdown

To use compute pushdown with BigQuery, create a Python repository and install the most recent version of the transforms-tables library.

An Ibis ↗ connection to BigQuery will be established based on the connection details of the BigQuery tables configured as inputs and/or outputs. Data can be transformed using the Ibis DataFrame API. For complete guidance on the Ibis API, consult the Ibis documentation ↗.

An example of a BigQuery-backed Ibis transform is shown below:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
from transforms.api import transform
from transforms.tables import (
    TableLightweightInput,
    TableLightweightOutput,
    TableInput,
    TableOutput,
    BigQueryTable
)


@transform.using(
    source_table=TableInput("ri.tables.main.table.1234"),
    output_table=TableOutput(
        "ri.tables.main.table.5678",
        "ri.magritte..source.1234",
        BigQueryTable("PROJECT", "DATASET", "TABLE"),
    ),
)
def compute(source_table: TableLightweightInput, output_table: TableLightweightOutput):
    table = source_table.ibis().table()
    # Data transformation with Ibis
    output_table.ibis().write(table)