SAP (Custom source)

The SAP (Custom source) connector allows you to connect Foundry to SAP systems using a manually configured source with the magritte-sap-source connector type. This connector uses a custom YAML-based setup rather than the guided configuration available in newer SAP connectors.

Supported capabilities

Setup

1. Open the Data Connection application and select + New Source in the upper right corner of the screen.

2. Select Add Custom Source.

3. Give the source a name and location.

4. In the custom YAML section, enter the source definition:

   Copied!
   1
   2
   3
   type: magritte-sap-source
   url: https://<host>:<port>/sap/palantir
   usernamePassword: <username>:{{password}}

   • <host> is the hostname of the SAP application server.
   • <port> is the HTTPS (or HTTP) port for the Internet Communication Manager (ICM).
   • The username and password are those of the technical user created for Foundry when installing the Connector.

   Enter {{password}} exactly as shown. After saving, enter the password value under Encrypted values on the right side of the dialog.

5. Select Save.

Learn more about setting up a connector in Foundry.

Enable streaming syncs (SLT only)

To support streaming syncs, the source must be configured with an explicit SLT connection type. Add the connectionType parameter to the source YAML:

Copied!
1
2
3
4
5
6
7
type: magritte-sap-source
url: https://<host>:<port>/sap/palantir
usernamePassword: <username>:{{password}}
connectionType:
  type: slt
  slt:
    context: <context>

The context value is the unique identifier of the RFC connection between the SLT Replication Server and the source ERP system. Learn more about configuring SLT.

Source configuration

The following parameters can be configured on the source.

Networking and connectivity

Make sure to properly configure egress policies to allow Foundry to reach the SAP system. For on-premises SAP environments, agent proxy policies are typically required to route traffic correctly.

Many SAP systems use custom-signed certificates, which can cause SSL handshake exceptions when configuring the connection for the first time. Make sure you have the correct custom certificates from your system and add them to the source.

Source exploration

Once the source has been created, use the SAP Source Explorer for interactive exploration.

1. Navigate to the source and select Explore and create syncs.

   

2. For remote and SLT connections, select the appropriate context.

   

3. Select the tables or modules of interest and add them to the graph.

   

Batch syncs

Create a sync

Create a sync for each SAP object you want to extract to Foundry.

1. From the source overview page, select + New next to batch sync.
2. Configure the standard settings: name, target dataset, and schedule.
3. Set Transaction type to one of the following options:
   • APPEND — for incremental updates. See Incremental syncs for details.
   • SNAPSHOT — for a full load.
4. Select an SAP object type (see SAP object types below).
5. If using SLT or connecting to a remote system, select a Context. See Install a remote agent for details on contexts.
6. Enter the Object name. As you type, the field suggests matching objects based on the selected SAP object type.
7. If configuring an incremental sync, provide an Incremental field.
8. Optionally, configure additional parameters in the Extras tab.

SAP object types

The following object types are available from the SAP object type dropdown in the sync configuration UI.

Remote object types require a Context value that identifies the remote system. Learn more about remote connections.

BEx query configuration

BEx is a multidimensional query framework built on SAP BW InfoProviders. BEx queries use standard SAP BW access methods and inherit all SAP BW authorization concepts. They represent business logic applied to InfoProvider views.

When you select BW BEx Query as the object type, the sync editor shows a dedicated BEx query builder with the following options:

For BEx paging configuration, see Advanced YAML parameters in the sync parameters section.

Function configuration

SAP function modules can both extract data from and write data back to an SAP system. The Connector can only extract data from functions that have at least one tabular output parameter.

If a function returns more than one output parameter, use the paramName parameter to specify which output to write to the Foundry dataset:

Copied!
1
2
3
4
type: magritte-sap-source-adapter
sapType: function
obj: BAPI_USER_GETLIST
paramName: USERLIST

Input parameters can be configured using inputParams. Three value types are supported:

Example with a table-type input parameter:

Copied!
1
2
3
4
5
6
type: magritte-sap-source-adapter
sapType: function
obj: EM_GET_NUMBER_OF_ENTRIES
inputParams:
  IT_TABLES: '[{"TABNAME":"TSTC"}]'
paramName: IT_TABLES

Advanced-only object types

The following object types are not available from the dropdown and must be configured from the Advanced tab.

Transaction codes

Extracts data generated by SAP reports. Only reports using ABAP List Viewer (ALV) are supported.

By transaction code name:

Copied!
1
2
3
type: magritte-sap-source-adapter
sapType: tcode
obj: ZTEST_ALV

By program name:

Copied!
1
2
3
4
type: magritte-sap-source-adapter
sapType: tcode
obj: RSUSR200
programType: program

The following optional parameters can be added to the sync YAML:

HANA views

Extracts data from HANA Views enabled in the SAP application layer. See Ingest HANA views from SAP for prerequisites.

Copied!
1
2
3
type: magritte-sap-source-adapter
sapType: hanaview
obj: ZEXT_SBOOK

CDS views

Extracts data from ABAP CDS (Core Data Services) Views. ABAP CDS defines semantic data models on the central database of the application server.

Incremental syncs

Incremental syncs are stateful syncs that enable append-style transactions from the same table. To enable incremental syncs, set Transaction type to APPEND.

The following incremental types are available:

The table below shows which incremental types are supported per object type:

The incremental field should ideally be a monotonically increasing value. The system uses a "greater than or equal to" comparison to avoid missing data if a sync runs midway through a given date. As a result, duplicate values may appear in the Foundry dataset and should be removed as a first step in the transformation pipeline.

Twin table configuration

The Incremental Twin Table setting specifies the twin table. The Incremental Twin Mapping setting defines the join conditions between the primary and twin tables.

Mapping syntax:

{PRIMARY_TABLE_NAME}-{FIELD_NAME}={TWIN_TABLE_NAME}-{FIELD_NAME}

Multiple join conditions can be combined with a semicolon (AND operator).

Request-based incremental for InfoProviders

In SAP BW, DSOs and InfoCubes are loaded using Data Transfer Processes (DTPs). Each DTP creates a loading request, and the Connector uses these requests to capture changed data.

Configure via the Advanced tab:

Copied!
1
2
3
4
type: magritte-sap-source-adapter
sapType: infoprovider
obj: <technical-name-of-infoprovider>
incrementalType: REQUEST

Reset incremental state

To force a full reload and re-initialize incremental ingest:

1. Switch to the Advanced tab in the sync configuration.
2. Add resetIncrementalState: true to the YAML.
3. Set the transaction type to SNAPSHOT (keep any incremental* parameters in place).
4. Run the sync. This performs a full snapshot, replacing all files in the dataset.
5. Remove resetIncrementalState: true from the YAML.
6. Set the transaction type back to APPEND.

Subsequent syncs will resume incremental appends as normal.

Sync parameters

The following parameters are available in the Extras tab or via the Advanced YAML view.

General parameters

Debug logging

Set to On to enable debug logging for this sync.

Continuous resource check

When On, all paging requests are subject to a resource check (memory, CPU). When Off, only the initial page request is checked. See performance parameters for details.

Resource check

Set to Off to disable all resource checks for this sync.

Filter

Filters the data extracted from SAP. All field names must match the data dictionary.

Examples:

• Price between 500 and 650: PRICE=500:650
• Customer A, B, or C: CUSTOMER=A,B,C
• Price between 500 and 650 AND customer A, B, or C: PRICE=500:650;CUSTOMER=A,B,C
• Material starting with PAL, DIS, or SAP: MATERIAL=PAL*,DIS*,SAP*
• Date on or after 9 August 2019 (format YYYYMMDD): DATE>=20190809

Dynamic filters

Dynamic filters use special keywords and functions to create more flexible filter expressions. Available from add-on version SP26 and later.

Fixed keywords

Date calculation functions

Functions can be used directly with fixed keywords or nested:

• Single function: [ADDDAY([TODAY], 1)]
• Nested: [GETDAY([ADDDAY([FIRSTDAYOFMONTH], -1)])]

Examples:

• Records after today: BUDAT>[TODAY]
• Records after the last day of the previous month: BUDAT>[ADDDAY([FIRSTDAYOFMONTH], -1)]

Advanced YAML parameters

The following parameters require the Advanced tab and have no Basic view equivalent.

• maxRowsPerSync (SLT only): Limits the number of rows returned per sync run (approximate). Useful for splitting the initial sync of a very large table into smaller runs:

Copied!
1
maxRowsPerSync: 500000

• bexSettings (BEx only): Configures BEx query paging. Available from Connector version SP22 onward:

Copied!
1
2
3
bexSettings:
  bexPaging: true
  bexMemberLimit: 10

• bexPaging: Enables paging using automatically generated filters. Default is false.
• bexMemberLimit: Threshold for filter candidate selection. InfoObjects with more members than this value are excluded from filter generation. Default is 200. Minimum is 2.

-ignoreUnexpectedValues: If a sync fails with Unexpected value encountered in SAP data Failed to parse value XXX in field YYY, a date or number value cannot be parsed. To continue syncing and set unparsable values to null:

Copied!
1
ignoreUnexpectedValues: true

A warning is logged at the end of the sync with a summary of parse exceptions.

-outputSettingsOverride: Overrides the source-level Parquet file size setting for a specific sync:

Copied!
1
2
3
4
5
outputSettingsOverride:
  maxFileSize:
    type: rows
    rows:
      max: 10000

Copied!
1
2
3
4
5
outputSettingsOverride:
  maxFileSize:
    type: bytes
    bytes:
      approximateMax: 400MB

Streaming syncs

Streaming syncs are only supported for connections via an SAP SLT Replication Server and require a source configured with an explicit SLT connection type (see Enable streaming syncs above).

Each streaming sync creates and subscribes to its own Operational Delta Queue (ODQ) in the SLT Replication Server. Foundry polls the queue periodically (default interval: 1 second):

• If the queue is empty, no further requests are made until the next poll.
• If there are 50,000 records or fewer on the queue, they are consumed synchronously.
• If there are more than 50,000 records, consumption is paginated.

For lowest latency, the SLT Replication Server should have at least as many available dialog work processes as there are active streaming syncs.

Create a streaming sync

1. Open the SAP source. On the Overview page, scroll down to the streaming syncs table and select + Create streaming sync.

   

2. Enter the name of the SAP table to stream.

   

3. Choose a location for the output streaming dataset.

   

   Before proceeding, ensure the preview pane at the bottom of the screen has finished loading. The schema for the streaming dataset is derived from this preview. In some cases, the preview may only display the schema without data — this is sufficient.

4. Select Create streaming sync. You can choose to start the stream immediately or start it manually after creation.

   

Throughput and partition keys

Switching the throughput setting from Normal to Very high may improve performance but increases the number of partitions used. When more than one partition is used, partition keys must be set to guarantee ordering between unique records from SAP. These keys should correspond to the primary key of the table in SAP.

Webhooks

SAP webhooks allow you to write data back to SAP by invoking Business APIs (BAPIs) from Foundry. See Webhooks for an overview of how to set up a webhook.

The only task type available for SAP webhooks is sap-run-function-webhook-task-v0. The following example invokes BAPI_SALESORDER_CHANGE to modify the purchase date for a given sales document:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
    "function-name": "BAPI_SALESORDER_CHANGE",
    "inputs": {
        "SALESDOCUMENT": {{json sales-doc-id}},
        "ORDER_HEADER_IN": {
            "PURCH_DATE": {{json purchase-date}}
        },
        "ORDER_HEADER_INX": {
            "UPDATEFLAG": "U",
            "PURCH_DATE": "X"
        }
    },
    "output": "RETURN"
}

To target a remote SAP system, add a remote field to the task body:

Copied!
1
2
3
4
5
6
7
{
    ...
    "output": "RETURN",
    "remote": {
        "context": "<SAP_CONTEXT_NAME>"
    }
}

Related documentation

The following guides cover workflows that involve both SAP-side and Foundry-side configuration:

• Extract long text from SAP: Decompress and ingest long texts from the STXL table.
• Configure custom authorizations and role management: Set up custom authorization roles for the SAP add-on.
• Configure extractors: Configure SAP BW Business Content extractors.
• Ingest HANA views from SAP: Publish and ingest HANA external views.
• User-attributed SAP writeback with OAuth 2.0: Configure OAuth 2.0 for named-user writeback to SAP.