• Documentation
    • Search documentation
    karat

    +

    K

    API Reference ↗
    Data connectivity & integrationSAPFoundry SAP SetupCreate a new streaming sync

    Create a new streaming sync

    Set up a source

    SAP streaming ingest is only supported for connections to an SAP system via the SAP SLT Replication Server.

    If one does not already exist, create a new SAP source that explicitly sets the connection type as SLT and includes the context to identify the source system. Follow the standard steps to create a new source, and use custom YAML of the following form:

    Copied!
    1
2
3
4
5
6
7
type: magritte-sap-source
url: https://<host>:<port>/sap/palantir
usernamePassword: <username>:{{password}}
connectionType:
  type: slt
  slt:
    context: <context>

    The context is the unique identifier of the RFC connection, as discussed in the SLT configuration guide.

    Load considerations

    Each streaming sync creates and subscribes to its own operational delta queue (ODQ) in the SLT Replication Server.

    Streaming ingest works as follows:

    • Foundry will poll the queue periodically; the poll interval defaults to 1 second and can be modified when creating the streaming sync.
    • If there are no records on the queue, no further requests are made.
    • If there are less than or equal to 50,000 records (the default page size) on the queue, those records will be consumed synchronously.
    • If there are greater than 50,000 records on the queue, record consumption will be paginated.

    In load testing, we observed that the lowest stream latency is achieved when there are at least as many available dialog work processes in the SLT Replication Server as there are active streaming syncs. When there are fewer available dialog work processes than active streaming syncs, latency is likely to be increased as streaming syncs compete for an available process to handle polling requests.

    Create a streaming sync

    1. Open the SAP source. You should see a table of existing streaming syncs on the Overview page; you may need to scroll down the page to view. Select + Create streaming sync at the top of this table.

      Create streaming sync

    2. In the first section, enter the name of the SAP table to stream.

    Sync settings

    1. In the second section, choose a location for the output streaming dataset.

    Output dataset

    Before proceeding to the next step, ensure that the preview pane at the bottom of the screen has loaded. The schema for the streaming dataset is derived from this preview and will be incorrect if the preview has not finished loading. In some cases, the preview may only show the schema and no data; this is sufficient.

    1. Select Create streaming sync at the top right of the screen. You can choose to run the stream immediately or start it manually after creation.

    Create streaming sync button

    To ensure that a stream automatically restarts in the event of either Data Connection agent or SAP system downtime, set a schedule on the streaming dataset with a 1-minute time trigger.

    Throughput and partition keys

    Switching the throughput setting from Normal to Very high may help increase performance. However, this will increase the number of partitions used. When more than a single partition is used, partition keys will need to be set in order to guarantee ordering between unique records from SAP. These keys should make up the primary key for the table in SAP.