Amazon Kinesis

Connect Foundry to Amazon Kinesis to read data from a Kinesis stream into a Foundry stream in realtime.

Supported capabilities

Data model

The Kinesis connector parses message contents into unicode strings. Use a downstream streaming transform (for example, parse_json in Pipeline Builder) to parse structured data.

• The partition_key column will contain the partition key that was used to post the message to Kinesis.
• The kinesis_ingestion_timestamp column will contain the timestamp when the message was posted to Kinesis.
• The foundry_ingestion_timestamp column will contain the timestamp when the message was ingested by Foundry.

Performance and limitations

The connector always uses a single consumer thread per active shard on the source Kinesis stream.

Streaming syncs are meant to be consistent, long-running jobs. Any interruption to a streaming sync is a potential outage, depending on the expected outcomes.

Currently, streaming syncs have the following limitations:

• Jobs from agent connections restart during maintenance windows (typically once a week) to pick up upgrades. Expected downtime is less than five minutes.
• Jobs from direct connections restart at least once every 48 hours. Expected downtime is single-digit minutes (assuming resource availability allows jobs to restart immediately).

We recommend connecting through two agents per source to minimize downtime. Be sure the agents do not have overlapping maintenance windows.

Message ordering

The Kinesis connector guarantees message delivery order for messages with the same partition_key. Messages with different partition_key values may be processed in any order.

Setup

1. Open the Data Connection application and select + New Source in the upper right corner of the screen.
2. Select Kinesis from the available connector types.
3. Choose to use a direct connection over the Internet or to connect through an intermediary agent.
   • We recommend connecting through two agents per source to minimize downtime. Be sure the agents do not have overlapping maintenance windows.
4. Follow the additional configuration prompts to continue the set up of your connector using the information in the sections below.

Learn more about setting up a connector in Foundry.

Connection settings

Authentication

Select an authentication method for your Kinesis connection: AWS Instance or Static Credentials.

Below is a sample IAM policy with examples of the permissions required to read from and write to specified kinesis streams.

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadKinesisStream",
      "Effect": "Allow",
      "Action": [
        "kinesis:ListShards",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords",
        "kinesis:DescribeStream"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/read-stream-name"
    },
    {
        "Sid": "WriteKinesisStream",
        "Effect": "Allow",
        "Action": [
            "kinesis:PutRecords"
        ],
        "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/write-stream-name"
    }
  ]
}

AWS Instance

When your Foundry agent is running on an AWS resource with a provisioned IAM role (e.g. an EC2 instance), the Kinesis connector will use the provisioned IAM role to connect to Kinesis streams. No additional configuration is required.

Static Credentials

Static Credentials refers to standard AWS authentication with an Access Key ID and Secret Access Key tied to an IAM user.

STS Role

The Kinesis connector can optionally assume an STS role before connecting to a Kinesis stream. Refer to the AWS documentation ↗ for details about these parameters.

Networking

The connector must have access to the AWS Kinesis API and optionally the AWS STS API if using an STS role.

• Kinesis API: https://kinesis.<region>.amazonaws.com
• STS API: https://sts.<region>.amazonaws.com

Sync data from Kinesis

Learn how to set up a sync with Kinesis in the Set up a streaming sync tutorial.

Export data to Kinesis

The connector supports exporting to external Kinesis streams in Data Connection.

To export to Kinesis, first enable exports for your Kinesis connector. Then, create a new export.

Export configuration options