AWS MSK Setup Guide Beta

Follow our setup guide to connect AWS MSK to Fivetran.

Prerequisiteslink

To connect AWS MSK to Fivetran, you need an AWS MSK cluster.

Setup instructionslink

Find ENI IDlink

1. Open the AWS MSK console and go to your cluster.

2. In the Brokers summary section, for each broker find the Attached ENI ID. Make a note of the ENI ID(s).

   

Associate addresslink

1. Open the EC2 dashboard and select Network Interfaces.

2. For every broker ENI ID you found in Step 1, select Actions > Associate address to associate an Elastic IP address.

   

3. Make a note of the associated Elastic IPs for each broker. You will need them to configure Fivetran.

Configure security groupslink

1. On the EC2 dashboard, select Security Groups.

2. In the Security Groups of the ENI IDs you found in Step 1, whitelist Fivetran’s IP addresses.

Set propertylink

Set the advertised.listeners property for the corresponding port of each broker to the Elastic IP associated with the ENI of the broker. Use dynamic configuration to configure the broker-level properties.

Finish Fivetran configurationlink

1. In the connector setup form, enter the Destination schema name of your choice.

2. Enter the dedicated Consumer Group for Fivetran.

3. Enter your Servers details using the <host>:<port> format. Use the Elastic IPs you found in Step 2 as <host>. For example, if the Elastic IP for the broker is 10.100.10.10 and the authentication is SASL/SCRAM, then enter the server name as 10.100.10.10:9096.

   NOTE: Click + to add more than one server details.

4. Select your message type: Json, Avro, Protobuf, or Text.

   • If you choose Json, select a sync type: Packed or Unpacked.

   • If you choose Avro or Protobuf, enter the following schema registry credentials:

     • Schema Registry URLs
     • Schema Registry Key
     • Schema Registry Secret

5. Select a security protocol: PLAINTEXT or SASL/SCRAM.

   • If you choose SASL/SCRAM, enter your secret’s Username and Password.

6. Click Save & Test. Fivetran will take it from here and sync your AWS MSK data.

Related Contentlink

description Connector Overview

assignment Release Notes

settings API Connector Configuration

home Documentation Home