AWS MSK Setup Guide Beta

Follow our setup guide to connect AWS MSK to Fivetran.

Prerequisiteslink

To connect AWS MSK to Fivetran, you need an AWS MSK cluster.

Setup instructionslink

(Optional) Configure AWS PrivateLink BETAlink

IMPORTANT: You must have a Business Critical plan to use AWS PrivateLink.

AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. Learn more in AWS’ PrivateLink documentation.

Follow our AWS PrivateLink setup instructions to configure PrivateLink for your AWS MSK platform.

For more information about configuring MSK clusters with AWS PrivateLink, see AWS’ documentation.

IMPORTANT: In PrivateLink connections, you don’t need to expose brokers to public IP addresses. If you are using a PrivateLink connection, skip ahead to the Finish Fivetran configuration step.

Find ENI IDlink

1. Open the AWS MSK console and go to your cluster.

2. In the Brokers summary section, for each broker find the Attached ENI ID. Make a note of the ENI ID(s).

   

Associate addresslink

1. Open the EC2 dashboard and select Network Interfaces.

2. For every broker ENI ID you found in Step 2, select Actions > Associate address to associate an Elastic IP address.

   

3. Make a note of the associated Elastic IPs for each broker. You will need them to configure Fivetran.

Configure security groupslink

1. On the EC2 dashboard, select Security Groups.

2. In the Security Groups of the ENI IDs you found in Step 2, whitelist Fivetran’s IP addresses.

Set propertylink

Set the advertised.listeners property for the corresponding port of each broker to the Elastic IP associated with the ENI of the broker. Use dynamic configuration to configure the broker-level properties.

Finish Fivetran configurationlink

1. In the connector setup form, enter the Destination schema name of your choice.

2. Enter the dedicated Consumer Group for Fivetran.

3. Enter your Servers details using the <host>:<port> format. Use the Elastic IPs you found in Step 3 as <host>. For example, if the Elastic IP for the broker is 10.100.10.10 and the authentication is SASL/SCRAM, then enter the server name as 10.100.10.10:9096.

   NOTE: Click + to add more than one server details.

4. Select your message type: Json, Avro, Protobuf, or Text.

   • If you choose Json, select a sync type: Packed or Unpacked.

   • If you choose Avro or Protobuf, enter the following schema registry credentials:

     • Schema Registry URLs
     • Schema Registry Key
     • Schema Registry Secret

5. Select a security protocol: PLAINTEXT or SASL/SCRAM.

   • If you choose SASL/SCRAM, enter your secret’s Username and Password.

6. Select your connection method: Connect directly or Connect via PrivateLink. If you choose Connect via PrivateLink, Fivetran connects to your message brokers using AWS PrivateLink.

7. Click Save & Test. Fivetran will take it from here and sync your AWS MSK data.

Related Contentlink

description Connector Overview

assignment Release Notes

settings API Connector Configuration

home Documentation Home