Follow our setup guide to connect AWS MSK to Fivetran.
To connect AWS MSK to Fivetran, you need an AWS MSK cluster.
(Optional) Configure AWS PrivateLink BETAlink
IMPORTANT: You must have a Business Critical plan to use AWS PrivateLink.
AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. Learn more in AWS’ PrivateLink documentation.
Follow our AWS PrivateLink setup instructions to configure PrivateLink for your AWS MSK platform.
For more information about configuring MSK clusters with AWS PrivateLink, see AWS’ documentation.
IMPORTANT: In PrivateLink connections, you don’t need to expose brokers to public IP addresses. If you are using a PrivateLink connection, skip ahead to the Finish Fivetran configuration step.
Find ENI IDlink
Open the AWS MSK console and go to your cluster.
In the Brokers summary section, for each broker find the Attached ENI ID. Make a note of the ENI ID(s).
Open the EC2 dashboard and select Network Interfaces.
For every broker ENI ID you found in Step 2, select Actions > Associate address to associate an Elastic IP address.
Make a note of the associated Elastic IPs for each broker. You will need them to configure Fivetran.
Configure security groupslink
On the EC2 dashboard, select Security Groups.
advertised.listeners property for the corresponding port of each broker to the Elastic IP associated with the ENI of the broker. Use dynamic configuration to configure the broker-level properties.
Finish Fivetran configurationlink
In the connector setup form, enter the Destination schema name of your choice.
Enter the dedicated Consumer Group for Fivetran.
Enter your Servers details using the
<host>:<port>format. Use the Elastic IPs you found in Step 3 as
<host>. For example, if the Elastic IP for the broker is
10.100.10.10and the authentication is
SASL/SCRAM, then enter the server name as
NOTE: Click + to add more than one server details.
Select your message type: Json, Avro, Protobuf, or Text.
If you choose Json, select a sync type: Packed or Unpacked.
If you choose Avro or Protobuf, enter the following schema registry credentials:
- Schema Registry URLs
- Schema Registry Key
- Schema Registry Secret
Select a security protocol: PLAINTEXT or SASL/SCRAM.
- If you choose SASL/SCRAM, enter your secret’s Username and Password.
Select your connection method: Connect directly or Connect via PrivateLink. If you choose Connect via PrivateLink, Fivetran connects to your message brokers using AWS PrivateLink.
Click Save & Test. Fivetran will take it from here and sync your AWS MSK data.