AWS MSK - PrivateLink Setup Instructions Beta

Follow our setup instructions to connect AWS MSK using PrivateLink to Fivetran.

MSK brokers with a single shared interface endpointlink

All brokers in the cluster are fronted with a single unique Network Load Balancer (NLB).

Connection overviewlink

After you set up the connector, Fivetran (using our accounts) talks to the brokers using their provisioned default Domain Name System (DNS) names:

1. We resolve the broker DNS name to the interface endpoint IP address inside our Virtual Private Cloud (VPC).
2. We initiate a TCP connection to the interface endpoint over the port you send to us (in the setup form).
3. The NLB listener within your VPC receives the connection in the port you specified.
4. The NLB listener’s corresponding target group transfers the request to one of the brokers registered to it. In response, the broker sends back the advertised DNS name and port (for example, 9001) to us.
5. We resolve the broker endpoint address to the interface endpoint IP and then initiate a connection to the same interface endpoint over the advertised TCP port (for example, 9001).

This connection is routed to the NLB listener for the TCP port (9001) and at the end goes to the broker.

Setup instructionslink

1. Modify the port that the MSK broker is advertising by executing the following command against each running broker. The following example command shows changing the advertised port on a specific broker b-1 to 9001. Replace the values of bootstrap-server, entity-name, CLIENT_SECURE, REPLICATION, and REPLICATION_SECURE.

   /bin/kafka-configs.sh \
   --bootstrap-server b-1.msk-pl-test.loqxk3.c23.kafka.us-east-1.amazonaws.com:9092 \
   --entity-type brokers \
   --entity-name 1 \
   --alter \
   --add-config advertised.listeners=[\
   CLIENT://b-1.msk-pl-test.loqxk3.c23.kafka.us-east-1.amazonaws.com:9001,\
   REPLICATION://b-1-internal.msk-pl-test.loqxk3.c23.kafka.us-east-1.amazonaws.com:9093,\
   REPLICATION_SECURE://b-1-internal.msk-pl-test.loqxk3.c23.kafka.us-east-1.amazonaws.com:9095]
   

2. Create target groups dedicated for each broker (B1, B2, etc.) with target type as IP, protocol as TCP, port as 9092, and in the same VPC as the MSK cluster. Register the respective MSK broker to each target group by its IP address.

3. Create an NLB that is enabled for the same subnets that the MSK brokers are in.

4. Create a TCP listener for every broker’s advertised port (9001, 9002) that forwards to the corresponding target group you created (B1, B2).

5. Create a VPC endpoint service configuration and specify your NLB.

6. Allowlist Fivetran’s AWS VPC Account ID (arn:aws:iam::834469178297:root) to allow access to your VPC endpoint service. Send the service name (VPCe) to your Fivetran account manager. For example, com.amazonaws.vpce.<region_id>.vpce-svc-xxxxxxxxxxxxxxxxx.

   TIP: To learn how to allowlistlist the Fivetran account ID, see AWS’ endpoint service permission documentation.

7. Allowlist the Fivetran VPC network for your network firewalls’ region (NACLs, Security Groups) to allow access to the service/source/destination.

   Cluster	VPC Network
   AWS Fivetran Connectors/Services us-east-1 North Virginia, USA	10.128.0.0/18
   AWS Fivetran Connectors/Services us-east-2 Ohio, USA	10.129.64.0/18
   AWS Fivetran Connectors/Services us-west-2 Oregon, USA	10.129.0.0/18
   AWS Fivetran Connectors/Services ap-southeast-2 Sydney, Australia	10.130.0.0/18
   AWS Fivetran Connectors/Services eu-west-2 London	10.131.0.0/18
   AWS Fivetran Connectors/Services eu-central-1 Frankfurt	10.132.0.0/18

8. Send the hostname of the service and the broker’s boot-strap server’s DNS entries to your Fivetran account manager. We will finish the setup on our side.

9. To activate the connection, accept the interface endpoint connection request from Fivetran. By default, connection requests must be manually accepted. However, you can configure the acceptance settings for your endpoint service so that any connection requests are automatically accepted.