There are four ways to connect Fivetran to your database:
- Directly by whitelisting Fivetran’s IP
- Using an SSH tunnel
- Using a reverse SSH tunnel
- Using AWS PrivateLink or Azure Private Link (Business Critical only)
Whitelist Fivetran’s IPlink
The fastest and easiest way to connect is to allow Fivetran’s IP direct access to your database port. For more information about how to do this, visit the setup guide for your database.
SSH Tunnellink
If it’s not possible to provide direct access to your database port, you can connect to Fivetran via an SSH tunnel. You can also choose this connection method for an added layer of security. To connect via an SSH tunnel, you need to setup an SSH tunnel server that has access to your database port. The tunnel server’s SSH port needs to be accessible from Fivetran’s IP. You’ll also need to create an SSH user for Fivetran.
IMPORTANT: Fivetran generates a unique public SSH key for each destination. We support multiple connectors on a single SSH tunnel depending on the data volume and network bandwidth.
Create SSH userlink
Log in to your SSH tunnel host and run the following commands:
-
Create group
fivetran
:sudo groupadd fivetran
-
Create user
fivetran
:sudo useradd -m -g fivetran fivetran
-
Switch to the
fivetran
user:sudo su - fivetran
-
Create the
.ssh
directory:mkdir ~/.ssh
-
Set permissions:
chmod 700 ~/.ssh
-
Change to the
.ssh
directory:cd ~/.ssh
-
Create the
authorized_keys
file:touch authorized_keys
-
Set permissions:
chmod 600 authorized_keys
-
Using your favorite text editor, add the public SSH key from the database setup page in your Fivetran dashboard to the
authorized_keys
file. The key must be all on one line. Make sure that you don’t introduce any line breaks when cutting and pasting. The public SSH key is generated uniquely for each Fivetran destination.
As an extra layer of security, Fivetran enables TLS on your SSH connection by default. We recommend that you keep TLS enabled unless you know it is safe to forgo it. To disable TLS, set the Require TLS through tunnel toggle to OFF.
IMPORTANT: If you set the Require TLS through tunnel toggle to OFF, Fivetran first attempts to connect over TLS inside the SSH tunnel. If this fails, Fivetran automatically retries the connection in clear text inside the SSH tunnel. You are responsible for configuring this option as per your corporate security policies.
Once the user is created, you’ll need to allow port access.
Allow port accesslink
Make sure that port access is allowed from:
- Fivetran’s IP to your tunnel server’s SSH port
- Your SSH tunnel server to your source database port
If your SSH server and database happen to be in AWS, you can follow the instructions below to configure port access.
AWS
-
To configure an SSH server in AWS, open the EC2 console and select Running Instances:
-
Select the instance you intend to use as an SSH tunnel:
-
Select the Security groups and then select default:
-
Select the Inbound tab.
-
Click Edit.
-
Fill in Fivetran’s IP and your SSH port (do not use a load balancer).
-
For VPC or EC2 classic, add a security rule:
-
Select SSH, enter Fivetran’s IP, and click Save:
-
To complete setting up your database connector, follow the setup instructions for your specific database. You can confirm your server’s SSH key by comparing the SHA 256 displayed when running the setup tests.
Reverse SSH Tunnellink
You can also connect Fivetran to your database via a Reverse SSH Tunnel if you are unable to provide direct port access to your instance.
To set up a reverse SSH tunnel to connect to Fivetran, contact Fivetran’s Technical Sales team and provide the following SSH keys:
- Fivetran user’s SSH public key
- Your SSH public key
- The high port you will use for the tunnel
- The IP address (CIDR notation) of your SSH bastion host
To generate your SSH public key, do the following on your SSH host:
-
Generate an SSH key pair. Make a note of the key path - you will need it to complete your setup.
IMPORTANT: Do not enter a password for the key file. When prompted to provide a password, hit Enter instead.
ssh-keygen
-
View the contents of the public key. Copy the public key and send it to Fivetran’s Technical Sales team along with the Fivetran user’s public key.
cat ~/.ssh/id_rsa.pub
Once we have both keys, you will need the following information to complete your setup:
- Username
fivetran
of the SSH tunnel user you created - Reverse SSH IP address (contact Fivetran Technical Sales to get this)
- SSH high port. It should be unique per connector instance and should not be a reserved port number (for instance, port
22
is reserved for SSH connections and port443
is reserved for HTTPS).TIP: For the SSH high port number, we recommend adding a single digit - usually
1
- as a prefix to the source database port. For example, if you connect SQL Server, your database’s default port is1433
. Therefore, we recommend using port11433
as the SSH high port for your first SQL Server connector, port11434
for your second connector, and so on. - Internal IP address or name of the local database host machine
- Internal open port for communication with the database host
- File path to the private key on the SSH host machine (this is normally
id_rsa.pem
or simplyid_rsa
)
Use the values above to replace the placeholder variables in the following script, then run it on the SSH host in a single line:
TIP: To track the progress of this script, remove the
-f
flag and add the-v
flag to enable verbose logging. Without the flag, you will not see confirmation when the script finishes running successfully.
autossh -M 0 -f -N -R <SSH_HIGH_PORT>:<LOCAL_DB_MACHINE_NAME_OR_IP>:<LOCAL_DB_MACHINE_PORT> <FIVETRAN_SSH_USERNAME>@<FIVETRAN_SUPPLIED_IP> -g -i <PATH_TO_PRIVATE_KEY> -o ServerAliveInterval=10 -o ServerAliveCountMax=1 -o ExitOnForwardFailure=yes
If you use this autossh
script again later for the same SSH high port, you need to terminate your original autossh
script before proceeding.
After establishing a successful Reverse SSH connection, enter the following into the Fivetran setup form for your database:
Field | Value | Description |
---|---|---|
Host | localhost | Allows your SSH host to handle port routing |
Port | { SSH high port } | e.g., 13306. The port that your SSH host will translate |
User | { Database user } | |
Password | { Database user’s password } | |
Database | { Database name } | The name of the database you want to replicate |
Connection Method | Connect via an SSH Tunnel | |
SSH Host | { IP Address } | Supplied by Fivetran |
SSH Port | 22 | |
SSH User | fivetran |
All fields in { brackets } must be replaced with your own values.
AWS PrivateLink BETAlink
AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. PrivateLink is the most secure connection method. Learn more in AWS’ PrivateLink documentation.
Fivetran uses PrivateLink to move your data securely between our system and your AWS-hosted or on-premises source. PrivateLink works differently depending on your source type:
- If your data source is hosted in AWS, Fivetran can connect to your source using a PrivateLink connection. We query and process the data from the source into our system.
- If your data source is hosted on-premises, Fivetran can use AWS Direct Connect to access your source data. AWS Direct Connect establishes a private network connection between your premises and an AWS VPC. We connect to that AWS VPC using a PrivateLink connection, then query and process the data from the source into our system. Learn more in AWS’ Direct Connect documentation.
You can also use PrivateLink with the following destinations:
As an extra layer of security, Fivetran enables TLS on your PrivateLink connection by default. We recommend that you keep TLS enabled unless you know it is safe to forgo it. To disable TLS, set the Require TLS when using PrivateLink toggle to OFF.
IMPORTANT: If you set the Require TLS when using PrivateLink toggle to OFF, Fivetran first attempts to connect over TLS. If this fails, Fivetran automatically retries the connection in clear text. You are responsible for configuring this option according to your corporate security policies.
Prerequisiteslink
To set up AWS PrivateLink, you need:
- A Fivetran instance configured to run in AWS
- An AWS-hosted (EC2, RDS, or S3 only) or on-premises* source in one of our supported regions
* Your on-premises source must be one of our supported databases. See a complete list in our Databases documentation.
How you connect AWS PrivateLink to your source depends on whether your source is hosted in AWS or on-premises. Follow the setup instructions for your source type below.
Configure PrivateLink for AWS-hosted sourcelink
We support connecting to the following AWS-hosted sources using PrivateLink:
NOTE: Third-party managed database services, such as MongoDB Atlas, may not be supported. Contact your Fivetran account manager to confirm if your managed database service is supported.
AWS Host | Supported Connectors |
---|---|
Amazon Aurora | MySQL PostgreSQL |
Amazon EC2 | MariaDB MySQL MongoDB Oracle PostgreSQL SQL Server DynamoDB |
Amazon RDS | MariaDB MySQL Oracle PostgreSQL SQL Server |
Other | Amazon S3 AWS Lambda Salesforce |
NOTE: Amazon S3 does not require any configuration. If your S3 bucket is in the same region as your Fivetran account, your network traffic does not traverse the public Internet. The Amazon S3 Gateway Endpoints ensures that regional traffic stays within the AWS network.
You must have an AWS endpoint service configured for your source before you set up a PrivateLink connection with Fivetran. AWS endpoint services only work with network load balancers (NLB), so you must create an NLB inside your VPC if you do not already have one. The NLB receives requests from Fivetran and routes it to your source.
Since endpoint service configurations are out of Fivetran’s control, we recommend that you contact your AWS representatives for help setting up PrivateLink. However, we do provide the following high-level instructions based on how customers typically configure their data sources:
-
In your VPC, create an NLB for your data source and configure it for each subnet (availability zone) in which the service should be available. For help, see the Create an NLB section.
IMPORTANT: Skip this step if your data source is already running behind an NLB.
-
Create a VPC endpoint service configuration and specify your NLB.
-
Whitelist Fivetran’s AWS VPC Account ID (
arn:aws:iam::834469178297:root
) to allow access to your VPC endpoint service. Send the service name (VPCe) to your Fivetran account manager. For example,com.amazonaws.vpce.<region_id>.vpce-svc-xxxxxxxxxxxxxxxxx
.TIP: To learn how to whitelist the Fivetran account ID, see AWS’ endpoint service permission documentation.
-
Send the host name of the service/source to your Fivetran account manager. Fivetran will finish the setup on our side.
-
To activate the connection, accept the interface endpoint connection request from Fivetran. By default, connection requests must be manually accepted. However, you can configure the acceptance settings for your endpoint service so that any connection requests are automatically accepted.
Create an NLB
On a single IP service (EC2, non-RDS database, etc.)
To create an NLB on a single IP service, follow the instructions in AWS’ creating a network load balancer documentation.
On RDS
NLB can only route traffic to an EC2 instance, an IP address, or a Lambda function through target groups. Since RDS doesn’t have a dedicated IP address or EC2 instance ID, there are two different ways to configure an NLB to route traffic to an RDS database - using a port forwarding instance or using an RDS IP address. Follow the instructions below for your chosen method.
Using a port forwarding instance
-
You must deploy an EC2 instance that is configured to do port forwarding (accepting requests from the NLB and forwarding those requests to the RDS database). Here is a sample script that you can use to set up the EC2 port forwarding instance:
#!/bin/bash PREVLOGFILE=/root/ip.txt # Note the below section of the code is important in the event of a server restart. if test -f "$PREVLOGFILE"; then truncate -s 0 $PREVLOGFILE echo "State file $PREVLOGFILE has been emptied" fi python -m SimpleHTTPServer 801 & # NOTE: THIS PORT MUST MATCH THE... echo 1 -> /proc/sys/net/ipv4/ip_forward export RDS_ENDPOINT=<<PROSPECT RDS INSTANCE ENDPOINT>> #NOTE: DO NOT INCLUDE THE <<>> CHARACTERS, NO QUOTATION MARKS. export RDS_PORT=<<PROSPECTS RDS INSTANCE PORT>> #NOTE: DO NOT INCLUDE THE <<>> CHARACTERS, NO QUOTATION MARKS. iptables -t nat -A POSTROUTING -j MASQUERADE while true do LOGFILE=/root/ip.txt Current_IP=$(dig +short $RDS_ENDPOINT | tail -n1) #NOTE: THE "/ TAIL -n1" piece is critical to ensure only the IP address of the RDS instnce is picked. if [ $LOGFILE = "" ] ; then iptables -I INPUT -i eth1 -s $Current_IP -j ACCEPT echo $Current_IP > $LOGFILE else Old_IP=$(cat $LOGFILE) if [ "$Current_IP" = "$Old_IP" ] ; then echo "IP address has not changed ($Old_IP -> $Current_IP)" else iptables -t nat -D PREROUTING -p tcp --dport 80 -j DNAT --to-destination $Old_IP:$RDS_PORT iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination $Current_IP:$RDS_PORT sysctl net.ipv4.ip_forward=1 iptables-save echo $Current_IP > $LOGFILE echo "IP address has changed ($Old_IP -> $Current_IP)" fi fi sleep 5 done
-
Once you’ve finished setting up the port forwarding instance, configure the NLB listener and target group to route traffic to the portforwarder EC2 instance.
Using an RDS IP address
RDS provides an endpoint to access your database when you set up an RDS. This endpoint resolves to an IP address. AWS doesn’t recommend using this IP address, since it can change without notice. To work around this limitation, you can deploy a lambda function to periodically check the IP address and update the NLB target group when it changes.
To use the RDS IP address in your NLB target group, do the following:
-
Run the
nslookup
ordig
command with the domain name of RDS endpoint as the input to find the IP address:dig +short YOUR_RDS_DNS_ENDPOINT
-
Setup your NLB target group with the IP address.
-
Deploy a lambda function to periodically perform
nslookup
on the RDS endpoint to see if the IP address has changed and update the target group with the new IP address.
Configure PrivateLink for on-premises sourcelink
Contact your account manager for help setting up PrivateLink for your on-premises source. Our team will help you set up AWS Direct Connect to an AWS VPC, which Fivetran can connect to using PrivateLink.
Azure Private Link BETAlink
Azure Private Link allows Virtual Networks (VNets) and Azure-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. Learn more in Microsoft’s Azure Private Link documentation.
Fivetran uses Private Link to move your data securely between our system and your Azure-hosted, Azure Virtual Machine-hosted, or on-premises source. Private Link works differently depending on your source type:
- If your data source is hosted in Azure, Fivetran can connect to your source using a Private Link connection. We query and process the data from the source into our system.
- If your data source is hosted in an Azure Virtual Machine or on-premises, you can use Azure Private Link Service to connect Fivetran to your source. To use Azure Private Link Service, you must create an Azure Private Link Service with port forwarding VMs. The VMs relay the network traffic into your Azure VNet, which then connects to your source through Azure ExpressRoute.
You can also use Private Link with the following destinations:
- Azure SynapseBETA
- SnowflakeBETA
As an extra layer of security, Fivetran enables TLS on your Private Link connection by default. We recommend that you keep TLS enabled unless you know it is safe to forgo it. To disable TLS, set the Require TLS when using PrivateLink toggle to OFF.
IMPORTANT: If you set the Require TLS when using Private Link toggle to OFF, Fivetran first attempts to connect over TLS. If this fails, Fivetran automatically retries the connection in clear text. You are responsible for configuring this option according to your corporate security policies.
Prerequisiteslink
To set up Azure Private Link, you need:
- A Fivetran instance configured to run in Azure
- An Azure-hosted, Azure Virtual Machine-hosted, or on-premises* source in one of our supported regions
* Your on-premises source must be one of our supported databases. See a complete list in our Databases documentation.
How you connect Azure Private Link to your source depends on whether your source is hosted in Azure, in an Azure Virtual Machine, or on-premises. Follow the setup instructions for your source type below.
Setup instructions for Azure-hosted sourceslink
We support connecting to the following Azure-hosted sources using Private Link:
- Azure Blob Storage
- Azure MariaDB
- Azure MySQL
- Azure PostgreSQL
- Azure SQL Database
- Azure SQL Managed Instance
To establish a Private Link connection, do the following:
- Verify that your Azure-hosted source supports Private Endpoint in Microsoft’s Private Endpoint documentation.
- Contact your Fivetran account manager and provide your Azure-hosted service’s fully-qualified resource ID, including the resource name and resource type of your Azure-hosted source.
- Wait to receive Private Endpoint request details from Fivetran. We create a Private Endpoint using your resource ID, type, and subresource. We then initiate a Private Link connection request as part of the Private Endpoint setup and share the details of that request with you.
- In the Azure Portal or CLI, verify and approve the Private Link connection request from Fivetran. Fivetran then finishes setting up Private Link for your Azure-hosted service on our side.
Setup instructions for Azure Virtual Machine-hosted or on-premises sourceslink
How you set up Private Link for your Azure Virtual Machine-hosted or on-premises source depends on whether or not your source already has an Azure Private Link Service.
If your source is already configured with a Private Link Service, follow the setup instructions for Azure-hosted sources to complete your Fivetran setup.
If you do not have a Private Link Service yet, you will need to configure one before establishing a Private Link connection by following the steps below:
NOTE:
- You must have a VNet with access to your sources.
- If you are configuring your on-premise sources and they aren’t connected to Azure yet, follow the instructions in the Azure ExpressRoute documentation to connect to Azure.
- If you are configuring your Azure Virtual Machine-based sources, ensure that they are accessible from the VNet where the Private Link Service Load Balancer will be running.
- A single Azure Private Link Service can support one or multiple sources at the same time with different ports mapped to corresponding data source IP addresses. You will need to enter these ports in the Fivetran connector setup form.
- Inside the VNet, create a Standard Load Balancer with a NIC-based backend pool that has access to your data sources. Configure a health probe and a load balancer rule with the ports that your connectors will use. Learn how in Azure’s Create a load balancer guide.
- Create a Private Link Service associated with the load balancer that you created in step 1. Learn how in Azure’s Create a private link service guide.
- Create one or more Virtual Machines and place them in the backend pool of the load balancer you created in step 1.
- Enable IP forwarding on each of the backend Virtual Machine network interfaces. Learn how to enable IP forwarding in Azure’s Enable IP forwarding documentation.
- For security purposes, ensure that no public IP addresses are configured for the Virtual Machines. Private Link connections use only private IP addresses.
- Log in into each Load Balancer-backend Virtual Machine and run the script below. The script configures Network Address Translation (NAT), which forwards network packets from SLB-backend Virtual Machines to your data sources. The script below configures a single port forwarding, but feel free to add additional sets of iptables
PREROUTING
andPOSTROUTING
rules for additional ports:
#!/bin/bash
SOURCE_PORT=<local_port>
DESTINATION_IP=<destination_server_ip_address>
DESTINATION_PORT=<destination_server_port>
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -F
iptables -t nat -F
iptables -X
iptables -t nat -A PREROUTING -p tcp --dport ${SOURCE_PORT} -j DNAT --to-destination ${DESTINATION_IP}:${DESTINATION_PORT}
iptables -t nat -A POSTROUTING -p tcp -d ${DESTINATION_IP} --dport ${DESTINATION_PORT} -j SNAT --to-source $(hostname -i)
For additional help, read Microsoft’s Private Link Service documentation or contact your Fivetran account manager.