Follow these instructions to connect your MongoDB sharded cluster to your destination using Fivetran.
Prerequisiteslink
To connect your MongoDB database to Fivetran, you need:
- MongoDB version 2.6 to 5.0
- Your database host’s IP (e.g.,
1.2.3.4
) or domain (your.server.com
) - Your database’s port (usually
27017
) - TLS enabled on your database (if you want to connect to Fivetran directly)
NOTE: We don’t support SRV host in MongoDB sharded clusters. You must provide the MongoDB query router’s IP address.
Allow access link
This step allows Fivetran to access your database.
-
Connect to MongoDB’s query routerlink
Connect Fivetran to the mongos
query router node in your MongoDB sharded cluster.
-
Allow port accesslink
Configure your firewall and/or other access control systems to allow incoming connections to all your mongo instances from Fivetran’s IP or your SSH tunnel server’s IP address.
-
SSL supportlink
Our MongoDB connector supports encrypted connection to MongoDB over SSL if your database is setup to allow clients to connect without a certificate.
Create userlink
Create an identical Fivetran user in every primary shard node and the mongos
query router. Replace <username>
and <password>
with a username and password of your choice.
use admin
db.createUser({
user: "<username>",
pwd: "<password>",
roles: [ "readAnyDatabase", "clusterMonitor" ]
})
You can pick any username and password, but it must be consistent across all primary shard nodes and the mongos
query router.
The Mongo shell command to create a user is db.AddUser()
for versions 2.x.
If you want to find out which shards are connected to the mongos
query router, use the command db.adminCommand({ listShards: 1 })
.
(Optional) Configure AWS PrivateLink BETAlink
IMPORTANT: You must have a Business Critical plan to use AWS PrivateLink.
AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. Learn more in AWS’ PrivateLink documentation.
Follow our AWS PrivateLink setup guide to configure PrivateLink for your database.
IMPORTANT: Fivetran cannot connect to Mongo Atlas using Privatelink.
Choose schema prefixlink
Fivetran maps the schemas we discover in your source database to your destination and prepends the destination schema names with a prefix of your choice. For example, if your original database contains schemas foo and bar and if you choose the prefix pre
, then your destination schemas will be pre_foo
and pre_bar
.
Setup testslink
Fivetran performs the following tests to ensure that we can connect to your MongoDB sharded cluster and that it is properly configured:
- The Connecting to SSH Tunnel Test validates the SSH tunnel details you provided in the setup form. It then checks that we can connect to your database using the SSH Tunnel. (We skip this test if you aren’t connecting using SSH.)
- The Connecting to Host Test validates the database credentials you provided in the setup form. It then verifies that the database host is not private and checks that we can connect to the host.
- The Validating Certificate Test generates a pop-up window where you must choose which certificate you want Fivetran to use. It then validates that certificate and checks that we can connect to your database using TLS. (We skip this test if you aren’t connecting directly.)
- The Connecting to Database Test connects to your database instance and verifies that your database is at least version 2.6. It then checks that we can access the schemas in your database. Lastly, it verifies that we can access the oplog and that it contains at least 24 hours’ worth of changes.
- The Checking Query Permissions Test queries your MongoDB database to do the following:
- Fetch all databases and check if we have
listDatabase
permissions - Check if we have permissions to query any database and if there are any non-empty collections present
- Check if we have permissions to query any collection
- Fetch all databases and check if we have
NOTE: The tests may take a few minutes to finish running.
Connecting with MongoDB Atlaslink
In order to connect with MongoDB Atlas, follow the steps below:
Create an oplog userlink
-
Log in to the MongoDB Atlas dashboard.
-
In the Security section of the left menu, select Database Access.
-
Click New Database User.
-
Choose the password authentication method.
-
Enter a username and password for the new user.
-
In the Database User Privileges drop-down menu, select Grant Specific User Privileges.
-
Under Specific Privileges, add the following roles/privileges:
readAnyDatabase
read on the local database
clusterMonitor
-
Click Add User.
For general instructions, see MongoDB Atlas’s documentation.
Noteslink
Set the oplog size so that it can retain at least 24 hours’ worth of changes. We recommend increasing the size to accommodate seven days’ worth of data. See this MongoDB tutorial to learn how to adjust the size of your oplog or this MongoDB Atlas Doc.
If you are using MongoDB version 4.0 or later, you must change the read concern to read majority. Learn how in MongoDB’s write and read concern documentation.
Make sure that all the nodes in your sharded cluster are active because we fetch incremental data using oplogs from all the nodes.
Related Contentlink
description Connector Overview
account_tree Schema Information
settings API Connector Configuration