MongoDB Sharded Cluster Setup Guide

Follow these instructions to connect your MongoDB sharded cluster to your destination using Fivetran.

Prerequisiteslink

To connect your MongoDB database to Fivetran, you need:

• MongoDB version 2.6 to 4.2
• Your database host’s IP (e.g., 1.2.3.4) or domain (your.server.com)
• Your database’s port (usually 27017)
• TLS enabled on your database (if you want to connect to Fivetran directly)

To perform incremental updates with change streams PRIVATE PREVIEW, you need:

• MongoDB version 3.6 or above
• Read Majority read concern activated
• A replication set using replica set protocol version 1.

    cfg = rs.conf(); 
    cfg.protocolVersion=1; 
    rs.reconfig(cfg);

Note: Fivetran does not support Mongo tiers M0, M2, or M5 because MongoDB’s smaller managed tiers do not provide oplogs. We need oplogs to perform incremental updates. Mongo sharded cluster does not support SRV host. You must provide the MongoDB query router’s IP instead. We do support SRV host in Mongo Replica Sets.

Allow access link

This step allows Fivetran to access your database.

• Connect to MongoDB’s query routerlink

Connect Fivetran to the mongos query router node in your MongoDB sharded cluster.

• Allow port accesslink

Configure your firewall and/or other access control systems to allow incoming connections to all your mongo instances from Fivetran’s IP or your SSH tunnel server’s IP address.

• SSL supportlink

Our MongoDB connector supports encrypted connection to MongoDB over SSL if your database is setup to allow clients to connect without a certificate.

Create userlink

Create an identical Fivetran user in every primary shard node and the mongos query router. Replace <username> and <password> with a username and password of your choice.

use admin
db.createUser({
  user: "<username>",
  pwd: "<password>",
  roles: [ "readAnyDatabase", "clusterMonitor" ]
})

You can pick any username and password, but it must be consistent across all primary shard nodes and the mongos query router.

The Mongo shell command to create a user is db.AddUser() for versions 2.x.

If you want to find out which shards are connected to the mongos query router, use the command db.adminCommand({ listShards: 1 }).

(Optional) Configure AWS PrivateLink BETAlink

IMPORTANT: You must have a Business Critical plan to use AWS PrivateLink.

AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. Learn more in AWS’ PrivateLink documentation.

Follow our AWS PrivateLink setup guide to configure PrivateLink for your database.

Choose schema prefixlink

Fivetran maps the schemas we discover in your source database to your destination and prepends the destination schema names with a prefix of your choice. For example, if your original database contains schemas foo and bar and if you choose the prefix pre, then your destination schemas will be pre_foo and pre_bar.

Setup testslink

Fivetran performs the following tests to ensure that we can connect to your MongoDB sharded cluster and that it is properly configured:

• The Connecting to SSH Tunnel Test validates the SSH tunnel details you provided in the setup form. It then checks that we can connect to your database using the SSH Tunnel. (We skip this test if you aren’t connecting using SSH.)
• The Connecting to Host Test validates the database credentials you provided in the setup form. It then verifies that the database host is not private and checks that we can connect to the host.
• The Validating Certificate Test generates a pop-up window where you must choose which certificate you want Fivetran to use. It then validates that certificate and checks that we can connect to your database using TLS. (We skip this test if you aren’t connecting directly.)
• The Connecting to Database Test connects to your database instance and verifies that your database is at least version 2.6. It then checks that we can access the schemas in your database. Lastly, it verifies that we can access the oplog and that it contains at least 24 hours’ worth of changes.

NOTE: The tests may take a few minutes to finish running.

Connecting with MongoDB Atlaslink

In order to connect with MongoDB Atlas, follow the steps below:

Create an oplog userlink

1. Log in to the MongoDB Atlas dashboard.

2. In the Security section of the left menu, select Database Access.

3. Click New Database User.

4. Choose the password authentication method.

5. Enter a username and password for the new user.

6. In the Database User Privileges drop-down menu, select Grant Specific User Privileges.

7. Under Specific Privileges, add the following roles/privileges:

   • readAnyDatabase
   • read on the local database
   • clusterMonitor

8. Click Add User.

For general instructions, see MongoDB Atlas’s documentation.

Notelink

Set the oplog size so that it can retain at least 24 hours’ worth of changes. We recommend increasing the size to accommodate seven days’ worth of data. See this MongoDB tutorial to learn how to adjust the size of your oplog or this MongoDB Atlas Doc.

If you are using MongoDB version 4.0 or later, you must change the read concern to read majority. Learn how in MongoDB’s write and read concern documentation.

Related Contentlink

description Connector Overview

account_tree Schema Information

assignment Release Notes

settings API Connector Configuration

home Documentation Home