Understanding the intersection of Fivetran’s MongoDB connector and MAR.
Connector Sync Strategy and Options
- All MongoDB replication is done by utilizing the system generated _id primary key field in conjunction with changestream from the systems Oplog.
- The initial historical sync will replicate the selected Collections into your destination warehouse. This initial historical sync does NOT contribute toward MAR and is provided free for all new connector creations.
- If additional Collections need to be introduced at a later time, their initial historical load will not contribute to MAR.
What Contributes Towards MAR?
There are 5 primary actions that will contribute to MAR:
- A new document is added to an existing collection.
- A document is removed from an existing collection.
- A field is updated in an existing document.
- A field is removed in an existing document.
- A field is added to an existing document.
Managing MAR for this Connector
- Initial Historical syncs do not contribute to your Monthly Active Row count.
- Historical re-syncs and incremental syncs do contribute to MAR. A full re-sync completely overwrites the data in your destination with new data from your source
- Historical re-syncs can be triggered by BOTH the customer or Fivetran under explicit guidance from the customer.
- Monitoring data integrity issues and performing re-syncs to fix them when needed is part of the Fivetran managed service. We only perform, or ask you to perform, re-syncs when it’s absolutely necessary.
- If a collection was imported initially and then unchecked and was left for a period of time that exceeds the Oplog retention period, the collection will need to be re-synced which will generate MAR.
- Ensure that your Oplog retention period/size is set appropriately to allow for enough Oplog to incrementally sync collections in the event of a service outage or transient network connectivity error. In the event that the connector fails to connect to your instance for a period longer than the Oplog retention period (or at which point the Oplog increases above the specified size), we will need to initiate a re-sync which will increase MAR.
- It is recommended that you do not pause the connector for a period that is longer than the Oplog retention period/size as doing so will introduce the need to re-sync as above.