Question

Why do campaign_recipient and campaign_recipient_activitytables sync more rows than emails sent from Mailchimp?

Environment

Connector: Mailchimp

Answer

There are more email data synced than is sent from Mailchimp in a day because the connector is fetching older data than the previous day. The connector may still also be finishing syncing the historical email activity if this is soon after the historical sync.

In each sync, the Mailchimp connector will process email activities once all other endpoints are done. The email activities are used to fetch campaign_recipient and campaign_recipient_activity. As the email activities endpoint is extremely slow, we process email activities for up to 8 hours per sync and in subsequent syncs, we fetch the remaining data in the same way. We stop processing when the email activities processing exceeds 8 hours as we don't want the sync to run indefinitely until all email activities are synced.

For campaign_recipient the data is loaded only once. But for campaign_recipient_activity our implementation is currently non-incremental for the most part. We check if a campaign has any new activity (e.g., new clicks/opens), and if it does we fetch and upsert all data including previously upserted data.

This is highlighted at a high level in our Documentation. 

Cause

The slowness of the Mailchimp email activity endpoint.