Marketo Overview

Priority-first sync

We use priority-first sync to pull your activities data. Priority-first syncs fetch your most recent data first so that it’s quickly ready for you to use.

During your initial sync, Fivetran fetches your most recent seven days of data.

In all subsequent syncs, we continue to fetch your most recent data first. We also sync historical data until we reach the end of your activities data retention period. For example, if Change Score activities are retained for 90 days, we will sync 90 days’ worth of Change Score data.

Priority-first syncs allow us to keep your most recent data up to date while syncing your historical data over time.

BULK API

The latest version of our Marketo connector uses only BULK API to fetch activity data, because BULK API guarantees data integrity. This connector uses REST API to fetch all other data.

If you are using our latest Marketo connector, make sure you purchase sufficient BULK API quota from Marketo.

When Fivetran requests data from Marketo via BULK API, we can provide a start date and an end date for the range we want to sync using the sinceDatetime and untilDateTime query parameters. In this way, we fetch activities for a defined time range, save the state, and request the next time range. Because we can save state accurately, we preserve data integrity even if syncs fail or are interrupted.

Combined BULK and REST API

Earlier versions of our Marketo connector use REST API to fetch all data but activity data. These connectors use a combination of BULK API and REST API to fetch activities.

BULK API is more reliable and is the preferred method whenever possible. The BULK API part of the combined strategy works exactly the same as the BULK API-only strategy of the latest version of our connector.

When Fivetran uses REST API to request activity data from Marketo, we can request a start date using the sinceDatetime query parameter. However, unlike with BULK API, we don’t have the option to specify an end date, so we have to rely on sorted records to save the state. That is, when we see an activity, we store its timestamp in the state. We do this to keep track of the time until we have fetched the activities.

However, Marketo’s activities endpoint does not always provide the records in sorted order. When Marketo activity records are not in sorted order, we can’t save the state correctly. If the connection fails or we have to reschedule because we reach the API quota limit, we might end up saving the wrong state and missing some records.

Why don’t we simply wait to save the state until we have fetched all the activities instead of attempting to store the intermediate state? If we don’t store the intermediate state and the sync fails or is rescheduled, we have to start the entire sync from the beginning. Because Marketo has large amounts of data and its API is already very slow, that could take an extremely long time. Moreover, we could reach the API quota limit repeatedly without finishing the sync, and get stuck in a loop. Saving the intermediate state is a necessary, if imperfect, workaround to syncing through the Marketo REST API.

Semantic activity types

Semantic activity types define how changes propagate in Marketo. We fetch semantic activities from Marketo even if you don’t select them in the Schema tab of your connector’s dashboard. We can’t sync data for other tables (LEAD, PROGRAM_MEMBERSHIP) if we don’t fetch these activities. We use your BULK API to fetch the following semantic activities:

Activity types recommendation

Some activities trigger an automatic historical re-sync that impacts the sync speed of the connector. We do not recommend including the following email activity types:

Leads replication and Marketo Corona

Fivetran uses the updatedAt query parameter to retrieve only the new and updated data from the Leads API endpoint. However, this feature is only supported for those accounts that have enabled Corona.

To allow us to perform incremental replication of your leads data, contact Marketo to enable the feature for your account.

Sync strategy for PROGRAM_MEMBERSHIP table

When we sync the activity tables we obtain the program IDs from the activities. We then use the program IDs to fetch details about the programs. We individually check all the programs and then sync all the leads that belong to a specific program into the PROGRAM_MEMBERSHIP table.