Learn about how Fivetran syncs your data from your source into your destination.
Sync types and statuseslink
You must run a historical sync to begin using your connector. The first historical sync that Fivetran does for a connector is called the initial sync. During the historical sync, we extract and process all the historical data from the selected tables in the source. We then load data into the destination periodically.
After a successful initial sync, the connector runs in incremental sync mode. In this mode, only data that has been modified or added - also known as incremental changes - is extracted, processed, and loaded on schedule. We reimport tables when we aren’t able to fetch only incremental changes. We use cursors to record the history of the syncs.
In some cases, you may need to re-run a historical sync to fix a data integrity error. This is called a re-sync. We sync all historical data in the source tables and their columns according to your connector configuration.
Sync History chartlink
The Sync History chart visualizes the status of your syncs, how long they took, and how much data was synced each time. View your Sync History chart on the Status page of your Fivetran dashboard.
The chart breaks down your syncs into Extract, Process, and Load steps. You can view one hour, one day, or one week of sync history.
Each sync is represented by a colored bar. Hover over the sync bar to see your sync’s type, duration, start and end time, status, extract time, process time, load time, and data volume.
The bar’s color reflects the sync status:
|Green||Completed||Your sync finished running with no errors.|
|Blue||Incomplete||Your sync is currently running.|
|Purple||Rescheduled||Fivetran reschedules syncs in two situations:
|Gray||Unknown||Your sync has exhibited atypical behavior and has not completed for unknown reasons.|
|Red||Failed||Your sync could not finish running. The stage your sync failed in appears in light red.|
TIP: Learn how to troubleshoot failed syncs in our sync troubleshooting documentation.
For most of our database connectors, we provide more detailed data volume measurements for each sync phase in the sync history chart. These measurements help you understand exactly how much data we move through our pipeline in each phase. We define the data volume for each phase as follows:
- Extract Volume: Total volume of data we read from your source. This includes any data we filter out or don’t sync but read as part of our queries.
- Process Volume: Total volume of data we process before our deduplication process filters out data. Process volume also includes internal Fivetran metadata that we use during this phase.
- Load Volume: Total volume of data we upload to your destination after normalization.
For connectors that support priority-first sync, the forward and backward syncs are displayed as separate adjacent colored sync bars. When you hover over the sync bar, you will see ‘Priority-first sync’ specified as the sync type in the popover displayed.
Click View in the popover to see the historical sync progress for each table. The details include the date up which the data has been fetched and the total number of days of available data history to be fetched.
Sync frequency and schedulinglink
By default, Fivetran syncs your data from your source to your destination every six hours. If you want Fivetran to sync your data more or less often, you can adjust the sync frequency. The sync frequency defines the time period between the start times of two subsequent syncs. For example, if your sync frequency is 30 minutes, your next sync will start 30 minutes after the start of your last sync provided the latter successfully completed within 30 minutes from its start. The baseline sync start time for the first sync within the 24-hour cycle is midnight in the destination timezone.
IMPORTANT: When you add a new destination, Fivetran assigns it a fixed time offset. The offset can be any random value in minutes ranging from 0 to 60. It is derived from the destination ID hash. This offset is shared by every connector in the destination. This offset ensures that all Fivetran connectors do not launch at once, which would slow down the sync process. The offset value remains the same regardless of the set sync frequency.
Depending on your plan level, you can choose from the following sync frequencies:
- 5 minutes
- 15 minutes
- 30 minutes
- 1 hour
- 2 hours
- 3 hours
- 6 hours (default)
- 8 hours
- 12 hours
- 24 hours
Select your sync frequency on the Setup tab of your connector’s details page:
Alternatively, you can create a custom sync schedule using the Fivetran REST API (Standard Select, Standard, Enterprise and Business Critical accounts only). Learn how in our REST API documentation.
If a sync is already running at the time of the scheduled sync start, Fivetran postpones the scheduled sync to the next scheduled sync start time. Here’s an example for a connector with a 15-minute sync frequency and an offset value of 0 minutes:
|Sync Number||Start Time||End Time||Notes|
|3||08:30 08:45||08:55||Sync 3 is postponed because Sync 2 is still running. Sync 3 then starts at 08:45.|
Fivetran supports custom sync scheduling for connectors that sync once a day. You can choose what time your daily syncs start if you don’t want to use Fivetran’s default midnight starting time. If you don’t know how often your connector syncs, check the sync frequency on the Setup tab of the connector details page in your Fivetran dashboard.
Sync start times and offsetslink
Fivetran uses the following formula to calculate the actual start times:
actual sync start time = scheduled sync start time + remainder of (offset value / sync frequency)
For example, we calculate the actual start time of the second sync in a 24-hour cycle with the following values:
- the sync frequency is 15 minutes
- the offset value is 44 minutes
00:29 = 00:15 + 00:14 where
00:14 is the remainder of
00:44 / 00:15.
Thus, the actual sync start time is
Here is another example where we calculate the actual start time of the second sync in a 24-hour cycle with different values:
- the sync frequency is 15 minutes
- the offset value is 2 minutes
00:17 = 00:15 + 00:02 where
00:02 is the remainder of
00:02 / 00:15, which equals the offset value in this case.
In this example, the actual sync start time is
The following table provides further examples of sync start times, which depend on the relationship between the sync frequency and the offset value. For the sync start times presented in the table, we assume that all syncs are successfully completed by the start of the next sync.
|Scheduled Sync Times (Sync Frequency 15 minutes)||Actual Sync Times (Offset Value 2 minutes)||Actual Sync Times (Offset Value 44 minutes)||Actual Sync Times (Offset Value 15 minutes)|
NOTE: When you change the sync frequency, the next sync will follow the new sync schedule corresponding to the set sync frequency regardless of the last sync’s start time. For example, the sync frequency is set to 3 hours, the offset is 6 minutes, the last sync started at 15:06 and completed at 16:50. At 17:00 you set the sync frequency to 12 hours. The next sync will start at 00:06 rather than at 03:06.
For failed syncs, Fivetran calculates the start time for the next sync using the following formula:
time of the sync failure + 60 min OR sync frequency period where we use either
60 min or
sync frequency period, whichever value is less.
For example, if the sync failed at 13:38, and the sync frequency is 6 hours (360 min), then the start time of the next sync will be:
13:38 + 60 min = 14:38 since 60 min is less than 360 min.
For rescheduled syncs, the rules and formulas used to define the sync start time depend on the connector type. Although rescheduling is most commonly caused by Fivetran hitting API quotas in the source, causes of sync rescheduling vary depending on the connector type.
Checking the sync start time offset valuelink
You can use the sync frequency slider to check the sync start time offset value.
- On the Setup tab of the connector details page, set the sync frequency to 24 hours.
- Click Change.
- Open the Daily Run Time menu. The offset value is the minute value. For example, if the sync start times are 00:48, 01:48, 02:48, and so on, then the offset value is 48 minutes.
- Using the slider, set the sync frequency to the value you had before.
Syncing empty tables and columnslink
Most Fivetran connectors do not sync source tables and columns that have no data and don’t create them in your destination. However, Fivetran can create empty tables and columns in your destination for some connectors. See our Features documentation for a list of supported connectors.
NOTE: Under no circumstances will we sync an empty schema.