Question

Why is the initial historical sync taking so long?

Environment

All Connectors

Answer

There are multiple approaches to check why the historical sync may be taking a long time and if it is in a healthy state. Check the different options below to better understand the sync and spot signs of potential issues.

Dashboard Status Bar

Check the sync status bar in the dashboard to see if, after every 6-hour extract phase, there are process and load phases. This will confirm that the sync is not stuck and is progressing successfully.

Priority First Sync Rescheduling

If you see the sync is getting rescheduled, this is normal behavior for priority-first sync connectors. It confirms that the connector is loading the latest data to the warehouse and is then syncing the historical data. It ensures that your latest data is ready to use in the warehouse.

Schema Config Warnings

In the Schema section of the Connector Dashboard, you can hover over the connector table. When you do this it may show an Alert symbol and Alert information. This can indicate if the table will slow down a sync.

Example:

<//p>

Check Progress and Get Completion Estimation for tables

1. 1. Get the count of all records in the warehouse table: SELECT COUNT(*) FROM <table>
   2. Get the count of records present in the source table or endpoint:
      • For a DB connector source table, use a query like SELECT COUNT(*) FROM <table>
      • For an application connector, go to Source Application and identify the count of records for the data corresponding to the warehouse table
   3. Get the count of records in the Warehouse table for yesterday: SELECT COUNT(*) FROM <table>
WHERE _fivetran_synced = '<yesterdays_date>'
   4. Calculate an estimated remainder of sync time:
      (Warehouse records count - Source records count) / Number of records synced yesterday

If there is still a concern about the historical sync time, please submit a Support ticket. Include information on the checks you have made to understand the sync.

Cause

• A large amount of data at the source.
• Slow retrieval of data from the source API due to API limitations (Rate limits etc.).
• Specific tables take a very long time to sync.
• The connector is in priority-first sync.