Introduction
In general, your sources generate data - new and modified data. We incrementally sync those changes regularly. As an example, let's say that your source generates 10,000 records every day that have to be created or updated in your destination, and assume your sync frequency is 24hrs.
Number of changes: 10,000 | Sync frequency: 24 hours
- It will take longer to sync
- There is more data exposed to failures (the likelyhood of failures is low, but should they occur, there is more data on the run)
- There will be more strain on the API & destination as you'll need to hit both more frequently in a smaller time window
Let's think about the same situation, only this time on a higher sync frequency, let's say 15 minutes. On average you'll roughly sync 100 records every day, since there are roughly 96 slots of 15 minutes in 24 hours.
Number of changes: 10,000 | Sync frequency: 15 minutes
- Incremental syncs will take less time to complete. Data will be up to date early and fast.
- If there is a failure, we only need to recover from 100 records. A failure can be due to the source service being down, a network problem, etc.
- Load on the source connection (API, queries, writing to destination) is reduced as it is spread across a wider time range.
Recommendation
There is no standard way to define the best sync frequency. You need to study this case by case within your company and for each connector. Overall, you might want to answer the following questions.
- How up-to-date does your data need to be?
- What is the business impact for you in case of a downtime?
- Would the API rate limits in your source impact the syncing period?
- Are there any destination costs that need to be taken into account?
Our general recommendation is to keep sync frequencies high in order to spread the load and minimize failure impact. The MDS is a complex environment with lot's of moving gears. A high frequency means fresh data, lower incremental sync times, and best UX amongst your teams.
Important: Your connectors' sync frequency has no impact on your Fivetran bill.
Note: This is best practice. Depending on the use case its impact might be negligible or substantial. You can consider this yourself depending on your data pipelines and needs of course.