Connector Improvement: Customizable Table-Level Sync Intervals Within Connectors
Description: Some use cases require the ability to set different sync intervals for individual tables within the same connector. Currently, Fivetran applies the same sync frequency to all tables within a connector, which limits flexibility. Creating separate connectors for each sync frequency increases complexity and Database bloat since multiple schemas are created per connector.
Proposed Solution:
-
Table-Level Customization:
- Enable users to manage sync times directly at the table level.
- Allow specifying different sync intervals for specific tables within a connector.
-
Enhanced Connector Configuration:
- Extend the connector configuration options to include per-table sync settings.
- Provide an intuitive interface for users to define custom sync frequencies.
-
Impact and Benefits:
- Improved data synchronization efficiency.
- Reduced overhead from managing multiple connectors.
- Enhanced adaptability to diverse use cases.
-
Hi Koketso,
Do you have an idea of the business impact of this change? How much cost-savings/efficiency you are looking to gain by avoiding syncing all tables at a higher frequency? For example, if you have `table_a`, and `table_b` with a requirement that they be synced at 5 minute and 60 minute sync frequencies, respectively, in theory you could run both at the higher 5 minute frequency.
What is the cost of running this more aggressive schedule in your workflow?
Please reach out over email (eric@fivetran.com) if you'd like to discuss specific details outside the public forum.
Please sign in to leave a comment.
Comments
1 comment