Connector Improvement: Create/alter a table column without triggering a full table reloadPlanned
We have very large MySQL tables which require a very long time to load through the FiveTran ETL Process. We would like to make schema changes to this table, add columns, but are not able to because it will trigger a full re-load which will take days.
Hi Dan and Brett,
Thank you for here feedback here! We are currently working on a feature to allow DDL changes for MySQL connectors without requiring a re-sync of the connector. This feature would auto-populate default values and ensure that subsequent syncs fill the right data for each column.
Our current target is to release this 2-3 months from now, so we should have a private preview available by December 2021 or January 2022. Please stay tuned for more details on this project!
To add to this, we would also like to be able to add new columns to a table's schema definition without backfilling the target with a full table resync.
We are unable to add new columns to existing tables without incurring massive MAR costs. There needs to be greater control around when full table resyncs occur and this is one of them. These automatic full resync scenarios limit confidence in adopting Fivetran at significant scale as there are many scenarios that could result in huge unexpected bills.
Will there be a similar feature for SQL Server connectors?
This is great news and much needed for our scalability! Kevin Kim will this be available to ALL connectors (MySQL, SQL Server etc.)?
We are beginning with MySQL and Oracle, but will continue building this for the remaining databases.
Please sign in to leave a comment.