Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Create/alter a table column without triggering a full table reload

Planned
Dan Dolinshek User

We have very large MySQL tables which require a very long time to load through the FiveTran ETL Process. We would like to make schema changes to this table, add columns, but are not able to because it will trigger a full re-load which will take days.

Please sign in to leave a comment.

Comments

5 comments

Date Votes
  • Official comment
    Kevin Kim Fivetranner

    Hi Dan and Brett,

    Thank you for here feedback here! We are currently working on a feature to allow DDL changes for MySQL connectors without requiring a re-sync of the connector. This feature would auto-populate default values and ensure that subsequent syncs fill the right data for each column.

    Our current target is to release this 2-3 months from now, so we should have a private preview available by December 2021 or January 2022. Please stay tuned for more details on this project!

  • Brett Postin User

    To add to this, we would also like to be able to add new columns to a table's schema definition without backfilling the target with a full table resync.

    We are unable to add new columns to existing tables without incurring massive MAR costs. There needs to be greater control around when full table resyncs occur and this is one of them. These automatic full resync scenarios limit confidence in adopting Fivetran at significant scale as there are many scenarios that could result in huge unexpected bills.

  • Brett Postin User

    Hi Dan,

    Will there be a similar feature for SQL Server connectors?

    Thanks

  • Dan Dolinshek User

    This is great news and much needed for our scalability! Kevin Kim will this be available to ALL connectors (MySQL, SQL Server etc.)?

  • Kevin Kim Fivetranner

    We are beginning with MySQL and Oracle, but will continue building this for the remaining databases.