Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Decouple schema drift detection from the data sync schedule

Completed
Raj Padnani User

Hello,

We’re seeing significant load from Fivetran Aurora MySQL → Snowflake connector due to  recurring metadata queries that inspect information_schema for schema drift detection. This (large and ugly) schema check query executes on the writer node of our primary database every time the data sync happens, and ranks among the top SQL statements in Aurora Performance Insights.

We understand this check ensures schema consistency before each incremental sync, but our schemas are extremely stable and DDL changes are far less frequent. For such workloads, running a full schema introspection every 5 minutes (sync frequency) adds unnecessary overhead to the primary database without improving data integrity.

We’d like to request a configuration enhancement or internal flag to:

a) Decouple schema drift detection from the data sync schedule, or
b) Adjust its cadence (e.g., hourly or daily), independent of the sync interval.

This change would significantly reduce read I/O on Aurora while maintaining full sync reliability. A flag such as SCHEMA_INTROSPECTION_INTERVAL or equivalent would be ideal!

Happy to discuss this with your product support and to collaborate or test if this is available internally or is under consideration.

Thanks,
Raj Padnani

Please sign in to leave a comment.

Comments

7 comments

Date Votes
  • Official comment
    Valentin Kulichenko Fivetranner

    Hi Raj,

    This change is now rolled out. MySQL binlog-based connections now detect schema changes incrementally instead of requesting the full schema on every sync. This should improve performance and reduce database load in the cases like yours.

    Thanks,
    Val Kulichenko, Fivetran Product Team

  • Rean Fivetranner

    Hi Raj,

    Thank you for sharing your feedback with us!

    We’ve heard similar feedback from other customers using Aurora MySQL particularly those with stable schemas and frequent syncs around the overhead of recurring schema drift detection queries. You're absolutely right: these checks are important for ensuring structural integrity between source and destination, but they can introduce unnecessary load in low-change environments.

    We’re actively exploring ways to optimize schema introspection and have a task planned in the upcoming quarter to investigate this issue. We'd love to partner with you and get your feedback once we have a proposed solution in place.

    Thanks again for raising this and helping us shape the product!

     

    Best, 
    Rean Rodrigues
    Product Manager, Databases

  • Raj Padnani User
    • Edited

    Hi Rean,

    Thanks for the update. We will be glad to collaborate with you, just let us know. 

    Raj

     

  • Raj Padnani User
    • Edited

    Hi Rean,

    It's been a while since I heard from you. Do you have any update on this feature request? It's been a couple of quarters now.

    Thanks,

    Raj

     

  • Valentin Kulichenko Fivetranner

    Hi Raj,

    We're currently working on an improvement that will derive schema changes from DDL events in the binlog, instead of doing a full refresh on every sync. This is not exactly what you've been proposing, but should address your problem (as long as you're using binlog as the update method). We are planning to roll this out within the next couple of months, I will let you know when it's out.

    Thanks,
    Val Kulichenko, Fivetran Product Team

  • Raj Padnani User

    Hi Val,

    Thanks for the update. Could you please confirm which version of fivetran this change is available in? I want to make sure our deployment is up-to-date with that version.

    Thanks,

    Raj Padnani

  • Valentin Kulichenko Fivetranner

    Hi Raj,

    There is not specific version - it's rolled out to all users, so you should see the change. But please let us know if something doesn't work as you expect.

    Thanks,
    Val Kulichenko, Fivetran Product Team