Skip to main content
Documentation
Feature Requests Onboarding Trust Center Status Page

Community

Add option to automatically include (or exclude) new columns

Completed
Derek User

Fivetran currently has the option to automatically include new schemas and tables in the sync, however when selecting specific columns to sync, new columns are still added by default.

I have a use case where a source table has 500+ columns, more are created on a regular basis, and I only want to sync 13 of them - no more and no less. When a new column is created in the source, Fivetran automatically adds it to the sync.

This isn't the worst problem (and much better than the alternative of not syncing when you want it to!), however it would be ideal to have the option to sync or not sync new columns in a given table.

Please sign in to leave a comment.

Comments

18 comments

Date Votes
  • Official comment
    Amy Peterson Fivetranner
    • Edited

    Great news-- we now have this feature available in beta! Under the Schema tab for your connector, you can now define how to handle schema changes:

    You can also opt in to notifications when there are new columns available. We'll be rolling this feature out to all accounts over the next few weeks. If you'd like to have it enabled asap, please reach out to your customer success manager. Release note can be found here.

  • Mahdi Kefayati User

    This is really important to us as well because it can result in sensitive data leaking into data warehouse with a very different access control policies are applied.

  • Brent Adair User

    This feature is critical to prevent unexpected leaks of sensitive data. It's useful to have the ability for new columns to be added automatically. But that same feature becomes a MASSIVE RISK AND LIABILITY for data sources and tables that contain sensitive data. It is imperative that customers have the ability to configure when we want that behavior and when we don't. 

    This gap in your product requires a very painful and error-prone manual process to update our Fivetran config *every time* a column is added in a source table that is enabled for sync. 
     
    Note that the critical connector for us is NetSuite Suite Analytics.
     
    We need this fix asap. Please prioritize accordingly.
  • Nick Cordella User

    This would be very helpful to me as well. Once concern I have is that including fields by default makes it harder to manage MAR. Consider if my Salesforce team is adding fields that have a high rate of change but that shouldn't be synced to my warehouse destination. I would somehow need to keep a lookout for them in the Salesforce UI and deselect them as they are created in order to prevent those rows from being "active" via the extra fields. 

  • Abdelmoughit Errachid User

    Hi,

    This is a critical feature, we will not be able to proceed without, as

    - it can incur explosive unexpected costs
    - it can include columns in the warehouse with sensitive info, that should be only included after evaluating them manually

  • Kevin McCarthy User
    • Edited

    Hello, 

    I just wanted to add to this. The feature would be incredibly valuable to us. We have many tables coming in through the RDS MySQL connector with a large row count that have columns added and deleted. Most of the time, we are not interested in these added columns, but the resync that they trigger causes a huge jump in MAR. Ideally, I would want to be able to have an option to ignore these new columns or add them to the destination table but populate as null until I determine they are needed. 

  • Amy Peterson Fivetranner
    • Edited

    Hi Derek,

    Thanks for the request and the context! This is being actively worked on and should be available in your dashboard by June. 

    Amy

  • Derek User

    That's great, thanks so much Amy Peterson for the lightning-fast response.

  • Amy Peterson Fivetranner
    • Edited

    Hi all, thanks for the great feedback here! I wanted to provide a timing update. We expect to have the "automatically block new columns" functionality in private preview later this month (May 2021), but only for a subset of our connectors (most application and file connectors). Database connectors and connectors that behave similarly to databases (including Salesforce and Netsuite) will take a bit longer to introduce this functionality. We expect a July/August timeline for those connectors. Thanks!

    Amy

  • Amy Peterson Fivetranner
    • Edited

    Hi everyone, I wanted to provide another timing update. Unfortunately we have run into a few blockers on this project so our adjusted timeline for offering automatic column blocking is now August/September. Thanks for your patience!

    Amy

  • Fiona Zhao User

    Hi Amy, 

    We'd love to use this feature as well. Wondering if this is table-level setting or schema/connector level setting? Also, is it supported in Fivetran API?

    Thanks!

  • Amy Peterson Fivetranner

    Hi Fiona Zhao! This will be a connector-level setting and will be available via our API as well.

  • Fiona Zhao User

    Hi Amy and Fivetran team,

    Is this feature out ?

  • Amy Peterson Fivetranner

    Hi Fiona Zhao, 

    We're hoping to wrap up development of this feature by the end of the month and will reach out as soon as it's available in Private Preview. Thanks all for your patience as we build this out! 

    Amy

  • Brett Postin User

    Another one for this. Whilst automatically resyncing tables for new columns might be convenient in some scenarios, for Fivetran adoption at scale and the unexpected MAR costs it is terrifying.

    Greater control around when full table resyncs are allowed is a must for Fivetran at scale.

  • Matt Britton User
    • Edited

    We have a similar use case to some other folks on here: we would like to prevent new columns from being automatically synced to our data warehouse so we can avoid accidental exposure of PII.

    So this feature would be very helpful. Bonus points if Fivetran could notify me when the incoming schema includes a new column!

    Please notify me when this feature hits private preview.

  • Matt Britton User

    I'm still not seeing this - is it available when pulling from Postgres and sending to Snowflake?

  • Amy Peterson Fivetranner

    Matt Britton it's possible this feature hasn't hit your account yet as we are still in the process of rolling it out. To have it enabled ASAP, please reach out to your customer success manager. Thanks!