Skip to main content

Community

Other: Ability to filter data at row level before sync

Answered

Please sign in to leave a comment.

Comments

8 comments

  • Official comment
    Alexa Maturana-Lowe User

    Shil Joshi thanks for the request - as a part of our product philosophy, we're aiming for zero-configuration so we often make a single default choice of behalf of our users, such as delivering all the data instead of making this configurable so this request is counter to one of our core tenants. However, I'd be curious to understand why this is interesting to you - what are you trying to accomplish with filtering before the sync? 

     

    For us, we sometimes don't require all the data or we don't want this data to be accessible (for privacy concerns). We can handle this further down the process but it'll be super useful for us to do this inside Fivetran :)

    Hi @.... Making default choices is great, but it would be very useful to enable users to override the default in this case and filter rows, the same way Fivetran already enables the filtering of columns and tables. As I mentioned on the similar ticket here, in the absence of this feature, we pay for the syncing of data that has no value to us.

    +1 on this one, we simply can't use Fivetran for privacy reasons

    I need this as well. I have a lot of data in my tables that is TEST data and I do not want it synced to my destination. My case is MySQL Binlog but would apply to others. Data is row-level - each row has an identifier that would signal whether or not it should be picked up.

    Upvote!

    It would be even better if we can do hashing at row level.

    I do agree with the fact that row level filtering would be a very useful option!

    We have a table of history data which has a lot of unnecessary rows and thus drastically increasing our MAR on certain months. As these are not of an interest to us, there is no point we pull it into our Data warehouse in the first place. For now we do not sync the particular table due to this issue. Being able to filter out the unnecessary rows based on a column filter would resolve this issue as we can extract just what we need and not pay for unnecessary data load.