Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Transformations: Insert only operation

Answered
Subbarao Bandi User

It would be great if fivetran can support insert only operation on destination tables.

CDC increase billing cost on big fat tables in destination.

Different query engines have different query plans, so cost will increase.

with Insert only operation, no need to do any lookups to update/merge the data, just insert the changes from source into destination tables.

as fivetran adds default columns along with one extra columns would suffice to identify when the change took place at source

 

 

Please sign in to leave a comment.

Comments

3 comments

Date Votes
  • Official comment
    Jimmy Fivetranner

    Hey Subbarao,

    Interesting idea! What would be the difference in cost to your team with support for purely append-only?

    Jimmy

  • Joao Lisboa User

    This would be a great addition to Fivetran product, especially for high-volume immutable data!

     

    > Interesting idea! What would be the difference in cost to your team with support for purely append-only?

    Jimmy for destinations like Snowflake, it could make a massive difference in the compute required to process inserts in comparison with merges/upserts (typically expensive operations), and that would certainly reflect in the overall warehousing costs for these connectors.

  • Matthew Nizol User

    This would be extremely valuable for our team, in particular if the append-only mode included a `_fivetran_change_type` (insert, update, delete) column such that it represented an append-only replica of a change feed.