Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Historic data becoming false on reload in Fivetran saas/hybrid connections

Answered
Safoora Jalali User

we are ingesting data from several sources which might purge old data later. We want to maintain old historic data in our end, resyncs should not mark existing data as false, and we should have capability to have duplicate data in destination. We are then refining the data and deduplicating it. We were able to achieve this in HVR using ‘restrict refresh’ condition 1=2 on target. We need some control in saas as well. Resyncs should not mark all existing data as false. 

Please sign in to leave a comment.

Comments

3 comments

Date Votes
  • Official comment
    Eric O'Connor Fivetranner

    Hi Safoora,

    In this workflow is it sufficient to ignore _fivetran_deleted? Or is there some reason you need to disable soft deletes entirely? 

  • Safoora Jalali User

    Hi Eric,

    I don't get notified when comments are added to the feature request. hence, the delay in response. 

    We dont want to ignore all deletes. We want to ignore data which is deleted in source as part of data purging. Like, for some table, purge policy can be 90 days, so data older than 90 days gets deleted.

    We get this delete in fivetran destination also, however we have other layers on top where we recover this data, by applying purge policy to ignore deletes /updates after 90 days.

    However, during resyncs, fivetran marks all existing data as deleted which is problematic

     

  • Safoora Jalali User

    Since, we have this capability in HVR by using restrict refresh, we want the same for saas as well