Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Initial Snapshot + Incremental Feed ingestion

Answered
Iman Setiawan Admin User
Following the support ticket - #411678, we would like to raise a product feature request for native support of an Initial Snapshot + Incremental Feed ingestion pattern.
This is a broader enterprise ingestion requirement and is not specific to Azure SQL. It can apply where source systems, managed service providers, legacy platforms, SaaS applications or upstream data providers can supply:
  • an initial full snapshot or historical dataset; followed by
  • a X-day incremental-only data feed containing new, changed and/or explicitly deleted records.
The required Fivetran behaviour would be:
  • Load the initial snapshot as the destination baseline.
  • Treat subsequent datasets as incremental change feeds, not complete current-state snapshots.
  • Preserve records loaded previously when they are not present in a subsequent incremental feed.
  • Apply inserts and updates only for records supplied in the feed.
  • Process deletions only when they are explicitly identified.
  • Support sources with or without SCD2 attributes.
Where SCD2 data is supplied, the source may provide attributes such as effective_from, effective_to and current_flag, with an appropriate key identifying each version. For non-SCD2 sources, a standard primary or business key could be used.
Based on below support discussion, current state-comparison approaches such as Fivetran Teleport can interpret records missing from a later incremental extract as deleted because the source object is assumed to represent the authoritative current state. This makes an incremental-only feed difficult to use safely and can also introduce re-sync and downstream watermarking challenges.
We would therefore like Fivetran to consider a native Initial Snapshot + Incremental Feed mode, with configurable deletion handling such as:
  • explicit delete indicator;
  • INSERT / UPDATE / DELETE operation field;
  • source SCD2 attributes where available; or
  • no deletion unless explicitly supplied.
This capability would be particularly useful where customers cannot enable CDC/Change Tracking or require the upstream provider to maintain a persistent full-history replication table purely for Fivetran.
Could you please:
  1. Raise this as a formal product feature request;
  2. Confirm whether similar functionality already exists or is planned on the roadmap; and
  3. Associate the request with our account/ticket for product prioritisation.
Happy to provide further use-case details if useful for the Product team. Thanks.

Regards,
Iman Setiawan

Please sign in to leave a comment.

Comments

1 comment

Date Votes
  • Official comment
    Rean Fivetranner

    Hi Iman,

    Thank you for outlining this use case and the required behavior. We’re evaluating support for third-party databases that don't allow change tracking, change data capture, and transaction log reading. We don’t have a committed timeline to share at this stage.

    In the meantime, you can consider using the Fivetran Connector SDK to implement this pattern in a custom connector. Your connector logic would load the initial snapshot, apply subsequent inserts and updates by primary key, and issue deletes only for explicit deletion events. 

    This approach requires developing and maintaining custom connector code, but gives you control over the ingestion and deletion semantics you’ve described.

    Best, 
    Rean Rodrigues - Fivetran Product Team