Initial Snapshot + Incremental Feed ingestion

Following the support ticket - #411678, we would like to raise a product feature request for native support of an Initial Snapshot + Incremental Feed ingestion pattern.

This is a broader enterprise ingestion requirement and is not specific to Azure SQL. It can apply where source systems, managed service providers, legacy platforms, SaaS applications or upstream data providers can supply:

an initial full snapshot or historical dataset; followed by
a X-day incremental-only data feed containing new, changed and/or explicitly deleted records.

The required Fivetran behaviour would be:

Load the initial snapshot as the destination baseline.
Treat subsequent datasets as incremental change feeds, not complete current-state snapshots.
Preserve records loaded previously when they are not present in a subsequent incremental feed.
Apply inserts and updates only for records supplied in the feed.
Process deletions only when they are explicitly identified.
Support sources with or without SCD2 attributes.

Where SCD2 data is supplied, the source may provide attributes such as effective_from, effective_to and current_flag, with an appropriate key identifying each version. For non-SCD2 sources, a standard primary or business key could be used.

Based on below support discussion, current state-comparison approaches such as Fivetran Teleport can interpret records missing from a later incremental extract as deleted because the source object is assumed to represent the authoritative current state. This makes an incremental-only feed difficult to use safely and can also introduce re-sync and downstream watermarking challenges.

We would therefore like Fivetran to consider a native Initial Snapshot + Incremental Feed mode, with configurable deletion handling such as:

explicit delete indicator;
INSERT / UPDATE / DELETE operation field;
source SCD2 attributes where available; or
no deletion unless explicitly supplied.

This capability would be particularly useful where customers cannot enable CDC/Change Tracking or require the upstream provider to maintain a persistent full-history replication table purely for Fivetran.

Could you please:

Raise this as a formal product feature request;
Confirm whether similar functionality already exists or is planned on the roadmap; and
Associate the request with our account/ticket for product prioritisation.

Happy to provide further use-case details if useful for the Product team. Thanks.

Regards,
Iman Setiawan

Community

Initial Snapshot + Incremental Feed ingestion

Comments