transformation tables are rebuilt on every run instead of incrementally updated
Dear Fivetran team,
we are currently using the following Quickstart data models:
- google_ads__account_report
- facebook_ads__account_report
- linkedin_ads__account_report
- microsoft_ads__account_report
We noticed that these models appear to be fully recreated/materialized on each transformation run rather than incrementally updated. Example from the transformation logs:
5 of 5 START sql table model google_ads_reports.google_ads__account_report
5 of 5 OK created sql table model google_ads_reports.google_ads__account_report [SUCCESS 5787 in 0.97s]
This causes issues for downstream processing in our Snowflake destination, because the transformed reporting tables appear to be recreated on each run. As a result, table-level settings and metadata such as CHANGE_TRACKING get lost or reset, and downstream objects that rely on change tracking (e.g. dynamic tables) cannot consume the tables incrementally. Instead, they may need to reprocess or reinitialize based on the full table.
Is there a supported way to configure these Quickstart models to run incrementally (with a lookback window of 30 days for example)?
Our goal is to avoid full table recreation on every transformation run where possible, so that downstream processing can consume only new or changed records.
Thanks a lot for your support!
Warm Regards,
Martha
Please sign in to leave a comment.
Comments
0 comments