Skip to main content
Documentation
Feature Requests Trust Center Status Page

Community

Transformations: Trigger tables and New Data Sync for dbt Transformations

Nikita Palchinskiy User
  • Edited

Hello Fivetran team!

Currently, dbt transformation Fully integrated scheduling triggers the transformations every time the connector is synced. And it syncs on schedule, not on the exact source tables update, and the transformations run every time by the same schedule. 

Could New Data Sync and Trigger tables (as for Basic SQLs) be added to the dbt Transformation Schedule, triggering it at the end of the sync only if the sync has updated any of the selected tables, when fresh data arrives from Fivetran?

Thank you!

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Ruud Geerlings User

    Common transformations that are integrated with connectors are executed only when new data is processed. This should also be the case for the dbt transformations that are integrated with connectors.

    Running a dbt transformation when there is no new data available is not logical.

  • Ruud Geerlings User

    Because the DBT transformation runs at the sync schedule and not on new data arrival there will be costs/credits involved on the database level. Because even though nothing must be transformed (pre and post are same) the transformation runs. These extra costs will be significant when the related SQL is complex and there is a frequent sync and there is only one daily data refresh done.
    Example: Sync of connector is every hour. The exact data delivery is once a day so 23 times there is no new data found. But is runs the DBT transformation 24 times. The execution costs of the SQL on the (cloud) database is therefore charged 24 times. The total database costs/credits increase is significant!

    This is really a showstopper to use Fivetran in the future!