Transformations: Fivetran to enable dbt snapshots and add this in its integrated scheduling
It would be a very useful feature, if you enable
1. dbt snapshots in fivetran integrated scheduling
2. fivetran to take choice from users to run models/snapshots/tests using either dbt run or dbt build in its integrated scheduling
As models may contain snapshots, tests and requires dependancy setup among the models therefore, instead of running them one by one let dbt do by using a single command dbt build.
-
Fully agree with this! I love being able to kick off my models using integrated scheduling (DBT Core), and its a real shame I have to manage snapshots via deployment.yml and cron timelines to try and second guess when the connector will have finished syncing.
-
Being limited to 'dbt run' with integrated scheduling renders the Fivetran + dbt integration pretty much pointless. With dbt cloud, it's extremely easy to configure any dbt job to run on a cron. I can't think of a reason to use deployments.yml to setup scheduled jobs from Fivetran, if that is limited to just cron-based triggering.
We snapshot some of the data that gets loaded by Fivetran. These snapshots are critical for several downstream models. We effectively cannot run most of our models without first ensuring the snapshots have been updated.
I was hoping to be able to refresh these snapshots as soon as Fivetran is finished loading updates.
'dbt build' allows users to refresh models, seeds, tests, and snapshots on each execution. It is a critical command for many dbt projects nowadays. We never use 'dbt run' for any of our dbt jobs.
-
leave it to Fivetran to force users to switch from basic transformations to dbt and not even provide support for all of dbt's basic features like snapshots.
Please sign in to leave a comment.
Comments
3 comments