Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Transformation (dbt) triggered on git push

Planned
Chip Ach User

Because running our dbt transformations takes time and resources on the destination, we only trigger the entirety of our transformations once a day in our deployment.yml. (We have a more limited set of actions scheduled to run more frequently.) Most of the time, they do nothing because nothing has changed in our dbt setup; this is a waste of both our resources at the destination and yours. What would be great is if we could set up a run-of-the mill GitHub hook when we commit something to a particular branch (master) that triggers dbt to run on demand.

Thanks!

Please sign in to leave a comment.

Comments

4 comments

Date Votes
  • Official comment
    Alexander User

    Hi, product manager here! That's a cool idea. We are working on improving our dbt functionality. Our first priority is integrating the scheduling of dbt jobs with the schedule of data syncs through Fivetran connectors. I'm not clear if that would help in this case. Does a commit trigger fresh data in a connector sync?

    We have many more improvements under consideration for Fivetran + dbt this year. We will look at this one as an option. Are there other aspects of dbt scheduling / triggering you want to see improved, too?

  • Tom Abraham User

    Alexander,

    What's the status of this? That was one of the attractions of using DBT, was the belief that we'd have transforms that ran in coordination with source table synchs. But, right now, I see no way to do anything other than have a fixed schedule set using cron syntax in the deployment.yml file. How/when will we be able to have transforms synched to the synchs? :-)

    Tom Abraham

    Data Engineer

    Caring.com

  • Alexander User

    Tom, we are working on this in Q2 and hope to have it delivered by the end of that quarter. We will be ramping up our investment in transformations all year

  • Matt Woods User

    dbt Cloud's implementation of Slim CI (How we sped up our CI runs by 10x using Slim CI - Deployment - dbt Community Discourse (getdbt.com)) is an excellent implementation of this. We have clients for which dbt Cloud is not an option and would love to see something similar implemented with the Fivetran+dbt integration.