Transformation (dbt) triggered on git push

Chip

• January 14, 2021 20:55

Because running our dbt transformations takes time and resources on the destination, we only trigger the entirety of our transformations once a day in our deployment.yml. (We have a more limited set of actions scheduled to run more frequently.) Most of the time, they do nothing because nothing has changed in our dbt setup; this is a waste of both our resources at the destination and yours. What would be great is if we could set up a run-of-the mill GitHub hook when we commit something to a particular branch (master) that triggers dbt to run on demand.

Thanks!

0

• Official comment

  

  Alexander

  • February 09, 2021 01:41

  Hi, product manager here! That's a cool idea. We are working on improving our dbt functionality. Our first priority is integrating the scheduling of dbt jobs with the schedule of data syncs through Fivetran connectors. I'm not clear if that would help in this case. Does a commit trigger fresh data in a connector sync?

  We have many more improvements under consideration for Fivetran + dbt this year. We will look at this one as an option. Are there other aspects of dbt scheduling / triggering you want to see improved, too?
• 

  Tom

  • February 10, 2021 22:23

  Alexander,

  What's the status of this? That was one of the attractions of using DBT, was the belief that we'd have transforms that ran in coordination with source table synchs. But, right now, I see no way to do anything other than have a fixed schedule set using cron syntax in the deployment.yml file. How/when will we be able to have transforms synched to the synchs? :-)

  Tom Abraham

  Data Engineer

  Caring.com

  0

Please sign in to leave a comment.