Transformations: DBT job improvement: end of step actions
Answered
Today I fixed the error on DBT job step by simply running dbt run --full-refresh locally.
What if Fivetran's deployment.yml file had tags something like
jobs:
- name: dev_15min
steps:
- name: run-dbt-dev
command: dbt run --project-dir ./<folder>
post-step-action: #optional
-name: do-something
command: ["SELECT ... FROM METADATA", "EXECUTE PROC SP_SOME"]
error-handling-action: #optional
-name: run-full-refresh
command: dbt run --project-dir ./<folder> --full-refresh
command: ["SELECT ... FROM..."]
In my experience full-refresh could fix many failing jobs, and for Fivetran framework this would be just an additional (conditional) step for dbt commands. As dbt framework grows, having the conditional steps could become more useful.
SQL steps would need certainly need a different implementation.
-
Thanks for your request Vadim. How often would you say running full-refresh solves an error in your workflow? Would having the option to refresh your data directly from the Fivetran UI be helpful here?
Please sign in to leave a comment.
Comments
1 comment