You would like to use the deployment.yml file to do advanced orchestration with your dbt git repository. Some things you might be considering:
- Running specific models
- Running separate tasks
- Running tests
dbt Transformations enabled
Fivetran provides a deployment.yml file that enables Fivetran to run your dbt commands on a schedule. You can edit the jobs in the deployment.yml file to reflect how and when you want Fivetran to run your dbt commands.
The deployment.yml file allows you to define multiple jobs, each of which translate to a dbt transformation job in Fivetran UI. The jobs include three main components:
- Name: Give each job a unique name. The name will be displayed in the Fivetran dashboard once your jobs are imported.
- Schedule: Define when this job should run, using cron format.
- Steps: A job perform multiple steps sequentially, so define each step here
- Name: Give each step in your job a name. This will enable you to track the steps in the logs.
- Command: Enter the dbt command that should run in this step.
A common workflow is to have two steps within a job; first run models then test models to ensure data integrity:
- name: daily
schedule: 0 12 * * *
- name: run models
command: dbt run
- name: test models
command: dbt test
It is also common to run models in specific order by having steps that only run certain models:
- name: daily
schedule: 0 12 * * * # This example will run every day at 12:00p
- name: run first sales model
command: dbt run --models sales_model_1
- name: run second sales model
command: dbt run --models sales_model_2