I would like the ability to utilize DBT's state-aware selectors in my transformations in order to improve our CI/CD pipeline performance. With Fivetran's current DBT offering, a job called by our CI/CD pipeline to test or deploy changes would need to build every model in our projects. Whereas if state-aware selectors were supported we could build only the relevant models and their child models to test our changes. This is a big deal because it can reduce the run time of these jobs from hours to just a few minutes, depending on what has been modified.

For context, we would like to migrate our DBT jobs & model runs to Fivetran within the next year, and this feature is the only thing that is keeping us from beginning to work on migration now.

On a technical level, DBT uses artifacts to compare states, specifically the manifest.json artifact. So I think ideally we would be able to check a box on a job to have it generate artifacts on completion that would be saved for state comparisons. In our use case this would be the job called by our CI/CD pipeline to deploy changes once a pull request has been tested and approved for deployment. As part of that testing, we have a testing job that is called whenever we create or modify a pull request to build only modified models and their children. This job utilizes the state selector to validate that our changes build successfully for those models, and that we don't waste time building unnecessary models.

Please let me know if any further information or assistance is needed. Below I've included some DBT documentation for the state selector, manifest, and artifacts.

DBT command for our current testing and deployment jobs: dbt build --select state:modified+

DBT Documentation

State Selector: Node selector methods | dbt Developer Hub

Manifest: Manifest JSON file | dbt Developer Hub

Artifacts: About dbt artifacts | dbt Developer Hub