Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: dbt Cloud Connector

Answered
Nick Willard User

https://fivetran.com/connector-erd/dbt_cloud

The dbt Cloud connector is missing system log level information that would allow it to be useful. For instance, I can observe the dbt run or build command and the time associated for the command to be finished, but there is nothing about the underlying models the run command parses and runs.

For instance, I can see the following steps:

Clone git repository
Create profile from connection [database]
Invoke dbt deps
Invoke dbt source freshness
Invoke dbt run --select +model --exclude models+

What I cannot observe in any field or underlying piece of information would be:

14:05:26  Running dbt...
14:05:29  Found ### models, # snapshot, ## data tests, ### seeds, #### sources, ### macros
14:05:29  
14:05:29
14:05:29  Concurrency: 8 threads (target='default')
14:05:29  
14:05:30 1 of 166 START sql table model schema.model_a [RUN]
14:05:30 2 of 166 START sql table model schema.model_b  [RUN]
14:05:40 1 of 166 OK created sql table model model schema.model_a [RUN]
14:05:50 2 of 166 OK created sql table model model schema.model_b  [RUN]

The lack of this information effectively makes the connector useless to me at this time.

Please sign in to leave a comment.

Comments

3 comments

Date Votes
  • Official comment
    Unnikrishnan Nair Fivetranner

    Hi Nick,
    We have added this to our backlog. 
    Are you looking for models run and those details specifically or other details as well? Curious to understand your usecase for this information and how you intend to use it so that we can model it right.

    Thanks,
    Unni

  • Nick Willard User

    The goal of this data is to not need to rely on external packages in dbt to record this information: https://hub.getdbt.com/elementary-data/elementary/latest/

    As an Analytics Engineer, I would like to be able to query a database of historic model runs and find the longest running jobs for purposes of optimization within my warehouse.

    In the above example:

    • Model A ran for 10 seconds
    • Model B ran for 20 seconds

    I would therefore prioritize optimizing model B. While it's possible to query our warehouse's usage tables, they do not perfectly correspond to the models run or capture the entire sequence of steps included in the model run.

  • Unnikrishnan Nair Fivetranner

    Thanks for the inputs Nick. 
    We have added this for the team to start their analysis and shall keep you posted as we make progress.