Skip to main content
Documentation
Feature Requests Onboarding Trust Center Status Page

Community

Transformations: Add timeout_seconds to dbt Advanced Options

Matt Millen User

Given that the profiles.yml file cannot be viewed or edited certain parameters should be available in the configuration that enable editing. This is already being done with Threads and needs to also be done with timeout_seconds as larger jobs can take longer than the 300 second default

Please sign in to leave a comment.

Comments

7 comments

Date Votes
  • Anna User

    Hi Matt Millen, thanks for the request! The dbt docs imply that no default timeout is set, and we don't set one on our side. Many of our users' transformations take hours to successfully run, which is a pattern we intend to support. Are you seeing errors in the UI that indicate your transformations are timing out after 5 minutes?

  • Matt Millen User

    Yes, I am getting timeout errors if the run exceeds 300s. Quoting a couple of lines of my recent logs that encountered errors

    [0m01:08:30  3 of 10 ERROR creating incremental model analytics.games ....................... [ERROR in 306.00s]
    [0m01:08:34  Runtime Error in model games (models/analytics/games.sql)
[0m01:08:34    Operation did not complete within the designated timeout.

    The doc you linked to says there is a default 300 second timeout, contrary to your statement that no default timeout is set. Perhaps this is just a BigQuery default?

  • Anna User

    You're right, I just assumed the non-parenthetical was true!

    No timeout is set by default. (For historical reasons, some query types use a default of 300 seconds when the job_execution_timeout_seconds configuration is not set.) 

    I'm following up internally on this, it does seem like a query + warehouse limitation. I'm open to increasing that limit– what command are you running, and how long does it typically take to execute?

  • Matt Millen User

    I have an incremental materialized model which is "Scheduled in Code" every 2 hours. There are 10 models run as part of the schedule and out of the last 20 runs I have had one of the 10 models fail 3 times due to timeout. So it's working the majority of the time, just occasionally it creeps over 300s. The other 9 never get too close to 300s fwiw

  • Matt Millen User

    Is there any update on when this timeout might become an available config? I've been getting a lot more failures lately and it's now a blocker to migrating all my transformations. With basic SQL being sunsetted next year I'd consider it a must have

  • Zack Keller User

    Also looking for a potential update here - we're running into timeout issues with some of our queries and setting this option to a higher value is an easy solve.

  • Anna User

    Matt Millen Zack Keller the default timeout for transformations in all destinations has been updated from 300 seconds to 6 hours this week. Let me know if you're still running into issues with this.