Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Other: Dependency on upstream jobs in transformation scheduling

Answered
Siva Sankar Gulipalem User

Hi Team,

We have a requirement to establish an upstream dependency between a Source Sync Connector and a Transformation job in Fivetran.

Example Scenario:

  • Source Connector: DEV_S_ABC_CLIENT (scheduled to sync every 15 minutes)
  • Transformation Connector: DEV_T_CLIENT

Currently, the Transformation Connector is set to trigger at 12:00 AM daily from Sunday to Friday. We need to add a dependency so that the Transformation only starts after the 12:00 AM sync of the Source Connector.

We've reviewed the Terraform documentation, which appears to only support scheduling with intervals (as shown here). However, it doesn’t address our specific need for dependency management.

Could you consider developing a feature that allows us to create this type of upstream dependency? This functionality would be highly valuable for ensuring data consistency and accuracy in our workflows.

Thank you for considering this request. We appreciate your support.

Please sign in to leave a comment.

Comments

3 comments

Date Votes
  • Official comment
    Jimmy Fivetranner

    Hi Siva,

    If you are trying to use Terraform or our REST API, you can use select the field schedule_type as 'TIME_OF_DAY', and then select the days and time_of_day field values, you should be able to accomplish what you're looking for. 

    You can see more details in the field params here: https://fivetran.com/docs/rest-api/transformations-for-dbt-core-management#payloadparameters_2

    If you want to do it via the UI, just set the schedule to custom and frequency for 24 hours and that will allow you to specify a time of day. 

    Lemme know if that doesn't cover it!

    Jimmy

  • Siva Sankar Gulipalem User

    Thanks for your response, Jimmy. The connector is currently scheduled to run every 24 hours. However, the requirement is to establish a time-based dependency and run only after upstream connector sync is complete. 

  • Jimmy Fivetranner

    Hi Siva,

    Apologies, I realize I misread this. If you have the following scenario:

    • Connector Syncs every 15 minutes
    • Transformation is scheduled to run at 12AM every day

    And you want the transformation to run after the 12am sync of the connector, we already account for this scenario. We won't run the transformation until the connector has finished syncing data. You can read more about this here under custom scheduling: https://fivetran.com/docs/transformations/transformation-scheduling#customscheduling