Skip to main content

Community

Date/Time Picker for Sync Frequency on Transformations & Connectors

Answered

Please sign in to leave a comment.

Comments

12 comments

  • Official comment

    It's a conscious decision to have relatively simple & inflexible scheduling in the UI. Trying to time when connectors should run is an anti-pattern because it creates a fragile tightly coupled system.  For example, what happens if the upstream job fails & the fresh data is not ready when the Fivetran connector runs once a day?  This idea that tightly coupled complex systems are fragile & prone to error is explored in the classic book Normal Accidents by Charles Perrow.

    A better strategy is to loosely couple your upstream data processing from Fivetran.  The way to do this is running Fivetran frequently. Effectively, the Fivetran connector will be polling for changes & syncing them as they occur.  Our pricing model only charges you for Monthly Active Rows, so frequent connector syncs that don't move any data are free.

    Hi Gary! Thanks for submitting this request.

    Regarding sync schedules for connectors-- we have that capability today via our API. Would that meet your needs or do you prefer a solution in the dashboard?

    Regarding schedules for transformations, our dbt Transformations (newly released in beta) supports custom schedules like you mention. 

    Hi Amy!

    That's awesome the API has it. But I'd love to have the solution in the UI as well. I'm technical enough to be dangerous with APIs, but would much rather prefer a UI :)

    As for transformations, I just commended on the other post too:  I'd love to use dbt, but that's a separate tool that I would have to get approval and budget for (while I'll definitely make the case, it's just hard during these times). My IT team doesn't like us using free tools for sensitive data, so couldn't use their free tier either, which is why I asked for this. It'd be great to just ever so slightly expand the capabilities that you all traditionally have until I can get dbt going.

    thank you!

    Gary Sahota Thanks for sharing! We currently have the ability to refine the time that the connector runs when you're selecting the every 24 hour option but we don't yet have this for Transformations. Is that something that would work for this use case?

    Alexa Maturana-Lowe - This could work! Would love to be able to fire off transformations on specific date/times (like the 1st Monday of the month or something), but this solves my connector use case!

    Fabulous news! Gary Sahota curious why you'd want to do first Monday of the month for a transformation? 

    Sure thing, Alexa Maturana-Lowe! We actually take raw usage data from Mixpanel and load it into Snowflake using Fivetran (to ultimately display via Tableau dashboards). One metric we'd like to display is Monthly Active Users. So, on the first of a given month, we take all the previous month's usage data to get MAU by feature.

    This transformation only needs to kick off once a month, so it doesn't really make sense to recalculate the same metric every 24 hours (or even every week). So having a specific date/time picker would let me Calculate Monthly Active, Weekly Active, Daily Active in 3 transformations fired off once a month, week, day respectively.

    Happy to chat live if needed!

    Being able to set a connector to run at a specific time say 6am after the source is updated, would be great instead of saying to run it every 24 hours.  We have tables in the source that are updated once a day at a specific time, so it would be good to schedule the connector to run right after they are updated.

    Some competitors allow for CRON notation scheduling of connectors. Would be an awesome feature for Fivetran for having more contorl over intra-day schedules or schedules that only need to run once a week / month.

    This is a feature I would like to see too. I have a connector that takes more than 3 hours to run. Id like the connector to start in the morning at 3am and finish by 6 or 7 am when our morning reports go out. Even if I increase the sync frequency, our daily morning reports might be stale. When can we expect this feature to be available?

    Hi Arpit - you can select the start time for once-a-day syncs (24 hour frequency)


    Cheers,

    Fraser

    We'd also love better control on the sync schedule through the UI. We have a couple of obvious scenarios:

    - Some data needs to be synced frequently during office hours for nearish live reporting. Out of hours no-one cares, it just costs, which is bad.

    - Month end reporting. For 4 days around month end we do frequent syncs between systems, but for the rest of the month it's not important.

    In both these instances, I would have to manually update the schedule.

Didn’t find what you need?

Contact support