Destination Improvement: Add Support for Daylight Savings Time in Destination Time Zone and Schedules
Fivetran allows you to set a destination time zone to automatically manage the offset of your sync schedules from UTC. However, this functionality, completely ignores daylight savings time. This means that for all schedules with a sync less frequently than hourly, the schedules shift by and hour multiple times a year. This creates issues down stream with timing for other integrations along with setting appropriate expectations to users of when to expect refreshed data.
-
Official comment
Hi Everyone,
We've released a small fix that will automatically switch the UTC offset (timezone) to reflect the clock shift of the destination for those that have opted in.
On the destination edit/setup page, you can opt-in to have the timezone shifted during daylight savings. If anyone has 10+ destinations, I can have engineering check it for all destinations. I want to also call out that this functionality will only support US Daylight Savings Time, which falls on November 6th this year..
By shifting the timezone/UTC offset, the connectors to that destination will have the same schedule as before the clocks shifted.
-
It's actually surprising that this has to be requested as a "feature". Our 25 year old mainframe manages daylight saving, AWS, GCP, and Azure manage daylight saving, all of our ETL/ELT tools manage daylight savings, and then we get to Fivetran that is "modern" and it can't handle daylight savings. Not only does it not handle daylight savings but it completely throws the end user off because the interface says the job is schedule for X time but it actually runs an hour earlier. If the schedule tells you one thing and the actual sync time is at a completely different time that's a bug and not a feature.
Someone really missed the boat on this one.
Being told by support that we could fix the daylight savings issue by scheduling outside of Fivetran and kicking off syncs using their API is a real slap in the face. We use Fivetran because of it's simplicity and the fact we don't have to write code around the solution. If we have to start writing code to fix simple flaws like this, the ROI of this product quickly falls off.
-
Bryan Meier I completely agree and had the same conversation with Fivetran support prior to submitting this request.
-
Hi All - Thanks for the feedback here and sorry for the frustration this has caused. We do want to improve how we're handling daylight savings, so I appreciate you bringing this up. It sounds like there are two cases where our lack of robust support impacts you:
- 24 hour sync frequencies no longer run at the same time as before (+/- 1 hour)
- Sync frequencies longer than 1 hour either occur 1 hour earlier or later than the frequency
Am I correct in thinking that the issue with 24 hour syncs is a bigger problem as it continues to run at a different time than before? For the second example (sync frequencies longer than an hour), it sounds like the issue is about the one sync following DST but then the rest are fine. Is that correct?
-
@... I would consider both scenarios to be the same problem since the outcome of both is data not being available when users or other systems expect it to be. The impact of the 24hr syncs could end up being greater if the downstream processing is also occurring once per day, but both present similar challenges. This problem can then be compounded (like in our case for many key sources) if the source of the data is also based off a batch process.
I'm happy to discuss specific use cases if that is helpful at any point.
-
I would agree what Kyle Spotts mentioned. The 1 hour either direction could effect any situation depending on a company's requirements. I could see the least impact being syncs that are running every 15 mins or less but as that time length grows the criticality does as well.
-
We are actually seeing a data quality issue related to how our Workday and Coupa connectors ingest datetime columns during a Daylight savings time period. Once ingested into our target warehouse, the data is off by an hour. When comparing reports from the source system to what is replicated in our Snowflake environment, the datetime columns are off by 1 hour. This occurred during our initial load and continues to occur with timestamp field that fall within that window(mid-March to early November). This is a production data quality issue for us and should not be considered a feature request.
-
John Blankenship - I agree that what you're describing is in fact a data quality issue and should be treated as a bug. This request is centered around when a connector runs and how the schedule is impacted by DST. I've gone ahead and talked to our support team and they're reopening the ticket. You can anticipate someone reaching out tomorrow as they dig into the issue around data in the destination being incorrect.
It also sounds like there is feedback on how the scheduling shifts during DST, I've asked to be included on your next sync with our team so that we can discuss your feedback directly.
Kyle Spotts & Bryan Meier - Can you confirm that your issue relates to how the schedule shifts during DST and that it is something you're looking for us to address before the next shift when we fall back? If so, I'll be reaching out to each of your shortly to get time scheduled so that we can discuss your specific needs/use-case.
-
@... Thanks for checking. I can confirm that getting this addressed prior to the next time change would be great for us and I'm happy to discuss.
-
Hi All - Wanted to just share an update that we're still scoping out this project and should be able to share another update in a few weeks.
-
Thanks for the update Andrew. Looking forward to hearing about the solution Fivetran comes up with. Thanks again!
-
Is there any update on this? Four months ago there was to be an update in a few weeks.
-
Hi All - Sorry for not sending this update sooner!
We're looking at potentially rearchitecting how we handle timezones next year to add in robust daylight savings time support and other enhancements around how we display/handle time. To help with the more immediate needs during daylight savings time, we are planning on adding some support in the coming weeks.
We will be releasing a small fix that will automatically switch the UTC offset (timezone) to reflect the clock shift of the destination for those that have opted in. We will be releasing an update to our destination page where you can opt-in to have the timezone shifted during daylight savings. I'll make a post here once the checkbox is there, so everyone who wants to can opt-in. If anyone has 10+ destinations, I can have engineering check it for all destinations. I want to also call out that this functionality will only support US Daylight Savings Time.By shifting the timezone/UTC offset, the connectors to that destination will have the same schedule as before the clocks shifted.
-
Thanks for the update @....
-
@... I see in your latest comment this is currently being added for US time zones only. It would be great if you could also support daylight savings for countries other than the US.
-
-
Hi Everyone - Beginning early next week, you will start to see the option to shift your timezone with DST:
This will opt you in to have your schedules shifted for that destination. I will send out another message one it is there, but wanted to give an update as we head toward Nov 6th. -
Thanks for the update, Andrew!
-
@..., great news. I've not seen this show up yet in my connector. Will it be rolling out during the week?
-
-
@..., my bad. I was looking on the connection setup, not the destination itself. I've updated the destination settings. Thank you!
I have a job that is supposed to run at 12:30 am, but has been running at 1:30am. I look forward to seeing it run at the scheduled time.
-
Hi Everyone,
I wanted to clarify what this minor fix does and doesn't do. This toggle will just shift the UTC offset as part of shifting the clocks on the first Sunday in November and in the other direction on the second Sunday in March. It will just automate the action some users were taking to update their destination timezone with DST, which will keep your sync schedule aligned with the times you're used to seeing when looking at your own clock.
For example, those on the east coast are currently in "Eastern Daylight Time" (UTC-4), but in our UI, this is actually displayed as "Atlantic Standard Time" (UTC-4). This release will only shift the UTC offset from UTC-4 to UCT-5 on November 6th.
We know this still isn't ideal and we're exploring adjusting to be true time zones where some will shift with DST, while others won't, based on the region selected. -
I have a job that is supposed to run at 12:30 am, but has been running at 1:30am. I look forward to seeing it run at the scheduled time.
If you didn't shift your destinations or sync times for 24 hours syncs "back" to the right time, then you shouldn't toggle this on as it will keep that sync at 1:30 am relative to your clock. After Sunday, if you turned it on later we should then shift the UTC offset in March so that it remains at 12:30 am relative to your clock.
-
Hello,
Why this functionality support only US Daylight Savings Time ?
Thanks,
Please sign in to leave a comment.
Comments
24 comments