Support arbitrary regions for final managed datalake S3 storage bucket region
Which destination?:
Managed DataLake - AWS S3
Additional details:
This was originally a support request, they asked me to additionally post it here for visibility - we're blocked on expanding to our production us-west-1 environment because it's not a supported storage region. Background and details below.
We're in the process of expanding our Fivetran rollout across multiple environments and have hit a significant roadblock: the managed data lake destination does not allow us to use our aws s3 storage bucket in us-west-1.
We understand why data processing regions are limited - that's where your infrastructure lives, that's a reasonable constraint. But the storage region is different: it's simply the final location where Fivetran copies the output files. What's particularly puzzling is that you already allow the storage region to differ from the processing region, which confirms your architecture already decouples the two concerns. I can't think of any technical reason for Fivetran to gate which region our bucket sits in - the restriction seems arbitrary. We understand there will be cross-region transfer costs, and we accept those costs.
We already have substantial infrastructure in us-west-1 and have an established datalake there. Expanding our Fivetran usage to cover additional environments now requires us to either restructure that infrastructure or accept that one of our regions simply can't participate - neither of which is acceptable.
This restriction is also effectively undocumented. The documentation is very clear about the restriction on available data processing regions, but it is not at all clear that it also applies to the customer's own storage bucket. There is no mention anywhere that the storage bucket region is subject to any restrictions beyond what the customer is willing to pay in transfer costs - nor would someone expect there to be any such restrictions. The first indication we received that this was a limitation was an API error during production rollout expansion - after significant investment in this integration. Running into an undocumented hard blocker this late in a project lifecycle is a shockingly significant stumbling block.
This is blocking a concrete expansion for us today. We'd appreciate:
1. Allow any AWS region as the storage destination region, independent of the processing region.
2. At minimum, document this restriction clearly in the destination setup documentation, the API reference, and the data residency documentation so customers can discover it before committing to an architecture.
-
Official comment
Hi Chris --
Please work with your account team to get support for your requested region prioritized.
Thanks for taking the time to write in, it is much appreciated! -
We are evaluating the move to MDLS on Azure and this is a blocker for us as well.
-
Also, Chris --- the bucket / region requirement is documented here:
https://fivetran.com/docs/managed-data-lake-service/setup-guide#prerequisites -
Yes, that documentation was added a couple of days ago as a result of my support request 😊.
We would still appreciate it if this restriction could be reconsidered. In the meantime, could someone help me understand the technical rationale for restricting the destination region?
I'm not asking for a new processing region. Fivetran already fully supports copying the resulting files to a remote region after processing is complete, so I'm struggling to understand why there is a restriction on which remote regions those files can be copied to.
The newly added documentation says this restriction is intended to avoid egress costs, but that rationale isn't clear to me. The documentation for the Managed Data Lake service explicitly states that network data-transfer charges apply when the Fivetran processing region and the storage account region differ (documentation). This clearly indicates that cross-region transfers are already supported, with the associated transfer costs.
If there is a technical limitation I'd appreciate some clarification on what it is.
Thanks!
-
Hi Chris ---
It's a business limitation more than a technical one. We've had too many customers come back after not understanding the data xfer costs and blame us. We're preventing it by design this way. Also, adding new regions isn't terribly difficult for us, and new region is a much better outcome for Fivetran and the majority of our customers. -
Ahhh interesting, I see. So do you do some sort of internal routing voodoo to avoid the data xfer costs for your supported regions, when the storage region and processing region differ?
-
More architecture trick, less routing voodoo. The catalog service itself was regionalized, so essentially, the processing region will always match the bucket region. Volia! No dataxfer costs.
Also -- thank you for taking the time to get the docs updated. As a small thank you, let me ask / qualify ----
Do you have other regions that you might need that we don't offer?
I have a request from your account team that I am already working to prioritize, re: new region, but given the nature of this feature request, I wonder if there aren't others that you were thinking about? -
We need Azure West US 2 :)
-
Josh ----
Yes, I'm working on that (Azure US West 2) for you with your account team(s).
We have confirmed that we're planning to add this region in Nov, possibly Dec.
No further action needed from you at this time! -
Hi Pieter,
Ahhhhh that's a good trick. Ok that makes sense, thanks for the insight.
Regarding additional regions, I've reviewed our full rollout plan and identified that us-gov-east-1 is also required, and currently unsupported by Fivetran. To be honest, I think this is actually the more significant requirement of the two.
While us-west-1 is our immediate stumbling block, there's a reasonable argument that we should consider migrating away from it anyway, given that it's somewhat of a second-tier region. A migration would have some cost and effort associated with it, but it may make sense for us in the longer term.
us-gov-east-1 is a different story - it's a fully supported tier-1 region, and we don't have the same rationale for moving away from it.
If I had to prioritize one of these two regions, I'd prioritize us-gov-east-1.
-
Hi again Chris --
Ok, good to know. I know we have us-gov-west-1, but to be honest, I'm not optimistic that we have enough demand to build out us-gov-east-1. I will look into it. can you post that as a seperate feature request, not in this thread? -
Sure, I'll add a feature request specifically for us-gov-east-1.
Your explanation got me thinking, though. Could a potential solution be to allow customers to clearly and explicitly opt in to cross-region transfers and accept the associated transfer costs? For example, an API-only field like
data_transfer_costs=accepted, not necessarily even exposed in the UI?If I'm understanding the history correctly, this might essentially mean allowing customers to opt back into the previous behavior, rather than introducing an entirely new capability. The current restriction could remain the default, protecting customers from unexpected costs, while those who understand the trade-offs could explicitly accept them. Given that this is how it was previously, this might be a relatively small change?
It might also be a more practical solution than adding processing regions in every case, particularly for regions like us-gov-east-1 where demand may not justify the extra infrastructure.
Either way, I'll add a feature request for us-gov-east-1
-
totally reasonable idea Chris, I asked pretty much that same question.
there so much work to do on the core product, something like this would be considered a finishing touch, coming much later after foundational work is done. e.g. valid, but nice-to-have when compared with the foundational capabilities the team is focused on for MDLS right now.
Please sign in to leave a comment.
Comments
13 comments