Support arbitrary regions for final managed datalake S3 storage bucket region
Which destination?:
Managed DataLake - AWS S3
Additional details:
This was originally a support request, they asked me to additionally post it here for visibility - we're blocked on expanding to our production us-west-1 environment because it's not a supported storage region. Background and details below.
We're in the process of expanding our Fivetran rollout across multiple environments and have hit a significant roadblock: the managed data lake destination does not allow us to use our aws s3 storage bucket in us-west-1.
We understand why data processing regions are limited - that's where your infrastructure lives, that's a reasonable constraint. But the storage region is different: it's simply the final location where Fivetran copies the output files. What's particularly puzzling is that you already allow the storage region to differ from the processing region, which confirms your architecture already decouples the two concerns. I can't think of any technical reason for Fivetran to gate which region our bucket sits in - the restriction seems arbitrary. We understand there will be cross-region transfer costs, and we accept those costs.
We already have substantial infrastructure in us-west-1 and have an established datalake there. Expanding our Fivetran usage to cover additional environments now requires us to either restructure that infrastructure or accept that one of our regions simply can't participate - neither of which is acceptable.
This restriction is also effectively undocumented. The documentation is very clear about the restriction on available data processing regions, but it is not at all clear that it also applies to the customer's own storage bucket. There is no mention anywhere that the storage bucket region is subject to any restrictions beyond what the customer is willing to pay in transfer costs - nor would someone expect there to be any such restrictions. The first indication we received that this was a limitation was an API error during production rollout expansion - after significant investment in this integration. Running into an undocumented hard blocker this late in a project lifecycle is a shockingly significant stumbling block.
This is blocking a concrete expansion for us today. We'd appreciate:
1. Allow any AWS region as the storage destination region, independent of the processing region.
2. At minimum, document this restriction clearly in the destination setup documentation, the API reference, and the data residency documentation so customers can discover it before committing to an architecture.
-
Official comment
Hi Chris --
Please work with your account team to get support for your requested region prioritized.
Thanks for taking the time to write in, it is much appreciated! -
We are evaluating the move to MDLS on Azure and this is a blocker for us as well.
-
Also, Chris --- the bucket / region requirement is documented here:
https://fivetran.com/docs/managed-data-lake-service/setup-guide#prerequisites -
Yes, that documentation was added a couple of days ago as a result of my support request 😊.
We would still appreciate it if this restriction could be reconsidered. In the meantime, could someone help me understand the technical rationale for restricting the destination region?
I'm not asking for a new processing region. Fivetran already fully supports copying the resulting files to a remote region after processing is complete, so I'm struggling to understand why there is a restriction on which remote regions those files can be copied to.
The newly added documentation says this restriction is intended to avoid egress costs, but that rationale isn't clear to me. The documentation for the Managed Data Lake service explicitly states that network data-transfer charges apply when the Fivetran processing region and the storage account region differ (documentation). This clearly indicates that cross-region transfers are already supported, with the associated transfer costs.
If there is a technical limitation I'd appreciate some clarification on what it is.
Thanks!
Please sign in to leave a comment.
Comments
4 comments