Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Request: Custom S3 Output Path & MDLS Migration Enhancements

Answered
Devansh Khamar User

1) Custom S3 Output Path Independent of Connection Name

Title:
Ability to configure custom S3 output path independent of connector name

Body:
Currently, for S3 MDLS destinations, the folder path is automatically derived from the connector name, and two connectors cannot share the same name even across different destinations. This prevents us from preserving the exact same S3 folder structure in two different environments.

​
We request a feature that allows defining a custom S3 output path (or base prefix) per connector that is decoupled from the connector name, so that during migrations we can point new connectors to the same logical folder paths as the original ones without renaming or restructuring downstream pipelines.

 

2) Dual‑Write and Metadata‑Aware Historical Migration for MDLS

Title:
MDLS migration support: dual‑write to two destinations and metadata‑aware historical migration

Body:​

A dual‑write capability, where a connector can write to two MDLS destinations (two S3 buckets/regions) simultaneously.​

We request first‑class product support for MDLS destination migrations (SOP, tooling, and UI/API options) that support dual‑destination writes and metadata‑aware cutover, so customers can execute fast, low‑risk regional and account migrations.

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Official comment
    Casey Karst Fivetranner

    Thanks for bringing these asks in! 

    For 1, the risk there is the delete orphan operation which runs as part of our table maintenance. That operation looks at the referenced files in a single table and then compares that to all other files in the bucket. The list of files that are not referenced by a current snapshot are deleted. This is why we limit each table to a single folder. 

    For 2, this is a really interesting ask and something that we have been discussing. What is driving your migration from one bucket to another? 

  • Devansh Khamar User

    We are Migrating to different Region and AWS Account due to the below reasons.

    Migrating to the US-EAST-1 environment allows us to:

    • Access new platform features earlier

    • Take advantage of services or capabilities that may only exist in that region

    • Establish clear separation between development and production environments