Skip to main content
Documentation
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Multiple table load option for a single S3 Connector

Planned
Ramya Damarla User

Currently one connector loads all files in S3 bucket into single table in Snowflake. What is  recommended that, each file in S3 bucket should be loaded into separate tables in Snowflake. Otherwise we will end up creating one connector for each file in an S3 bucket.

We recommend that S3 should be added to the Magic Folder Connector.

https://fivetran.com/docs/files#magicfolder

Please sign in to leave a comment.

Comments

8 comments

Date Votes
  • Official comment
    Alison Fivetranner

    Hi Ramya,

    Thank you for sharing this idea is it something we are considering for our future roadmap.

    Can you share a little more about what the use case is where this one to one file to table pattern would be valuable? - where would the files come from, how would then get into the S3 bucket you would configure the connector to monitor.

    Best regards

    Alison

  • Ido Kotler User

    I cant believe i need to create a connector to each file in the Bucket. 

    the use case is obvious: the source system exports its data to S3 for us to use. we are looking at dozens of different tables. 

  • Sam Harvey User

    It would be great to have this available for Azure storage connectors too. Common sense to have this as a feature.

  • Vincent Rodriguez Le Roy User
    • Edited

    Hello, 


    I have the same issue, it is clearly a pain point. We have several parquet file stored in different paths within a single s3 bucket, each file corresponding to a table that have different structures. With the current system we need to create a connector for each file to have them in separate tables. 
    I am also very surprised that the connector was conceived this way. 
    Please could you update us about the status of this roadmap you mentioned @alison ? 

    Thank you in advance for the support 

  • Kalyan Manepally User

    Yes we are looking for this feature to be available for a while now

  • Cason Male User

    Same. This feature would save a lot of manual setup. 

  • Moritz Wendel User

    Google Cloud Storage too...

  • Parmeet Kohli Fivetranner

    Hi All, thank you for following up on this feature. We will be working on a first connector in Q4 and following it up with others.

    Best,
    Parmeet