Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: S3 connector - dynamic table creation

Completed
Sandro Frattura User

Imagine an option where an ingestion is pointed at an S3 folder.  files within it are named `user_10162023`, `user_10162022`, `user_10162021` etc.     Currently, we designate that ingestion to create a table named "user"

what if new files start showing up named `customer_10162023` or `orders_10162023`
 
It would be great to have Fivetran will automatically create customer and orders tables based on these.
 
so the idea would be a "S3-dynamic" ingestion whereby we deignate a RegExp matching and have fivetran map to the appropriate tables...even if new ones arrive.
 
Given a file list like this:
  • user_10162021
  • user_10162022
  • user_10162023
  • customer_10162023
  • orders_10162023
and given the instruction (in RegExp) "the table name is everything before the first underscore", it would know to create user, customer and orders tables.

Please sign in to leave a comment.

Comments

3 comments

Date Votes
  • Official comment
    Alison Fivetranner

    Hi Sandra,

    Thank you for sharing your use case. I'm wondering how generally this would be useful and applicable.

    I'd love to hear from other customers who have data structured as a series of sets of files where the file name prefix would denote the table name.

    Best regards

    Alison

  • Sandro Frattura User

    I certainly can't speak for other customers, but et me tell you our current project.     We will getting 20+ different categories of files in S3.   Each has a prefix as i note above (user_xxx, customer_xxxx, orders_xxxx)

    In the current state of s3 ingestion, every time. a new "type" of file comes in (say tomorrow they start sending "cancellations_xxxxxx"), I have to create a new S3 ingestion with Fivetran.

    instead, with dynamic ingestion (think like "magic folder" feature for google drive), i could have one ingestion, but that ingestion can auto-spawn -N- number of tables based on the file prefix

  • Parmeet Kohli Fivetranner

    Hi everyone, thank you for raising this request and your patience. Multiple table support is now available on S3. While it isn't dynamic, you don't need to create a connection for each file pattern.

    Thank you,
    Parmeet