Connector Improvement: AWS S3 connector support for multiple table with single connector based on file pattern
Completed
Do we have a feature available where we can read folder wise file from s3 and dump data into their respective table in Iceberg with single connector.
Currently as per documentation we need to build 3 connectors if have 3 different folder in S3 bucket to dump data into 3 table. Please let me what are feasible solution.
We are looking for option where we can define a pattern for same type of file in a Single folder into a single table. So with this we don't need to build many connector for single folder to handle multiple files into multiple tables.
-
Official comment
Hi Aman, thanks for submitting this request. Multiple tables in a single connection is in early access for S3 and should be rolling out more broadly in the next couple of weeks. We just enabled it for your account.
Thank you,
Parmeet
Please sign in to leave a comment.
Comments
1 comment