Connector Improvement: Dynamic Connector to Schema/Table
Completed
For the future would we be able to have one connector be able to sync to multiple schemas/tables. For example, right now our connectors are linked to an s3 bucket where we keep client data. The folder structure for all clients is bucket/{clientName}/{clientTable}/files, which then goes into snowflake as schema -> clientName, table ->clientTable. Could we have just one connector be able to dynamically pull the clientName and clientTable strings to set what the resulting schema/table in snowflake will be.
-
Official comment
Hi Ron,
Thank you for using our S3 connector and taking the time to request this feature.
S3 buckets can be organized in many different ways so we have been loath to force a configuration that might work well for some but be a problem for others. Using our API to quickly configure multiple connectors has been the approach that has worked for many. Is there are reason this approach is problematic for you?
Best regards
Alison
-
Hey Alison,
It's not necessarily that the approach is problematic for me, it's more that it would just be significantly more convenient to only have to manage one connector that's following the same pattern for multiple tables instead of creating and managing 100s of connectors that only change the source folder being pointed to and the destination table name. I understand that may cause massive headaches on the back end for keeping track of data syncing but wanted to throw it out as a potential quality of life improvement.
-
Got it - thank you so much for idea and further explanation.
Definitely one we will keep in mind for the future.
Alison
-
Hi Ron, thank you for raising this request and your patience. Multiple table support is now available on S3.
Best,
Parmeet
Please sign in to leave a comment.
Comments
4 comments