Connector Improvement: Truncate + Load functionality for S3
Answered
I would like a feature of the S3 connector to be a complete delete + insert of the destination table each time a new file is loaded into S3.
The use case I currently have is our 3rd party Marketing company is going to be dropping a daily file in an s3 bucket that represents the complete redshift table output on a daily basis. So, I was hoping to do a truncate + load operation on the destination table in Snowflake whenever a new file is dropped, essentially dropping and recreating the table from the new file. I don't think this functionality currently exists.
-
Official comment
Hi Anthony,
Thanks for this request! So this one file is a full representation, or snapshot, of the data per day. This is equivalent to our Magic Folder connectors, which take each source file and syncs each one as its own table - any updates to this file updates the table.
We do not have a plan to build this for our S3 connector at the moment, but I believe it is a valid request which I will discuss with our team.
In the meantime, would it be possible to use one of our Magic Folder connectors, which may accomplish this workflow? https://fivetran.com/docs/files/magic-folder
We currently support Box, Dropbox, Drive, OneDrive, and SharePoint will release soon.
Please sign in to leave a comment.
Comments
1 comment