Provide an INSERT OVERWRITE data integration for file based connectors (SharePoint, SFTP, ..)
Answered
Which connector?: SharePoint and SFT
Additional details: For integrating Excel and CSV files to a destination many times we need a simple process to completely refresh a table from the source file when it changes.
The existing options assume a continuous data flow and don't work well for simpler use cases like maintaining a parts list or reference data
These are the existing options in the connector:
Upsert file using file name and line number
Append file using file modified time
Upsert file using custom primary key
Add this one:
Insert Overwrite, complete reload of destination table when file changes
-
Official comment
Hi Brent, thank you for filing this request!
Does each destination table always come from one file with a stable name and location, with its contents replaced when the data changes? Or do you upload new versions as separate files with different names?
If the file name and location remain stable, such as
reference_data/parts_list.xlsx, “Upsert file using file name and line number” effectively provides the complete-refresh behavior you need.Best,
Parmeet
Please sign in to leave a comment.
Comments
1 comment