Connector Improvement: Google Drive Capture Deletes
Fivetran does not support capture deletes for the Google Drive connector. In the Merge Mode this should be easy to support at least at the file level.
This improvement would enable users to manage a set of unchanged but delete-able files Drive folder to be reflected correctly into a destination table.
Without this improvement, at any user upload mistake the target table would permanently have incorrect data and cannot be fixed by the user - it must be manually deleted from the destination then a historical sync performed.
-
Official comment
Hi Asaf,
Thank you for taking the time to help make Fivetran Connectors better.
I'd like to make sure I fully understand you use case and ask so I can consider the feature for addition.
When using the Google Drive Connector in Merge mode you have noticed that we only ever add data. Is that correct?
Is your ask that if a file gets deleted we mark the rows associated with that data as `fivetran_deleted` = TRUE? so that the data in your ware house is always an accurate representation of the data in the folder that is being sync'ed.
Have I understood you correctly?
Best RegardsAlison
-
From reading documentation in Merge Mode, my understanding is the _modified column would be null if the file was deleted. But I am not seeing that. I'm having a difficult time trying to identify deleted files in my Google Drive folder.
-
Hi Alison,
As per the documentation, deleting an entire file from the source would not have its rows in the destination deleted - in both modes of file connector.For merged folder:
If you delete a file in your source bucket, it no longer exists, so it will not have a last modified date. Therefore, the deletion will not be reflected in your destination - so data from the deleted file will still remain in your destination.
As well as Magic Folder:
If you delete a file from your cloud folder, we don't delete the table from your destination.
Please sign in to leave a comment.
Comments
3 comments