Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Capture deletes in Google Drive

Robert Goldstein User

As a user, I want to be able to delete files from our Google Drive and have the removals reflected in our data warehouse. Preferably automatic but an is_deleted flag would be acceptable as well.

Please sign in to leave a comment.

Comments

3 comments

Date Votes
  • Official comment
    Alison Fivetranner

    Hi Robert,

    Thank you for sharing, could you tell me more about the work flow you currently have?

    Our Google Drive connector currently checks all files in the targeted drive and syncs any with an updated_at more recently than the last sync - the challenge we have is identifying when a file was deleted as it is no longer present in the folder.

    Can you think of an approach for sharing 'deletes' - what would you think if we asked you to add "DELETE_" to the front of any file you wanted to have processed as a 'delete' rather than an addition.

    Best regards

    Alison

  • Robert Goldstein User

    Hi Alison

    Sure happy to share the workflow.

    Let me share a little on the use case for context because it helps share how Fivetran is helping us meet our stakeholders sometimes when they are in challenging places.

    One of our stakeholders was getting monthly CSV exports from various partner UIs and then appending them to a very large Google Sheet.

    Since these partners don't have a Fivetran connection or likely an API, I am having them leverage Fivetran to download the CSV and drop it into a Google Drive folder. Fivetran is connected to the folder and is set up to merge each Google Sheet loaded into a single table. We have that table connected to Looker, so that way they can query their entire history of partner data in a database that can handle the record count, unlike a single Google Sheet. It is far from perfect but it makes the data accessible for them to do what they need and make business decisions. 

    I think adding DELETE_ to the name of an existing file is an acceptable solution. Anything that allows us to tell Fivetran this data is no longer needed. 

    Otherwise what we have had to do is constantly update the Looker Explore to pre-filter out the 'dead' files based on their file name. I imagine if I started having them append DELETE_ to the dead files right now Fivetran would treat this as a new file so there is some engineering that is required by Fivetran to recognize this as a deleted file.

    Let me know if you need more info, thanks!

  • Robert Goldstein User

    Hello just checking back on this