Connector Improvement: Add Support for Capturing Deletes & Auto Soft Deleting to the Connector SDK
AnsweredCurrently the Connector SDK does not automatically soft delete entries that are updated and instead overwrites existing rows. In order to soft delete in the Connector SDK, the user must manually define which rows should be marked as deleted and must use the delete function.
The desired behavior from this request is to have the option to soft delete updated rows instead of overwriting when using the upsert function. For example, if we have the following entry where the pkey is unique and is used as the primary key:
pkey - 001 | status - active | last_updated - 10:30
and the entry changes to:
pkey - 001 | status - completed | last_updated - 10:45
The current behavior of the upsert function will overwrite the 10:30 entry in the destination with the 10:45 entry. The desired outcome would be to have the option in the upsert function to have the desintation look like the following automatically instead:
pkey - 001 | status - active | last_updated - 10:30 | _is_fivetran_deleted = true
pkey - 001 | status - completed | last_updated - 10:45 | _is_fivetran_deleted = false
-
Official comment
Hi Ryan,
Thanks for submitting this request. Sounds like you'd like to maintain complete history of the data (like Fivetran's History Mode in other connectors), which I can understand as a useful feature. I've added this to our backlog for now and we'll continue to monitor for demand from other users.
Best,
Sadie
Please sign in to leave a comment.
Comments
1 comment