Ability to do truncate + reload
Currently whenever we do re-sync or history reload for already existing table in the connector fivetran is doing merge + soft-delete, We need feature to do truncate + Reload.
Truncate + Reload:
We need truncate all the data from target and then reload the table with fresh data only.
merge + soft-delete
- Fivetran uses soft delete behavior.
- During a re-sync/historical sync, rows that are re-imported are set to
_fivetran_deleted = FALSE. - Rows that already exist in the destination but are not present in the refreshed source snapshot / active PK set are left in the table and marked
_fivetran_deleted = TRUE. - So Fivetran is doing a merge + soft-delete, not a truncate + insert.
-
Trying to manage a "dirty" destination table in Fivetran is an absolute nightmare. I have destination tables that only contain a few thousand records, but where some of these were created before the source system hard-deleted them. I managed to get Claude to create a hacky way in which I can get the custom connector to implement soft deletes (by saving all PKs from previous run in state table) but it would be SO much easier if I could reset and resync the destination table. To use my soft delete approach I have to manually prime a connector python file with id's that I've identified off-system. This is a ridiculous situation from a premier ELT tool. This is doubly frustrating because every time I set up a new connection in FiveTran I have to give it permissions I don't want it to have (like schema ownership ability to drop schemas, and create schema privileges) and yet it seems unwilling to actually use the permissions it demands.
Please sign in to leave a comment.
Comments
1 comment