Connector Improvement: Batching DELETE operations into smaller chunks
AnsweredReferring to ticket ID #298548, we are requesting Fivetran to improve on the current single Delete operation during sync.
This single, large DELETE transaction triggered a huge consumption of resources (physical disk space) for two main reasons:
-
Massive Undo Logs: MySQL's InnoDB engine writes a "before image" of all deleted rows to the undo logs to allow for transaction rollback. Deleting 1.9 million records in a single transaction caused these logs to swell to over 2TB. This space is not released until the transaction is fully committed or rolled back, a process which can be extremely slow.
- Binary Logs: If binary logging is enabled, this single large transaction is also written to the binary logs as one large event, further contributing to disk space consumption.
To prevent this issue, I believe the DELETE statement generated by the Fivetran connector should be broken into smaller, more manageable batches. Instead of a single, massive transaction, using a LIMIT clause to delete a fixed number of rows at a time would prevent the undo logs from growing to a size that overwhelms the disk.
This approach would significantly reduce the memory and disk I/O overhead, making the resync process much more stable and preventing the server from running out of disk space.
-
Official comment
hi YewHwa,
The ability to chunk deletes during MySQL loads is a valuable suggestion—thank you for sharing it. We've added this feature request to our backlog for destination improvements. To better understand the context, could you share more about why there’s a need to delete such a large volume of records in the destination?
We will keep this thread updated with any progress or changes regarding this request.
Best regards,
Please sign in to leave a comment.
Comments
1 comment