Fivetran History Mode - Atomic writes
Which connector?: Google Cloud SQL for PostgreSQL
Additional details:
We have an hourly job that reads from this connector (which utilizes Fivetran History Mode) and propagates changes downstream in our pipeline. It so happens that this hourly job read a given row from the source data delivered by Fivetran in the exact window between the previous version of this record being closed and the new row being added. To our SQL, this showed the row as closed with no newer opened row, so we judged the row as deleted.
Is there any way we can configure Fivetran connectors like these to perform writes to their destination atomically, i.e. combining the row-close operation and the new row creation into a single transaction? Then, at least, we wouldn't have propagated the intermediate state downstream during our current pipeline run, and would have read the correct state of the row on the next pipeline run.
Please sign in to leave a comment.
Comments
0 comments