Connector Improvement: Postgres Connector: Flush WAL even when no relevant changes
Completed
We have a number of databases which do not have any changes on weekends. With the test_decoding plugin, this results in the transaction logs growing very large (about 20GB per day) until the next change happens on Monday.
Currently I have been told this is because the connector will not be able to flush the WAL until it has at least one relevant change. The workaround is to perform dummy inserts with pg_cron or some other method.
It would be better if the WAL was able to be cleared every time a successful sync takes place, whether or not relevant changes have been detected.
-
Official comment
Hi Sam,
Thanks for your question here! I will share this feedback with our engineering team.
Have you had a chance to review the new beta feature for Postgres called pgoutput Publications? https://fivetran.com/docs/databases/postgresql#logicalreplication
This feature allows you to configure a Publication for your slot that only tracks changes for tables you have selected. That way, your slot won't fill up with transactions from other tables and instead will only include transactions from tables you are syncing with Fivetran. Would this alleviate the pain point you are currently facing? Please let me know!
-
Hi Kevin,
Sorry for the very late reply. This does sound ideal for us. We've had a very busy few months and the dummy insert cron job has been serving as a workaround for now.
As soon as I get time I will try pgoutput.
Thanks
-
As an update, we have now launched the pgoutput update method to General Availability! Please let me know if you have any questions about this feature, more information is available in our docs here: https://fivetran.com/docs/databases/postgresql#logicalreplication
Please sign in to leave a comment.
Comments
3 comments