Conector Improvement: Support PostgreSQL pgoutput plugin
Planned
Please implement the capability of using the pgoutput plugin and publication name for PostgreSQL connectors to consume WAL for PostgreSQL versions 10 and above.
The pgoutput plugin requires the use of a publication which will allow customers to eliminate the staging changes on the replication slot (i.e. truncates, massive inserts/updates) on staging tables without primary keys by specifying only the subset of tables to replicate and not capture changes to every table.
-
Official comment
Hi Jack!
Kevin from the Product team here. I agree that we should evaluate alternative Postgres WAL plugins such as pgoutput and wal2json. We currently only support test_decoding, which as you mention, has some limitations.Our engineering team has plans to investigate pgoutput's potential benefits to make our Postgres connector with Logical Replication significantly better.
Additionally, we are also planning some benchmarking projects to investigate key performance improvement areas in different processes end-to-end for Postgres.
We will make sure to thoroughly evaluate pgoutput and see if it makes a significant impact over test_decoding. Please stay tuned for any improvements we make throughout Q4 for Postgres WAL.
-
Hi Jack,
I wanted to provide an update to you that we have investigated pgoutput over test_decoding, and are making plans to swap the plugin our Postgres connector uses. Please stay tuned for updates while we scope out and commit to this work. No committed timeframes yet, but this improvement should be scoped and committed to shortly!
-
Following up here! We are building this right now and are planning to roll out pgoutput support in this quarter (Q2 2021)
Please sign in to leave a comment.
Comments
3 comments