Skip to main content
Documentation
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Append only mode for Postgres connector

Not planned
Simon User

I have a feature request that I'd like to see implemented for the postgres connector.

I have set up the postgres connector to synchronise using logical replication which appears to work without issue, I followed the instructions found here: https://fivetran.com/docs/databases/postgresql/setup-guide#logicalreplicationsetup

This works fine, but I think there should be an option for this connector to work in an "append only" mode. How it works now is that the connector will _update_ my destination tables so that they look like the Postgres table, effectively keeping the tables in sync (except deletes). This is probably what some people want, however it isn't really the best way to build a staging area for a data warehouse. It would be better if all changes made in postgres were _inserted_ into the destination table rather than making updates to the destination table. There would also need to be a flag in the destination table which would indicate if the record is a DELETE, INSERT or UPDATE operation.

Would be great to see something like this implemented

Please sign in to leave a comment.

Comments

6 comments

Date Votes
  • Official comment
    Kevin Kim Fivetranner

    Hi Simon! Kevin from Product here.

    We are currently working on a new sync mode called history mode, which would only append data to your destination, granted there would be a few system columns to track history over time (https://fivetran.com/docs/getting-started/feature/history-mode).

    Otherwise, for a true append only mode without additional history columns, I can take this back to the team for discussion and prioritization. Our current sync strategy is currently to create either:

    1. like-for-like replica in the warehouse
    2. historical change table using SCD type 2

    If you are interested in this history mode as an interim solution, we are happy to add you to a list of alpha testers for when it is ready. Please let me know!

  • Eric User

    Kevin,

    My org would be very interested in testing this.

    Thanks,

    Eric

  • Simon User

    Hi Kevin, apologies for the late response, complete forgot! I'd be happy to test this out as an alpha tester however I don't really have a full production use-case so I'd only be doing localised "contrived" testing. Still, happy to help with that too!

  • Fei User

    Hi Kevin,

    Is there any plan to support RDS (Mysql) for such history mode? If yes, I would like to test it out as an alpha tester too.
    And in the meanwhile, is there any other workaround to preserve the history?

    Thanks,

    Fei

  • Kevin Kim Fivetranner

    Hi Fei,

    MySQL History mode is currently in private preview. Please reach out to your Customer Success Manager and we can get it set up for you!

  • Fei User

    Hi Kevin,

    Thanks for the quick update. However, I couldn't reach my "Customer Success Manager". I also submitted a request but no response for two days. Maybe I don't have one since our accounts are still on trial? We need to test this feature asap.

    Thanks,
    Fei