Connector Improvement: Support to drop columns in destination when source schema changes
Answered
I am using a Google Cloud SQL connector for our production Postgres instance.
We are a small team and iterate quite fast in postgres - adding/dropping both tables and table columns. So, as part of data engineering, I find myself periodically having to audit columns as seen by Fivetran for the postgres source and what is present in our BigQuery destination. Then, I manually execute drop column commands to remove those columns that have been dropped in postgres, in the BigQuery destination.
It would be great if there was some additional functionality to do this via Fivetran. Probably not an automatic schema change detection and drop (I understand why Fivetran doesn't currently have this and wants to preserve backwards compatibility for schema changes). But, if there was a "special" option (maybe some one-off button for this?) to re-sync a postgres table and prune any columns that have been dropped in the source, in the destination table, that would be awesome.
-
Official comment
Hi Matt,
Thank you very much for sharing the feature request! Supporting automatic dropping of columns in the destination when the source schema changes is a valuable idea and could help streamline data management for many of our users. I've added your request to our feature improvement backlog for further review and prioritization.
We'll keep this thread updated with any progress or next steps. Thanks again for your input and engagement—your feedback helps us make Fivetran better for everyone.
Best regards,
Val Kulichenko
Please sign in to leave a comment.
Comments
1 comment