Connector Improvement: Google Cloud PostgreSQL ingestion of views
Hi all,
Can you confirm that the Google Cloud PostgreSQL connector does not currently support the ingestion of views?
I made an attempt to do so, but only tables are being ingested.
I see this post asking something similar from a year ago. Is there any update?
https://support.fivetran.com/hc/en-us/community/posts/360051897174-Load-Views-from-Postgres-RDS-
Edit: Fixed misspelling of PostgreSQL (PostgresSQL)
-
Official comment
Hi Steven,
Kevin from the product team here. While we do not currently support syncing from views, this is something on our backlog and we are looking into how we can support it. One of the issues with syncing views is that databases often do not log changes that affect views, so we cannot replicate them using our current CDC processes.
Could you describe your use case for these views and what sort of queries you will be running on them?
-
Hi Kevin,
I'm responding quite late.
I believe the original use case was to use a view for table and column renaming in order to give the consumer of the data a consistent interface on which to query against.
However, we have a new use case where there is sensitive data (at the record level) that we don't want to import to Snowflake, and importing a "cleaned" view from PostgreSQL would make this easier.
I'm not quite sure what you mean by "what sort of queries you will be running on them?".
I have the business purpose of the original case, which I don't want to share here.
But the Data Engineering purpose of the new case was to not ingest certain sensitive data to Snowflake. But the view's data would eventually be exposed to our EDW most likely. Sort of a scheduled select * I think.
Please sign in to leave a comment.
Comments
2 comments