Load Views from Postgres RDS
Answered
I would like to be available to select views as a source to load from my Postgres RDS connector. Currently, we have a large table that we do not want to load via Fivetran because it will eat away at our MAR. We know we only want to load a subset of this data, and have a view in the source that filters for this subset.
I would like to be able to load this view as a table in our Redshift warehouse. Currently, we can only load tables from a source.
-
Official comment
Hi Kyle,
Loading from views is a common request, but poses a key problem as views do not generate change data. For Postgres, we will not be able to read incremental syncs from a view.
One possibility is to introduce a snapshot-based sync, which will just fully re-import the table each time at a periodic interval. This is on our radar but we do not currently have a timeframe for this.
-
Hey Kevin Kim,
As you mentioned this is likely a popular request as we'd be eager for something similar as well albeit on the Azure platform. While snapshot sync would be a good starting option (much better than nothing), would it be possible to somehow allow users to select their "primary" and "high watermark" columns when setting up the schema, and use this to support the incremental load?
You can't possibly imagine how happy it would make me :D
Please sign in to leave a comment.
Comments
2 comments