Destination Improvement: Redshift destination: Super column needs the TRUNCATECOLUMNS option
Hi!
While testing the experimental support for the Super column in Redshift we stumbled upon a scenario where the sync fails. The failure happens when source table has a json column with values that don't fit into a Redshift VARCHAR type. This can be mitigated by adding the TRUNCATECOLUMNS option to the COPY command when moving the data.
Please consider this as an improvement to the connector!
-
Looks like this is the original feature request to support Redshift's SUPER type for semistructured data: https://support.fivetran.com/hc/en-us/community/posts/1500000908702-Destination-Improvement-Redshift-SUPER-type
However, the SUPER type can store 16MB while a varchar can store a max of 64kb, and it looks like the method in which this was implemented in Fivetran truncation can still occur on a payload bound for a SUPER type column if it exceeds 64kb.
Any updates on this one?
Please sign in to leave a comment.
Comments
1 comment