Destination Improvement: Columns with TIMESTAMP data type at source are synced as base64 encoded value in Redshift warehouse. Sync TIMESTAMP to Redshift without base64 encoding.
TIMESTAMP (or ROWVERSION) is an automatically incrementing field on a table any time a row is inserted or updated. This is a fact guaranteed by SQL Server that is very convenient to use as a way to observe sequential changes to the table. Many application programs rely on TIMESTAMP.
By base64 encoding a TIMESTAMP value, you are breaking the sequential property. To illustrate this consider the sequence 1, 2, 3 which is base64 encoded to MQ==, Mg==, Mw== respectively. The encoded ordering is then Mg== (2) < Mw== (3) < MQ== (1) which no longer preserves the order.
As another example, a SQL Server UNIQUEIDENTIFIER is mapped to a VARCHAR in Redshift without base64 encoding as Redshift does not support UUID type. UNIQUEIDENTIFIER is a specific type (it’s a sequence of bytes too) and that maps well -- so why is TIMESTAMP any different?
TIMESTAMP needs to have its own mapping as a unique type. We’d be happy if Fivetran can map TIMESTAMP to either a VARCHAR without base64 encoding (as the fixed-length hex string representation) OR a BIGINT (which is also 8 bytes) without Base 64 encoding.
Please sign in to leave a comment.
Comments
0 comments