Destination Improvement: S3 Data Lake - DataTypes (Preserve/Retain as in Source)
Answered
We understand from S3 Data Lake Destination Documentation that Fivetran converts to following datatypes as listed here
https://fivetran.com/docs/destinations/s3-data-lake#typetransformationandmapping
However this is causing precision issue and downstream breaking changes for users and reporting.
To give some background
MySql(binlog) -> Fiveran -> Snowflake(Physical Table) : Data types are preserved from MySql to Snowflake.
But if we do the following
MySql(binlog) -> Fivetran -> S3 Data Lake <- Snowflake Glue Iceberg table: Datatypes are not preserved in Glue Catalog and Snowflake Glue Iceberg table with source MySql.
Our end goal is to preserve the datatypes in destination (Snowflake) same as MySql. Due to this limitations it limits our ability to use this destination connector to it's full extent as it adds additional complexity/overhead to maintain the datatypes.
More details can be found in this ticket #262003
-
Official comment
Thanks for bringing this to our attention. We have confirmed the behavior and are going to find time to prioritize the fix where MDLS preserves destinations types where possible.
Please sign in to leave a comment.
Comments
1 comment