Configurable Timestamp Mapping from Databricks to Redshift
Dear Fivetran Support Team,
I hope this message finds you well.
We are currently using Fivetran SaaS to sync data from Databricks to Redshift. We have noticed that timestamp columns of type "timestamp without timezone" in Databricks are being mapped by Fivetran to Redshift’s timestamp with time zone (TIMESTAMPTZ) type by default.
For our use case, it is important that these timestamp columns be stored in Redshift as timestamp without time zone (i.e., without timezone information) to maintain consistency with existing database schema and application logic.
Currently, there does not appear to be a configurable option in Fivetran to control this data type mapping directly. We are therefore forced to perform additional transformations post-sync to convert timestamp columns to the desired type, which adds complexity to our pipeline.
We would like to kindly request the addition of a feature that allows configurable timestamp data type mapping in the Databricks to Redshift connector. Specifically, the ability to choose or enforce timestamp without time zone for timestamp columns during sync without requiring manual transformations.
Such a feature would improve data type consistency, reduce transformation overhead, and enhance the overall usability of the connector for our team and potentially other users with similar needs.
Thank you very much for considering this enhancement. Please let us know if you require any additional details or context from our side.
Best regards,
Dileep
Please sign in to leave a comment.
Comments
0 comments