Connector Improvement: Fix limitation on Databricks Connector for large STRING data
CompletedThis relates to this support ticket
Databricks to RDS Error - Required array length 2147483639 + 158 is too large – Fivetran Support
From the ticket we found that fivetran has a limit on the STRING column it can replicate in fivetran. This is due to the function used to generate the fivetran ID.
This behavior is problematic because for our use case, JSON data is captured in a single STRING column. Hence it will not be possible to use fivetran replication if a string is of a certain size.
Raising this feature request to resolve limitation and handle large STRING columns for the databricks connector.
-
Official comment
Hi Joel,
Thanks for your request. I’m happy to share that this work has now been completed and deployed.
Best regards,
Egidio Terra
Please sign in to leave a comment.
Comments
1 comment