Connector Improvement: Databricks connector to perform parallel extraction of the tables as per schema definition
Not planned
We encountered a major performance issue where databricks connector takes ages to replicate initial load.
This doesn't meet our SLA's . Requesting improvement to further optimise the extraction process to allow for parallel tables extraction instead of doing 1 by 1
The Databricks connector extracts data from configured tables one at a time, rather than in parallel. This sequential processing significantly increases the overall sync duration. – Fivetran must fix this issue or create multiple connectors for parallelism
-
Official comment
Hi Yash Pal,
Thank you for taking the time to share your suggestion regarding parallel extraction of tables using the Databricks connector. We appreciate your feedback—this is a valuable idea that could help improve data extraction performance and scalability.
I've added your request to our connector improvements backlog for consideration by our product and engineering teams. Please feel free to provide any additional details and requirements. We’ll keep this thread updated with any progress or changes related to your request.
Thanks again for your input!
Best regards,
Egidio Terra
Please sign in to leave a comment.
Comments
1 comment