Destination Improvement: Databricks Destination Cluster selection
Completed
From Boxycharm/BFA:
We have a feature request - currently the fivetran clusters only run on an all purpose cluster which is the most expensive cluster type in databricks.
It would be good if it could run on a jobs or even better the lite cluster.
To actually make this usable we would need to be able to change clusters though without triggering a full reload.
-
Official comment
Hi Sean,
Thank you for this feedback! Our Databricks destination does support two cluster options: All-Purpose Compute and SQL Compute, the latter of which is significantly cheaper.
We cannot run on the Jobs and Jobs Light compute because the queries we currently require features at the higher tier. You can view our documentation on how to get set up with the SQL Compute option here: https://fivetran.com/docs/destinations/databricks/databricks-setup-guide#connectsqlendpoint
Please let me know if you have any other questions!
Please sign in to leave a comment.
Comments
1 comment