Use Case/Scenario:
You are configuring Databricks for optimized use with Fivetran.
Fivetran Recommendation:
- Always configure your Databricks warehouse according to the Fivetran setup guide
- Enable autoscale
- Enable ‘Always On’ only if you utilise a 5 minute sync frequency
Context:
- Autoscaling is enabled because a Databricks cluster will weight either towards Compute or Memory-Optimised depending on how long queries take per-table; this is determined by the amount of data being synced per-table
- ‘Always On’ is only useful for low frequency syncs because the Databricks cluster start time is relatively low (with a boot time of around 1 minute)
Considerations:
If you experience issues with Databricks performance and you believe this may be related to how Fivetran interacts with your destination, then please raise a ticket with the Fivetran Support team.