What is the Optimal Configuration of Databricks for Fivetran?

Use Case/Scenario:

You are configuring Databricks for optimized use with Fivetran.

Fivetran Recommendation:

• Always configure your Databricks warehouse according to the Fivetran setup guide
• Enable autoscale
• Enable ‘Always On’ only if you utilise a 5 minute sync frequency

Context:

• Autoscaling is enabled because a Databricks cluster will weight either towards Compute or Memory-Optimised depending on how long queries take per-table; this is determined by the amount of data being synced per-table
• ‘Always On’ is only useful for low frequency syncs because the Databricks cluster start time is relatively low (with a boot time of around 1 minute)

Considerations:

If you experience issues with Databricks performance and you believe this may be related to how Fivetran interacts with your destination, then please raise a ticket with the Fivetran Support team.