Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Minimize Extract Duration Post Query Execution in Databricks Connectors

Answered
Amitoj Singh User

We’ve observed that while SELECT queries against Databricks complete quickly, the overall extract phase on Fivetran continues for a significantly longer period. This extended time appears to be due to Fivetran’s internal processing after the result set is retrieved, such as transforming, flattening, and preparing files for load.

We request Fivetran to optimize or reduce this processing time during the extract phase to improve overall sync performance.

Business Impact:

  • Data Delays: Slower syncs affect timely reporting.

  • SLA Risks: Extended durations make it harder to meet data freshness targets.

Please sign in to leave a comment.

Comments

1 comment

Date Votes
  • Official comment
    Egidio Terra Fivetranner

    Hi Amitoj,

    This request to optimize the extract phase post-query execution on Databricks connectors is noted and has been added to our feature improvements backlog. To help us better prioritize and scope this work, could you share more details about the volume and structure of data being processed, or specific bottlenecks observed in your workflow? Understanding your use case and pain points in greater detail would assist us in evaluating targeted enhancements.

    Updates regarding any progress will be communicated on this thread.

    Best regards,