Destination Improvement: Partition Support Request for Fivetran-Databricks Integration
Answered
Hi Team,
We're currently using Fivetran to sync data from various source systems to our Databricks Bronze tables. However, Fivetran doesn't currently support creating partitions on destination tables during the sync process.
Challenge: Since we incrementally sync data for our Silver layer transformations, we need to query only new records from Bronze. Without partitioning, our queries perform full table scans, which will significantly increase costs as data volume grows.
Request: We need Fivetran to support table partitioning at the destination to enable efficient incremental reads.
Reference: This relates to our Fivetran ticket #327878.
Please let us know if you need additional details or would like to discuss further.
Best regards, Sanjeeb
-
Official comment
Thanks Sanjeeb,
If I understand correctly, you are looking for the ability to partition the data based on sync? For many customers, we see that Fivetrans behavior of Copy on Write naturally orders the data based on sync time. Fivetran also updates the Parquet level min/max stats so most query engines can push down filtering to the file level. This typically gives improved performance for incremental materialization.
Are you seeing full scans from Databricks? If so I would love to connect and learn more about your specific use case.-Casey
-
Hi Casey - First of all Happy New year and hope you had a fantastic new year celebration !!
Thank you so much for your reply. Yes, it is better to have a call and discuss in detail. Can we arrange a call next week Monday i.e 19th Jan 2026 1 PM BST and send me the invite ( sanjeeb.mohapatra@syngenta.com and sampath.thammana@syngenta.com).
Regards - Sanjeeb
-
Hi Team - Any update on this request. We have not received any call invitation.
-
Hi Sanjeeb,
I just sent an email with a meeting invite.
Thanks,
Casey
Please sign in to leave a comment.
Comments
4 comments