Destination Improvement: Azure Data Lake Storage Delta Change Data Feed (Databricks)
AnsweredWe are currently evaluating a move away from the Databricks warehouse-based destination to the ADLS / Unity Catalog destination. Cost benefits from reduced warehouse DBUs need to be offset against increased costs in table processing, in some cases. Specifically, in the case of a Delta Live Table or Spark Structured Streaming pipeline using a Fivetran table as a source, it needs either an append-only source or CDF enabled when there are update/delete commits (i.e. merge commits) in order to leverage the incremental processing benefits available.
If you could enable CDF for the ADLS destination, the cost benefit of moving to ADLS would be more clear: losing the warehouse DBUs would compound with DBU savings from incremental table processing.
-
Official comment
Hi Thomas,
Thank you for taking the time to share your suggestion for Azure Data Lake Storage Delta Change Data Feed (Databricks) as a destination improvement. We appreciate your engagement—the idea is valuable and aligns with our goal to support robust data integration workflows.
I've added your request to our feature improvements backlog for further review.
We'll make sure to keep you and the community updated on this thread as we evaluate and make progress on this request.
Thanks again for your input!
Best,
Casey Karst
Please sign in to leave a comment.
Comments
1 comment