Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Destination ADLS2 - Data append in parquet format instead of delta

Safoora Jalali User

Which destination?:

All adls2 destination (MDLS)

Additional details:

Allow writing parquet files which get appended instead of delta format

Please sign in to leave a comment.

Comments

3 comments

Date Votes
  • Official comment
    Casey Karst Fivetranner

    Thanks for the request! 

    Fivetran Managed Data Lake (ADLS) writes the data as both Apache Iceberg and Delta so that the data can be consumed by downstream query engines. When the ADLS destination in HVR was created this functionality was not possible as table format support was nascent at best. 

    If Databricks did not execute VACUUM on externally managed tables and as you say remove active files would this still be a problem architecturally for you? 

  • Safoora Jalali User

    Similar to like we have in HVR

  • Fivetran Syncreon User

    This is a critical feature to have, as Databricks VACUUM may remove or make the landing data unavailable. Also, from a consistency and uniformity perspective, if HVR and Hybrid can write data to ADLS in Parquet format, why can’t the SaaS connection support the same capability?