Destination Improvement: Support row tracking on delta tables in MDLS
AnsweredWe want to have row tracking on delta tables in MDLS. We need it for better incremental processing in our pipelines.
Row tracking was introduced in Delta Lake 3.2.0, with Delta protocol requirements:
- minWriterVersion=7
- minReaderVersion=1
Current version of Delta protocol in MDLS destination:
- minWriterVersion=2
- minReaderVersion=1
As mentioned in this Fivetran support article:
We use the Delta Standalone Java library to create and manage tables. This library supports writing and reading Delta Lake tables with protocol version (writer: 2, reader: 1).
To support more recent versions of Delta lake protocol and its features, the Fivetran backend would have to switch from Delta Standalone library (which is now officially deprecated) to Delta Kernel APIs.
-
Official comment
Thanks Pavel,
This is something we have on the backlog and are looking to invest in in the future.-Casey
-
Hello Casey,
Thanks for the information. Please keep us updated on any progress related to this feature, so we could plan accordingly.
Thank you!
Please sign in to leave a comment.
Comments
2 comments