Managed Data Lake S3 - Iceberg table management
Planned
Which destination?:
Managed Data Lake S3
Additional details:
We would like the ability to manage Apache Iceberg tables within the data lake. Specifically, we need to:
-
Partition Iceberg tables so that very large tables can be queried efficiently without requiring full table scans.
-
Update and delete data from Iceberg tables. Our specific use cases include:
-
Removing PII from ingested data after a specific number of days (e.g., keeping full details for X days, then running a job to purge or mask PII columns).
-
Deleting rows older than X years.
-
-
Run table optimization and compaction to maintain and improve query efficiency.
-
Official comment
Hi Goncalo!
Thanks for the feedback. We are about to release a secondary user credential which has write capability to our MDLS tables. This enables you to solve 2 quite easily.
While you could theoritcally use the write credentials to run compaction, I would love to know more about your data sets and file sizes. Fivetran uses 32MB as the target file size because we have seen no degradation of r performance as the files get smaller and it improves write performance.
Finally partitioning of existing tables will not be supported. We are looking to adding data layout optimizations (either partitioning, sorting or clustering) to improve read performance.
If you have time I would love to talk more about your specifics. Feel free to reach out to casey.karst@fivetran.com and we can chat more! -
Hello Casey
We are about to release a secondary user credential which has write capability to our MDLS tables. This enables you to solve 2 quite easily.
Write credentials would be great! Out of curiosity do you have a rough timeline for that?
Please sign in to leave a comment.
Comments
2 comments