Destination Improvement: Merge option for new data files in AWS S3
Answered
The sync behaviour for AWS S3 is to create a new file for each table with all newly inserted records since the last sync. However, for our uses we would expect the tables to be fairly static and only have incremental inserts. Is there a way to have a single file in s3 for a table, and then update that file with each sync?
-
Official comment
hi Mike,
Iceberg treats data files as immutable. This means once a file is written, it’s never changed. Instead of modifying files, Iceberg writes new files to represent the updated state.
Best regards
Please sign in to leave a comment.
Comments
1 comment