Which destination?:

Managed Data Lake S3

Additional details:

We would like the ability to manage Apache Iceberg tables within the data lake. Specifically, we need to:

1. Partition Iceberg tables so that very large tables can be queried efficiently without requiring full table scans.

2. Update and delete data from Iceberg tables. Our specific use cases include:

   • Removing PII from ingested data after a specific number of days (e.g., keeping full details for X days, then running a job to purge or mask PII columns).

   • Deleting rows older than X years.

3. Run table optimization and compaction to maintain and improve query efficiency.