Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Destination Improvement: S3 Data Lake - Intelligent Tier Storage Class support

Answered
Ramki Rajendran User

We understand Fivetran performs the following maintenance periodically on S3 Standard Storage Class in AWS Glue Managed Iceberg table pipeline pattern.

  • Delete old Snapshots
  • Delete Orphan files
  • Delete old Metadata files

However on longer run to further optimize and keep the S3 Storage cost lower we are trying to see if we can transition old S3 objects loaded by Fivetran to Intelligent tier Storage Classes(Frequent, In-Frequent, Archive Instant Access Tier).

https://aws.amazon.com/s3/storage-classes/intelligent-tiering/

AWS is also recommending this.

https://aws.amazon.com/blogs/big-data/improve-operational-efficiencies-of-apache-iceberg-tables-built-on-amazon-s3-data-lakes/

We confirmed from Snowflake that they do support these S3 Storage Classes as well for Iceberg tables.

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Official comment
    Egidio Terra Fivetranner
    • Edited

    Hi Ramki,

    Apologies, this is a typo. The source of my response is the documentation page you refer to. It was supposed to read : " In fact, you can't use these storage classes."

    The other storage classes are compatible.

    Best,
    Egidio

     

  • Egidio Terra Fivetranner
    • Edited

    Hi Ramki,

    Thanks for the request. In fact, you can use these storage classes:

    • S3 Glacier Flexible Retrieval
    • S3 Glacier Deep Archive
    • S3 Intelligent-Tiering Archive Access tier
    • S3 Intelligent-Tiering Deep Archive Access tier

    Best regards,
    Egidio Terra