Skip to main content
Documentation
Feature Requests Trust Center Status Community

Community

Destination Improvement: S3 Data Lake - Date Partitions

Planned
David Rooney User

The S3 data lake feature uses Parquet format and normalised tables. We need a way to reduce the data scanned in queries when filtering by dates.

So if there was an option to partition data by Year, Month, Day then we could reduce the data scanned 1000x. 

I would expect this to be preconfigured when selecting the Schema in a connector. It would define the partitions for default properties such as Dates.

 

Please sign in to leave a comment.

Comments

1 comment

Date Votes
  • Official comment
    Coral Trivedi Fivetranner

    Hi David, 

    Thank you for your feedback! We are currently developing a partitioning strategy for the S3 Destination and plan to start implementation in Q4 of this year. 

    Stay tuned!