Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Aurora PostgreSQL -> S3 allow fields deleted from source to not be sent in the file

Planned
Samantha Duggan User

When syncing an Amazon Aurora PostgreSQL database to S3, fields that have been removed from the database are still sent to S3 with null values. We'd like to request the ability to cease sending these fields as they bloat the file and increase complexity when validating files against our data contracts.

Please sign in to leave a comment.

Comments

4 comments

Date Votes
  • Egidio Terra Fivetranner

    Hi Samantha,

    An improvement is currently underway that will automatically remove columns in the destination when they are deleted from the source. Could you elaborate on your expectations for how existing values in the destination should be handled in such cases?

    Best regards,

  • Samantha Duggan User

    Thank you, Egidio!

    For syncs to S3, we'd personally like the option to only send the fields that are currently in the database. As these syncs create distinct files in S3, we have no need to handle existing values in the destination. We treat each new file as a complete dataset.

  • Egidio Terra Fivetranner

    Hi Samantha,

    Thanks for the clarification. When you mention syncing to S3, are you referring to writing data into an S3-based data lake?

    Best,

  • Samantha Duggan User

    Yes, we are having this issue when our destination is "Managed Data Lake Service - AWS"