Connector Improvement: s3 connector for s3 replication bucket.
There may a myriad of reasons for having a replication buckets. One of which is if the organization has a team contracted to build out an application. They contractor can choose to replicate the bucket for the internal data engineering team to work with.
In such a situation, the s3 connector does not work with the replication bucket. It looks like the data that gets lost are the records that were added to the s3 replication bucket during a sync. If a sync starts at 1PM and completes at 1:05PM. Any records being pushed to the replication buckets between those 2 times are overlooked in Fivetran's connector setting.
Our hope is that the current connector makes the change that was overlooked in this setup. If not maybe have an additional s3 replication connector. Everything can be the same other than the setting of when it should look back for to find incremental changes.
Thank you
-
Official comment
Hi Ken,
Thank you for using Fivetran and providing such a detailed explanation of your need.
We are exploring what it would take to support replication buckets as they don't guarantee the order of copying the data from source bucket.
Additional upvotes from other customers who use replication buckets and need this also will help us with prioritization.Best regards
Alison
Please sign in to leave a comment.
Comments
1 comment