Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Amazon Selling Partner de-duplication "opt out" toggle

Answered
Nika McGill User

Which connector?:

Amazon Selling Partner

Additional details:

A toggle to opt-out of the native de-duplication. Currently the de-duplication is resulting in real data loss.

Our source contains records which resemble duplicates (do not have an ID marking them as unique) but that contribute to accurate reimbursement values if not fully ingested. The current de-duplication is resulting in about 5% of the source records being left out, and values not matching between source & target.

Having a toggle to allow "duplicates" would enable us to load the source data fully.

Thank you!

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Vignesh Rengaraj Fivetranner

    Hi Nika,

    I believe our Support team discussed this issue with us, and I had one follow-up question.

    Could you confirm whether these are exact duplicates, meaning the record ID and all other column values are identical across the rows? If so, could you share the business use case or Amazon Selling Partner API behavior that results in these identical rows being returned? This context will help us better understand the issue and determine how it should be handled.

    Best,
    Vicky
    Fivetran Product Team

  • Fabian Cabana User

    Hi Vicky!

    So for context, during UAT, we identified differences between the Settlements and Reimbursements reports downloaded directly from Amazon Seller Central (ASC) and the corresponding data ingested through your platform.

    The source reports downloaded from ASC contain more rows than the data available through your platform. Upon investigation, we found that the missing rows are duplicates: they contain identical values across all available columns. From an analytical perspective, these records can be removed by applying DISTINCT; however, our requirement is for you to ingest the reports exactly as provided by ASC—or to have the option to toggle this behavior on/off—since the downloaded ASC reports are considered our source of truth.

    We understand that your current process performs deduplication by generating a primary key using the available fields. Because these duplicate rows have identical values across all fields, they result in the same primary key and are therefore deduplicated during ingestion.

    Our requirement is for reports built using the ingested data to match the corresponding raw ASC reports 100%. Because of the deduplication performed by the connector, approximately 5% of the records are being excluded, which causes discrepancies in our aggregations and makes the resulting data unsuitable for downstream reporting.

    Let me know if you need any more detail! 

    Thanks!