We occasionally encounter duplicate records in our Snowflake destination table, even though the source table contains no duplicates and has a properly defined primary key. Because Snowflake does not enforce primary key constraints, these duplicate records persist in the destination.

When duplicates occur, performing a connector resync does not remove them. The only current remediation is to manually identify and delete the duplicate records from the destination table.

Feature Request

1. Duplicate remediation during resync

   • During a resync, validate destination records against the source and automatically remove duplicate rows that share the same primary key.
   • This would ensure that a resync restores the destination table to a clean, source-aligned state without requiring manual intervention.

2. Proactive duplicate detection

   • Detect potential duplicate primary keys during normal sync operations.
   • For example, if an update operation is expected to affect 5 rows based on source data but affects 8 rows in the destination, generate an alert or warning indicating that duplicate records may exist.
   • Providing visibility into these conditions would help customers identify and resolve data integrity issues before they impact downstream reporting and analytics.

Business Impact

Duplicate records can lead to inaccurate reporting, data quality issues, and manual cleanup efforts. Automated detection and remediation would improve trust in replicated data while reducing operational overhead.