Issue
A Duplicate Columns alert appears in the Fivetran dashboard, or as message data extracted by the Fivetran Log connector. The source system does not appear to show any duplicate columns.
Environment
Fivetran dashboard
Resolution
Please change the naming convention of your source system to work within the parameters of Fivetran’s naming convention.
Context
Fivetran’s naming convention may identify columns as duplicates even if your source system enforces uniqueness on the naming convention.
For example, consider the following unique column names:
- systemID
- system__ID
- system_ID
Whilst one column name is camel-cased, one has a double-underscore and one a single underscore, Fivetran normalizes them all to the same column name:
- system_id
Considerations
There will be scenarios where Fivetran excludes duplicate columns from being written to the warehouse where you don’t require that data for your downstream modelling.
If this is the case:
- Deselect those fields from being synced by Fivetran.
- Leave the naming conventions as they are in order to not disrupt functionality of the source system.