Scenario/Use Case:
You are experiencing a ‘Duplicate Columns’ alert in the Fivetran UI or as message data extracted by the Fivetran Log Connector. However, you have checked your source system and cannot identify any duplicate columns.
Fivetran Recommendation:
- Change the naming convention of your source system to work within the parameters of Fivetran’s naming convention
Context:
Fivetran’s naming convention may identify columns as duplicates even if your source system enforces uniqueness on the naming convention.
For example, consider the following unique column names:
- systemID
- system__ID
- system_ID
Whilst one column name is camel-cased, one has a double-underscore and one a single underscore, Fivetran normalises them all to the same column name:
- system_id
Considerations:
There will be scenarios where Fivetran excludes duplicate columns from being written to the warehouse where you don’t require that data for your downstream modelling.
If this is the case:
- Deselect those fields from being synced by Fivetran
- Leave the naming conventions as they are in order to not disrupt functionality of the source system