Connector Improvement: Support passing duplicate (when normalized by Fivetran) fields through Email (and all other) connectors
Currently, if you have two or more fields in a source file (ex: CSV) that normalize per Fivetran's normalization rules to the same field name in your destination, Fivetran auto-drops all but one of the dups, and doesn't give you any option to preserve all the data by, say, renaming the duplicates.
In many cases it's not possible or feasible to change the source fieldnames, which is the only 'good' way around this problem, so a solution is needed at the Fivetran load layer.
Instead, Fivetran should alert you (preferably at setup/test time!) to the presence of dups, clearly show them by source name (which is often differentiated) as well as Fivetran normalized name, and let you alter the Fivetran/destination names to make them unique so that you can include all your source data [or let you explicitly exclude one or another of the field duplicates].
This should work regardless of whether the source field names are the same in the source data or if they are different, and only the same post-Fivetran normalization. This COULD be something exposed and editable in the 'schema' control feature, since that's an intuitive place to look to make source column-level adjustments to data loads.
I'd like to see this in the Email connector, but it should also be rolled out to all other connectors since the exact same type of issues can crop up in any type of data source, so hopefully it can be a global handling vs something Fivetran needs to create per connector.
This is related to this ticket I originally filed: https://support.fivetran.com/hc/en-us/requests/117012
And there are also two other requests that relate to this, one making the same-ish request for S3: https://support.fivetran.com/hc/en-us/community/posts/10604388021399-Connector-Improvement-Handle-duplicate-columns-in-S3-connector
, and another related to duplicate field handling: https://support.fivetran.com/hc/en-us/community/posts/1500000428541-Ordering-of-Duplicate-Columns
Please sign in to leave a comment.
Comments
0 comments