Handle normalized column-name collisions in BigQuery without dropping source columns
Which destination?:
BigQuery
Additional details:
When replicating data from Db2 for i to BigQuery using Fivetran, two distinct source columns can normalize to the same destination column name.
Example:
Example:
SDINV#SDINV$
Under Fivetran naming, both normalize to:
sdinv_
Because of this collision, Fivetran omits one of the columns as a duplicate. This causes loss of source data during replication.
We also evaluated Source naming, but it does not solve this issue for BigQuery because the original source characters cannot always be represented as valid destination column names in Fivetran’s BigQuery loading path.
Requested enhancement:Please support deterministic handling of normalized column-name collisions so that both source columns can be retained in the destination. For example:
- automatically assign unique destination names such as
sdinv_andsdinv_2, or - allow configurable destination-side column renaming for colliding columns
Business impact:Both source columns contain required business data for downstream processing. The current behavior prevents complete ingestion into BigQuery and introduces data loss for affected tables.
Desired outcome:Allow Fivetran to preserve all source columns even when multiple source column names normalize to the same BigQuery-compatible destination name.
Please sign in to leave a comment.
Comments
0 comments