Connector Improvement: Allow adding custom columns to a destination table
Answered
Currently, when adding a new column to a destination table for a Fivetran connector, any future sync will nullify any values in that column in the MERGE statement that is issued by Fivetran.
Our use case was to add an ingestion_timestamp column that will default to CURRENT_TIMESTAMP, however, once we've done that, Fivetran's MERGE statement set the values to be null, which made the idea impossible.
It would be great to be able to define custom columns directly from the connector, or have some sort of setting that will let us instruct the syncs to not nullify unknown columns.
-
Official comment
Hi Shai,
Thank you for sharing your feedback! Could you let me know which source and destination you’re working with? This will help us investigate further and ensure we address your needs effectively. -
Hi Rean,
This was originally posted in regards to the "Google Cloud Function" connector, that fetches data from a specific cloud function we have, with a BigQuery destination.
I think the source (connector) is less relevant here and this might be relevant for any BigQuery destinated connection - the option to add a custom column that is not fetched from the connector, without having it nullified, would be great.
Use cases can be:
- Surrogate Keys that are generated automatically in the data warehouse but don't arrive from the source
- Auto-value columns like ingestion_timestamp or numeric indexes
- Clustering columns that are auto-calculated and extracted from existing columns
etc.
Please sign in to leave a comment.
Comments
2 comments