Destination Improvement: Flexibility and Adding new audit columns
Not planned
Hi Team,
While setting up pipeline on destination type Managed Data Lake Service, we are noticing two audit columns in place: _fivetran_deleted and _fivetran_synced.
However, we would prefer below enhancements/changes related to audit columns:
- Either allow renaming of audit columns(_fivetran_deleted, _fivetran_synced) or allow flexibility to create user-defined custom-mirrored columns where audit columns values can be copied to.
- Allow below audit columns to be in place:
- source_modified_at (timestamp when the row was updated/deleted/inserted on source)
- fivetran_ingested_at (timestamp when fivetran started ingesting/processing the row)
- sink_loaded_at (timestamp when the data is being inserted/updated/deleted on iceberg tables)
Let us know in case of any concerns related to the above feature addition.
-
Official comment
Hi Chandan,
Thanks for adding this feature request. These system columns are used by our internal data loading processes, including support for Soft Delete mode and can't be changed. As an alternative, you could create a view on top of the tables with the columns renamed to your preference.
Best regards,
Please sign in to leave a comment.
Comments
1 comment