Support column type changes without CTAS -- preserve CDF history on Databricks Delta tables
PlannedWhich destination?:
Problem:
When a column type changes in a MySQL source
(e.g. INT → BIGINT), Fivetran performs a
CREATE OR REPLACE TABLE AS SELECT (CTAS) on
the Databricks Delta destination table instead
of an in-place schema evolution. This completely
wipes all Change Data Feed (CDF) history on the
affected table.
Environment:
Source : Aurora MySQL 3.x
Destination : Databricks Unity Catalog
HVA version : 6.2.0/45
Proxy Agent : 1.2.3
Databricks DBR : 18.0.x
Delta Sharing : No
What we tested:
We enabled delta.enableTypeWidening=true
on affected tables but Fivetran still
triggered CTAS. The property had no effect
on Fivetran's behavior.
Expected behavior:
Fivetran should handle column type widening
(e.g. INT → BIGINT) using Databricks native
ALTER TABLE ... ALTER COLUMN TYPE or
delta.enableTypeWidening without recreating
the table, so CDF history is preserved.
Impact:
CDF history is the foundation of our
historical reporting and SCD2 views.
Every column type change in MySQL permanently
wipes CDF history on that table breaking
all downstream historical queries.
We are now fully live on prod with this
limitation as a known risk.
Support ticket reference: #396359
-
Hi Tarun,
Thanks for reaching out! We understand the importance of preserving CDF history and appreciate you flagging this in such detail.
Support for native in-place type widening is on our roadmap and currently under development — we'll share an update once it's ready.
Best
Best,
Please sign in to leave a comment.
Comments
1 comment