Hi Team,

As discussed during our calls. It has been ovbserved that current self-healing behaviour of fivetran severely restricts us from managing the changes gracefully. This is big risk as far as reliability is concerned.

Current behaviour is performing resyncs when a new column is added to source table. Considering databricks is big data source. Performing a full resync from there to destination as a default "approach" is fraught with danger. We are looking at an ability where is there is a change in our source, we are notified and pipeline should pause. There should be effective direction on how to then restart by performing required steps.

To detail we can think of below scenarios.

• When “Block All” is enabled and new columns are added (Breaking Change):
  • Connector should fail with a clear reason (schema change).
  • No automated re-syncs.
  • Ability to resume safely after applying the defined change process.
• When “Allow Columns” is enabled and new columns are added (Non-Breaking Change):
  • Connector should process new columns and apply incremental records.
  • No automated re-syncs.
• When columns change datatype (Breaking Change):
  • Connector should fail with a clear reason.
  • Change communication managed upstream/downstream.
  • Ability to resume connector (either full re-sync or incremental based on change severity).
  • No automated re-syncs.