Support seamless resumption of sync from customer-managed destination backups for Disaster Recovery (Connector State Reset)
AnsweredWhich connector?:
SAP ERP on HANA (NetWeaver API)
Additional details:
Connector / Destination: SAP ERP on HANA (NetWeaver API) to Google Cloud BigQuery (Applicable globally to other connectors)
Problem Statement / Current Limitation: In a Disaster Recovery (DR) scenario where our destination project is completely compromised (e.g., cyberattack), we must set up a brand-new BigQuery project. Fivetran currently requires us to create a new destination and run a full Historical Sync.
This approach introduces two major problems:
-
Irreversible Data Loss for History & Soft Delete Modes: A historical sync rebuilds only the current state of the source. We permanently lose our SCD Type 2 history (History Mode) and previously soft-deleted records (Soft Delete Mode).
-
Source System Overload: Triggering a full historical sync on a core ERP system (SAP HANA) places immense performance strain on our operational databases and takes a significant amount of time, causing prolonged downtime.
Proposed Solution / Feature Request: Provide a mechanism (via UI or API) to align/reset Fivetran's internal connector state (checkpoints/cursors) with a user-restored destination backup.
If we restore our destination tables from our own BigQuery Snapshots or GCS backups taken at a specific timestamp, we want to inform Fivetran of this restore point so that the connector can seamlessly resume incremental extraction (CDC) from that exact position, without triggering a historical sync.
Business Value / Impact:
-
Data Integrity: Ensures strict compliance and auditability by preserving SCD Type 2 history and soft-deleted records during a catastrophic failure.
-
Reduced RTO (Recovery Time Objective): Allows us to resume data pipelines immediately after a backup restoration, instead of waiting days for a massive historical sync.
-
Operational Stability: Prevents sudden, heavy extraction loads on critical source systems like SAP ERP during a disaster recovery process.
-
Hi Takashi,
Thank you for describing this disaster-recovery scenario so clearly. We agree that a historical sync is not equivalent to restoring a destination backup. It can rebuild the current source state, but it cannot recreate History Mode versions or previously deleted records that exist only in the backup. In addition, a complete historical sync can result in considerable recovery time and source-system load.
At present, Fivetran does not provide a supported mechanism for restoring a connection to an earlier connector checkpoint. This limitation is not specific to SAP ERP on HANA but applies across Fivetran.
Addressing this request would therefore require a Fivetran-wide approach, together with connector-specific recovery implementations. This makes feasibility assessment and roadmap planning more complex.
We will discuss the request internally and provide an update when we have greater clarity on what may be possible.
Best regards,
Edwin
Please sign in to leave a comment.
Comments
1 comment