Support seamless resumption of sync from customer-managed destination backups for Disaster Recovery (Connector State Reset)
Which connector?:
SAP ERP on HANA (NetWeaver API)
Additional details:
Connector / Destination: SAP ERP on HANA (NetWeaver API) to Google Cloud BigQuery (Applicable globally to other connectors)
Problem Statement / Current Limitation: In a Disaster Recovery (DR) scenario where our destination project is completely compromised (e.g., cyberattack), we must set up a brand-new BigQuery project. Fivetran currently requires us to create a new destination and run a full Historical Sync.
This approach introduces two major problems:
-
Irreversible Data Loss for History & Soft Delete Modes: A historical sync rebuilds only the current state of the source. We permanently lose our SCD Type 2 history (History Mode) and previously soft-deleted records (Soft Delete Mode).
-
Source System Overload: Triggering a full historical sync on a core ERP system (SAP HANA) places immense performance strain on our operational databases and takes a significant amount of time, causing prolonged downtime.
Proposed Solution / Feature Request: Provide a mechanism (via UI or API) to align/reset Fivetran's internal connector state (checkpoints/cursors) with a user-restored destination backup.
If we restore our destination tables from our own BigQuery Snapshots or GCS backups taken at a specific timestamp, we want to inform Fivetran of this restore point so that the connector can seamlessly resume incremental extraction (CDC) from that exact position, without triggering a historical sync.
Business Value / Impact:
-
Data Integrity: Ensures strict compliance and auditability by preserving SCD Type 2 history and soft-deleted records during a catastrophic failure.
-
Reduced RTO (Recovery Time Objective): Allows us to resume data pipelines immediately after a backup restoration, instead of waiting days for a massive historical sync.
-
Operational Stability: Prevents sudden, heavy extraction loads on critical source systems like SAP ERP during a disaster recovery process.
Please sign in to leave a comment.
Comments
0 comments