Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Fivetran HVA connector to provide an audit log or built-in reconciliation mechanism that tracks both the records changed at the source and the records written to the destination

Answered
Amar Patil User

Background / Context:
We are currently using the Fivetran HVA to sync data from SQL Server into our Snowflake environment.
From a data quality, compliance, and auditability standpoint, it is critical to ensure end-to-end data completeness — i.e., that all source records are successfully captured, transferred, and written to the destination.

At present, HVA does not provide a built-in record count or transaction-level validation mechanism, nor does it generate alerts for partial or incomplete loads. This makes it challenging to identify discrepancies in cases of network interruptions, schema changes, or CDC disruptions.

Feature Request:
We request the introduction of a native Record Count Auditing and Validation capability in Fivetran HVA that can:

  1. Capture record counts (or transaction counts) from both source and destination for each sync batch.

  2. Compare and log discrepancies automatically, flagging any mismatch or incomplete sync.

  3. Generate alerts or notifications through the Fivetran dashboard or webhook when differences are detected.

  4. Optionally, expose this metadata (counts, discrepancies, and status) via the Fivetran API for integration with monitoring tools.

Business Justification:

  • Ensures data integrity: Detects missing, partial, or duplicate data in real time.

  • Reduces operational overhead: Removes the need for external data validation scripts or monitoring pipelines.

  • Improves audit readiness: Provides traceable logs for compliance audits and data governance.

  • Enhances reliability: Builds greater confidence in automated data flows.

Current Limitation:
There is no existing alerting or validation mechanism within HVA that verifies the completeness of data transfer or provides source vs. destination counts post-sync.

Proposed Impact:
This feature would significantly improve the observability and trustworthiness of Fivetran-managed pipelines, aligning with enterprise-level data governance and regulatory compliance requirements.

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Joice John User

    Hi Amar

    We have a functionality called as Background Data validation which checks for data integrity after every sync. We are planning on surfacing this up to customers. Today its only for the managed service. Adding this functionality to HVA is not yet prioritized, but something we are looking into.

    Can you add some details which can help in prioritization : How you manage to do this today ? (via ad hoc scripts or reactive ? ). How often would you want to run this ? 

     

  • Amar Patil User

    Hi Joice,

    We don’t have an established background data validation process. After multiple clients reported a discrepancy, we began planning to create such a process internally.
    As of now, validations are done manually and reactively when an issue is reported. We intend to set up a daily validation process that compares data completeness and freshness between the source  and destination, validating record counts and totals.

    As we are building a Data and Intelligence platform, and our core value proposition (USP) is to provide timely and accurate insights to our clients. Having robust data validation built into the ingestion layer is critical to ensuring the accuracy and trustworthiness of the insights we deliver.

    Challenges with implementing this on our own:

    1. Parallel source connection requirement –
      To perform validation (e.g., record counts, sum of totals), we would need to create a parallel connection to the source system. This is difficult to build, introduces infrastructure complexity, and increases load on the source.

    2. Limited access for self-hosted clients –
      Around 40% of our clients are self-hosted or hosted by third parties, and we don’t have direct connectivity to their systems other than through Fivetran HVA. This significantly limits our ability to perform independent data validation.

    Having this functionality directly available within Fivetran would make validation consistent, reduce operational overhead, and ensure data integrity checks are available across all client types.