Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Destination Improvement: Ability to do compare with parquet files on S3

Answered
Thomas van Zyl User

Hi

We use HVR6 to replicate data from a SQLServer source to parquet files on AWS S3. Currently only CSV or XML file formats are supported for compare and NOT parquet. We need the ability to do a compare to ensure the source is in sync with the parquet files on AWS S3. Link to documentation: https://fivetran.com/docs/hvr6/action-reference/integrate#comparepattern

Thanks

Please sign in to leave a comment.

Comments

1 comment

Date Votes
  • Mark Van de Wiel User

    Hi Thomas,

    We support compare on S3 as a destination. However you must define Hive External Tables (e.g. on EMR) in order to enable this. Of course this means you have the additional cost of running Hadoop.

    HVR's compare capability is built on top of the assumption that we can run SQL statements on either side. The only exception we have to this is with delimited files.

    Hope this helps.
    Mark.