Enable HVR Compare Utility Support for Type JSON/parquet Files Stored in AWS S3 Targets
Answered
Which connector?: AWS S3 as target
Additional details:
As part of the TMNA migration from Qlik Replicate to Fivetran HVR, we extensively use AWS S3 JSON file targets for downstream data integration and application consumption. HVR currently supports generating JSON files to S3 through
FileFormat=Json; however, the Compare utility does not support JSON file locations as a compare source or target.
When attempting to execute Compare against a JSON/S3 location, HVR returns:
F_JR0F08:File capture using action FileFormat with parameter Json is currently not supported.
This prevents users from validating data consistency between source databases and JSON file-based targets.
Business Need
Many organizations use S3 JSON outputs as the final delivery layer for:
- Data lakes
- Analytics platforms
- Downstream application feeds
- Cloud-native integrations
- Kafka/S3 landing zones
Currently, there is no native mechanism within HVR to validate that JSON files generated in S3 match the source data.
Requested Capabilities
- Compare Source Database ↔ JSON Files in S3.
- Compare Database Target ↔ JSON Files in S3.
- Compare JSON Files ↔ JSON Files.
- Support:
- Row count validation
- Primary key comparison
- Non-key column comparison
- Difference reports
- Scheduled compare jobs
- Native HVR compare reporting
Business Benefits
- Faster migration testing and cutover validation.
- Reduced operational effort.
- Improved CDC data quality verification.
- Better support for modern data lake architectures using S3 JSON targets.
- Feature parity with enterprise validation requirements commonly used in Qlik Replicate environments.
-
Hi Naresh,
Thank you for the detailed write-up. The detail in the request makes the operational impact clear.
To summarise what you're asking for:
- Compare utility support for JSON and Parquet files in S3: extend the HVR Compare utility to accept S3 JSON/Parquet locations as a valid compare source and target, covering database-to-file, file-to-database, and file-to-file comparisons. Today this returns F_JR0F08 and is not supported.
- Comparison depth: within those comparisons, support row count validation, primary key comparison, and non-key column comparison.
- Reporting and scheduling: produce difference reports through native HVR compare reporting, and support scheduling compare jobs rather than running them manually.
These are significant, unplanned engineering efforts that touch multiple layers of the product and cannot be an easy delivery.
I have logged this as a feature request and will discuss it with the engineering team. I will come back to you once we have more clarity on workarounds or engineering solutions.
Best regards,
Edwin
Please sign in to leave a comment.
Comments
1 comment