Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Other: LDP: Better reporting on Compare results

Answered
Cody Powell User

We are implementing HVR 6.1 from HVR 5.7.0 from Oracle to BigQuery.  The native connector brought us the ability to do compare (side quest listed here).  Additionally, this allows us to start reporting SOX compliant data out of our BQ data warehouse.  We have found, however, that providing the proof that the data is in compliance to the Data Engineering team (those responsible for Data in our org) along with the auditors. 

Navigating the UI and reviewing the diff files one at a time is tedious and time consuming.  We'd love to see a mechanism by which the diffs could be stored in the metadata repo OR another database/schema all together.  This would provide reporting capabilities and allow both the Data Engineers and Database Engineers to effectively and specifically know when we have an issue and the scope.

For now, we are working to do this via a manual (quasi-automated process) that involves scripting and external tables within a database.  Overall, reporting out of the application in a meaningful way is a HUGE deficit and something we'd appreciate more time and attention focused on.  That said, this "simple" enhancement would bring a TON of value to our org.   

Please sign in to leave a comment.

Comments

3 comments

Date Votes
  • Mark Van de Wiel User

    Hi Cody,

    You specifically ask about the diff results. Is it acceptable to have diffs?

    I would have thought that you want to report on the actual compare results. This could be similar to the view in the events dialog but maybe you want a better consolidated view?

    Note there is a REST call to view the diff files:

    POST /api/v6.1.0.7/hubs/{hub}/channels/{channel}/locs/{loc}/tables/{table}/diffs/{ev_id}/get

    Mark.

  • Cody Powell User

    Hey Mark!

    I apologize it has taken me this long to respond; the email must have gotten filtered and missed.  

    We are leveraging the API to generate the data and still fumbling (as time allows) to feed that data into a table.  

    The diff files themselves are cumbersome for large tables that may have a significant amount of data out-of-sync (due to outage, or in our experience, upgrade).  They are also "fine" if there are no diffs detected by the compare process; however, at scale, in our current state, is not realistic.  

    The data engineers would love to see those results stored in a local database that they can write reports, see trends and provide a mechanism for the database engineers to troubleshoot if we see similar data week after week. 

    I know this is low-priority and the quality of life improvement this would bring is huge - similar to what we have today with Veridata.  In our current Oracle -> Oracle replication with GoldenGate, SOX requires us to provide system generated reports of data sync issues and how they were resolved.  The same will be true when we migrate all reporting off of our Operational data store and into BQ.

    I hope that provides more context; I'm no longer a part of the database engineering team at SEI; so my input here may be less "day-to-day" but my new role as a Site Reliability Engineer focuses on visibility.  I'm happy to have further discussions if you have time! 

  • Derek Bell User

    Similar to Mark's comment, I would like to see a report/export feature for compare results.