Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Other: HVR6: Guardrails

Harish Palaniswamy User

Recently, we setup a test channel within the same production hub to test and optimize our Refresh speeds into Redshift. Unfortunately, we forgot to modify one of the parameters, an Environment Action setting the search path to the same as what we had in production. 

This caused a data quality issue with the intermediate burst tables that HVR uses in its integrate jobs which we were able to solve with a capture rewind. It could be worthwhile to add some guardrails or sanity checks that prevent this from happening in the first place.

In this context, it should've warned us that there were two different channels writing to the same burst table and we would have caught the error sooner. 

Please sign in to leave a comment.

Comments

1 comment

Date Votes
  • Mark Van de Wiel User

    Hi Harish,

    I am sorry to hear that a mistake in the HVR configuration resulted in production data consistency issue. Your request for guardrails is reasonable. However, you must realize that HVR is a very powerful technology and any number of guardrails is not going to fully prevent errors.

    For example, assuming you implemented your channels in a single hub there is an argument that we should have known about this issue. However, what if the channels were in different hubs? Or in separate hubs on different hub servers?

    Why is your development environment mixed with the production workload? I imagine there is a cost consideration?

    Please note that we just released 6.1.5/10 early adopter release with explicit support for setting the burst table schema in the DbObjectGeneration action. Of course with Redshift you can set search_path to make the change but this option requires an environment variable, and is not available on all destinations. I hope that such feature will make unfortunate errors less likely.

    Mark.