Question

How can I configure the HVR services for high availability in a Cluster?

Environment

Local Data Processing

Answer

One of the advantages of a clustered environment is the ability to survive a node failure. Individual HVR processes always run on only one of the nodes at the time.

Note that this expert note applies to Unix and Linux environments only. On Windows use Windows Services in cluster mode to enable high availability for HVR services in the cluster.

HVR runs two daemons that start other HVR processes:

• hvrremotelistener
• hvrscheduler

An agent installation of HVR will only have the hvrremotelistener. A hub installation of HVR will have an hvrscheduler and (especially on Linux and Unix) in many cases also an hvrremotelistener defined.

In order for the hvrscheduler to be highly available in a cluster $HVR_CONFIG must be shared between the nodes, and the UID for the operating system user that runs HVR must be identical on every node. Point $HVR_CONFIG to a shared file system which can be NAS or a clustered file system. Note that the availability of the shared file system is crucial to the availability of HVR in the cluster. I.e. whilst an NFS share hosted by one of the nodes would work as a shared file system the availability of the shared file system then becomes dependent on the availability of this one node.

The HVR binaries can be installed on a shared file system or on the individual nodes. A single shared installation is easier to manage and takes up less storage space. However a shared storage system may perform slower than a storage system that is not shared, and this may or may not impact HVR’s processing depending on the transaction mix that must be captured. Currently, there is no rolling upgrade capability for HVR that entirely eliminates downtime.

Use file $HVR_HOME/lib/hvr_boot to make HVR highly available in a cluster environment. Copy the file $HVR_HOME/lib/hvr_boot and create an hvrtab file for every daemon that must be made highly available in the cluster. The copies of the files hvr_boot and hvrtab must be in the same location on every node in the cluster and may be stored on a shared file system for easier setup and maintenance. For example, if a cluster acts as a hub and runs the hvrremotelistener then there must be two copies of the file hvr_boot (e.g. file hvr_boot_scheduler and file hvr_boot_listener4343) and two copies of the file hvrtab (e.g. file hvrtab_scheduler and file hvrtab_listener4343). The respective hvrtab files must contain the correct line to invoke the service.

For example the contents of the file hvrtab_scheduler:

#hvrtab_scheduler to start the HVR scheduler in an Oracle RAC environment as OS user hvr
hvr hvrhub/!{Qb/Q.KqR}!@rac /mnt/shared/hvr/hvr_home /mnt/shared/hvr/hvr_config -EORACLE_HOME=<path to oracle_home>

For example the contents of the file hvrtab_listener4343:

#hvrtab_listener4343 to start the HVR listener on port 4343 in a clustered environment
root 4343 /mnt/shared/hvr/hvr_home /mnt/shared/hvr/hvr_config -EORACLE_HOME=<path to oracle_home>

Change the copies of the hvr_boot file to refer to the correct hvrtab files (at the beginning of the file).

For example in the file hvr_boot_scheduler:

HVRTAB=/mnt/shared/hvr/scripts/hvrtab_scheduler

For example in the file hvr_boot_listener4343:

HVRTAB=/mnt/shared/hvr/scripts/hvrtab_listener4343

The HVR scheduler must be registered as a cluster resource in the Clusterware software because it should only be run on one of the nodes at the time. Use Clusterware commands to do this logged in as the root user (on any one of the nodes).

The HVR listener must be registered as a local resource to run on every node in the cluster. Use Clusterware commands to do this logged in as the root user (on any one of the nodes).

Example using Oracle 11g Release 2 Clusterware

Below is an example using Oracle Clusterware commands.

Local Data Processing Scheduler Service

To create the resource in Oracle Clusterware, login as root with the environment set to use the Oracle Grid binaries. Then execute the following command:

[root@ol5-112-rac1 ~] crsctl add resource hvrschdlr 
-type cluster_resource 
-attr "ACTION_SCRIPT=/mnt/shared/hvr/scripts/hvr_boot_scheduler, CHECK_INTERVAL=30"

To start the resource, use:

[root@ol5-112-rac1 ~]# crsctl start resource hvrschdlr 
CRS-2672: Attempting to start 'hvrschdlr' on 'ol5-112-rac2' 
CRS-2676: Start of 'hvrschdlr' on 'ol5-112-rac2' succeeded

To relocate the resource to another node, use:

[root@ol5-112-rac1 ~]# crsctl relocate resource hvrschdlr 
CRS-2673: Attempting to stop 'hvrschdlr' on 'ol5-112-rac2' 
CRS-2677: Stop of 'hvrschdlr' on 'ol5-112-rac2' succeeded 
CRS-2672: Attempting to start 'hvrschdlr' on 'ol5-112-rac1' 
CRS-2676: Start of 'hvrschdlr' on 'ol5-112-rac1' succeeded

To check the status of the resource, use:

[root@ol5-112-rac1 ~]# crsctl status resource hvrschdlr 
NAME=hvrschdlr 
TYPE=cluster_resource 
TARGET=ONLINE 
STATE=ONLINE on ol5-112-rac1

To stop the resource, use:

[root@ol5-112-rac1 ~]# crsctl stop resource hvrschdlr 
CRS-2673: Attempting to stop 'hvrschdlr' on 'ol5-112-rac1' 
CRS-2677: Stop of 'hvrschdlr' on 'ol5-112-rac1' succeeded

To clean up the resource, use:

[root@ol5-112-rac1 ~]# crsctl delete resource hvrschdlr

HVR listener service

To create the resource in Oracle Clusterware, login as root with the environment set to use the Oracle Grid binaries. Then execute the following command:

[root@ol5-112-rac1 ~]# crsctl add resource hvrlsnr 
-type local_resource 
-attr "ACTION_SCRIPT=/mnt/shared/hvr/scripts/hvr_boot_listener4343, CHECK_INTERVAL=30"

To start the resource, use:

[root@ol5-112-rac1 lib]# crsctl start resource hvrlsnr
CRS-2672: Attempting to start 'hvrlsnr' on 'ol5-112-rac1'
CRS-2672: Attempting to start 'hvrlsnr' on 'ol5-112-rac2'
CRS-2676: Start of 'hvrlsnr' on 'ol5-112-rac1' succeeded
CRS-2676: Start of 'hvrlsnr' on 'ol5-112-rac2' succeeded

Use the commands stop resource and delete resource to stop and delete the resource respectively.