How can I replicate XML files from an Oracle Linux server to AWS S3 bucket using HVR?
A client has the need to replicate their audit logs (XML files) that reside on Oracle Linux server to AWS Redshit/S3 bucket. One of the options I am thinking to use HVR. Does HVR support moving files to (Amazon) AWS S3 bucket?
HVR is indeed capable of moving files between different locations, including S3.
The configuration is really straightforward. All you need is a simple channel setup with a "source" file type location (this would be your Linux server) and an S3 target location, with Capture and Integrate actions on them, respectively.
- An example of a setting up source remote file location on server orahost with an HVR agent.
If you don't have an agent installed on your source machine, you can use SFTP to access your files
2. After setting up the channel, HVR will simply copy any files it finds in the directory of the file location to the target location. You can control which files you want HVR to handle with /Pattern and /IgnorePattern options of the Capture action.
1) We are estimating 10K to 14K audit files per day (all together ~15 to 20GB). Can HVR handle this many files? In other words, do we need any additional resources to be added to the HVR hub side?
It should not be an issue if you have enough bandwidth between the source and your hub, and the hub and AWS. If you are expecting the number of files to be replicated being on the higher side, having a source location with an HVR agent installed is a recommended best practice. It provides compression and direct read access with little overhead to the files which is beneficial to minimize latency on the capture side.
2) We already have an agent (HVR remote listener agent) installed on source database servers as we already have oracle tables replication setup from both these database servers. Will the same agent work for XML files replication from these database servers to send over to the S3 target location in AWS?
Yes, you can use the same agent
The scope of this KB is only to describe how to move the files to an existing AWS Redshift/S3 instance.
- See AWS documentation for configuring Redshifthttps://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html
(Security groups, keys and SSL certificates need to be in place for seamless communication.)
- HVR Requirements for Redshift: https://www.hvr-software.com/docs/location-class-requirements/requirements-for-redshift
- HVR Quickstart for Redshift: https://www.hvr-software.com/docs/quick-start-guides/quick-start-for-hvr-redshift