Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Other: Capability to define router file size for Aurora PostgreSQL as source

Ramniwas Chaurasia User

Currently HVR creates a router file per transaction. For a highly transactional databases, it results into creating thousands of files per second. This sometimes results into integrate latency due to sorting and reading large number of router files. So, if we can club multiple transactions into a single router file and be able to define the router file can avoid this issue.

Thanks,

Ramniwas

Please sign in to leave a comment.

Comments

1 comment

Date Votes
  • Mark Van de Wiel User

    Hi Ramniwas,

    Indeed every HVR transaction file contains one or more transactions. A transaction never spans multiple transaction files.

    By default HVR optimizes for both throughput and low latency. The capture process continuously tries to reach the end of the log. When it gets there, and if some changes were committed, then it waits for .5 of a second to check once more if changes can be bundled. Once that second phase has completed changes are sent to the hub for downstream processing. The delay between finding a committed change and checking again can be adjusted environment variable HVR_SCAN_DELAY (unit is in seconds). Of course, increasing this value from 0.5 will increase capture latency (and with that it may impact integrate latency). However, you will get fewer and larger files.

    Likewise when HVR runs behind and is not at the end of the logs it will by default propagate changes (in a single transaction file) in 10s increments (provided it has passed a transaction boundary; otherwise it will continue to parse logs to find a transaction boundary and then propagate changes). This duration can also be adjusted with an environment variable HVR_SCAN_SUBCYCLE_MAX (again in seconds). Increasing it from the default of 10 will result in fewer (and hence larger) files, but the integrate may have to wait longer to continue processing.

    I don't think we should set a threshold on file size because it would make latency very unpredictable and cannot work well given we maintain transactional consistency with the transaction files.

    Hope this helps.
    Mark.