Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Refresh Job Progress Visibility Enhancement

Answered
Meghana Ashok Laxani User
  • Issue observed from customer tickets (last 3 months):
    • Bulk refreshes in HVR are taking significantly longer than expected.
    • Limited visibility into progress, bottlenecks, and estimated completion times. 
  • Customer impact:
    • Difficulty planning maintenance windows and cutovers.
    • Operational uncertainty during long-running refreshes.
    • Delayed troubleshooting when performance degrades.
  • Recommended improvements:
    • Pre-refresh runtime estimation.
    • Identification of large tables and bottlenecks.
    • Timeprofiler option for Refresh
    • Real-time progress, throughput, and more logging in the refresh log
  • Refresh Bottleneck Analysis
    • Visibility into whether delays originate from source reads, network transfer, or target loading.
    • Actionable recommendations to improve performance.
  • Historical Refresh Analytics
    • Comparison with previous refresh executions. 
  • Expected Customer Benefits
    • Greater predictability and operational confidence.
    • Better planning for refresh activities.
    • Faster identification of performance issues.

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Edwin Fivetranner

    Hi Meghana,

    To summarise your request:

    1. estimator for refresh. Real-time refresh progress (rows processed, rows remaining, throughput, phase) visible in the UI/API without requiring channel re-activation
    2. Database-native progress signals: surfacing row counts from target load statistics (COPY/INSERT progress, commit checkpoints) for databases like PostgreSQL, Oracle, and DB2
    3. refresh analytics vs previous
    4. Stuck vs. progressing detection: automatic differentiation between an actively loading refresh and one that has stalled (no row movement for N minutes). On this specific point, we do have additional alerting enhancements coming out in the next 6.3.5 release. Please validate how those work for you.

    For point 2, we have a similar request, and we are looking into how we can improve the product. For point 4, check whether the latest alerting capabilities make a difference.

    Still, I want to be transparent. These are significant, unplanned engineering efforts that touch multiple layers of the product. As a result, I do not expect we will be able to implement improvements quickly.

    We will come back to you once we have more clarity on prioritization and possible timelines.

    Best regards,

    Edwin

  • Edwin Fivetranner

    Hi Meghana, 

    HVR already provides partial coverage through Time Profiler, which has supported Refresh jobs since version 6.1.5/1.

    When enabled, Time Profiler records how much time the Refresh job spends on activities such as SQL execution, network I/O, disk I/O, and internal HVR processing. It distinguishes between Hub and Agent activity and can provide increasingly detailed logging through the LOW, MEDIUM, and HIGH trace levels. This can help determine whether a slow Refresh is primarily constrained by source reads, network transfer, target SQL, storage, or HVR processing.

    The collected breakdown is also available through HVR Statistics, providing some ability to review performance over time.

    However, Time Profiler is primarily a diagnostic capability, not a Refresh progress or forecasting capability. It does not currently provide:

    • A pre-refresh runtime estimate
    • An overall completion percentage or reliable ETA
    • A consolidated view of completed and remaining tables or slices
    • Automatic identification of the tables that will dominate runtime
    • Comparison of individual Refresh executions
    • Actionable tuning recommendations

    Therefore, Time Profiler addresses an important part of the requested bottleneck analysis, but it does not fully address the customer’s need for predictability, real-time progress, and completion forecasting. The remaining feature request should focus specifically on these gaps.

    Best regards,

    Edwin