Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Dashboard Improvement: Real‑time Refresh Progress Visibility Without Channel Re‑activation

Answered
Naresh Kumar Grandi User

Currently, during table refresh operations in HVR, operational visibility is very limited.

The Events section only shows high‑level states such as:

  • BUSY
  • DONE
  • RETRY

For tables in BUSY state:

  • We cannot see how many rows have been processed
  • We cannot estimate remaining duration
  • We cannot distinguish “stuck” vs “actively progressing” refreshes

To obtain deeper insight (row counts, SQL activity, progress), we must:

  • Enable HVR_SQL_TRACE
  • Which requires channel re‑activation
  • Which forces the refresh to restart from scratch

This behavior is not operationally acceptable in Production environments.

Business & Operational Impact

1. Production Risk

Re‑activating a channel solely for observability:

  • Abandons already‑processed data
  • Restarts large refreshes (hours / TB‑scale tables)
  • Increases load on source and target systems
  • Risks SLA violations

2. Lack of Predictability

Without progress metrics:

  • Application teams cannot estimate completion time
  • On‑call teams cannot plan support windows
  • Stakeholders lack confidence during long refreshes

3. Inefficient Troubleshooting

When a refresh appears “BUSY”:

  • We cannot determine if it is:
    • Actively loading rows
    • Blocked by locks
    • Stalled due to slow SQL
  • This leads to unnecessary escalations and retries

  • Why Re‑activation Is Not a Viable Solution

    Requiring channel re‑activation to enable HVR_SQL_TRACE is problematic because:

    • Re‑activation:
      • Resets refresh state
      • Discards in‑progress work
      • May re‑create replication objects
    • In PROD, re‑activation is:
      • Change‑controlled
      • Time‑consuming
      • Risky for downstream consumers

    Observability must not require destructive actions.

     

    Proposed Enhancement (Feature Request)

    ✅ Enable Non‑Disruptive Refresh Progress Visibility

    1. Real‑time Table Refresh Metrics (UI & API)

    Expose the following without re‑activation:

    • Total rows to be refreshed (if determinable)
    • Rows processed so far
    • Rows remaining
    • Current refresh phase (truncate / load / merge / apply)
    • Start time & elapsed time
    • Average throughput (rows/sec)

    2. Dynamic Runtime Tracing (No Restart)

    Allow:

    • Enabling/disabling HVR_SQL_TRACE at runtime
    • Scope limited to:
      • Specific channel
      • Specific table
      • Specific refresh job
    • No need for:
      • Channel re‑activation
      • Refresh restart

    3. Backend Support (Database‑aware)

    For targets like PostgreSQL / Oracle / DB2:

    • Surface row counts from:
      • Target load statistics
      • COPY / INSERT progress
      • Transaction commit checkpoints
    • Even approximate metrics are acceptable and valuable

    4. Stuck vs Progressing Detection

    Add logic to differentiate:

    • Actively progressing refresh
    • No row movement for N minutes (potential stall)

Please sign in to leave a comment.

Comments

1 comment

Date Votes
  • Edwin Fivetranner

    Hi Naresh,

    Thank you for the detailed write-up. The detail in the request makes the operational impact clear.

    To summarise what you're asking for:

    1. Real-time refresh progress (rows processed, rows remaining, throughput, phase) visible in the UI/API without requiring channel re-activation
    2. Dynamic SQL tracing: ability to enable/disable HVR_SQL_TRACE at runtime, scoped to a specific channel, table, or job, without restarting the refresh
    3. Database-native progress signals: surfacing row counts from target load statistics (COPY/INSERT progress, commit checkpoints) for databases like PostgreSQL, Oracle, and DB2
    4. Stuck vs. progressing detection: automatic differentiation between an actively loading refresh and one that has stalled (no row movement for N minutes). On this specific point, we do have additional alerting enhancements coming out in the next 6.3.5 release. Please validate how those work for you.

    We have logged these first three as feature requests and will share them with our product team. Still, I want to be transparent. These are significant engineering efforts, not planned, that touch multiple layers of the product. As a result, I do not expect we will be able to implement improvements quickly.

    We will come back to you once we have more clarity on prioritisation and possible timelines.

    Best regards,

    Edwin