Expose connector-level and container-level memory metrics in Dashboard for Connector SDK connections
As part of the upcoming Connector SDK memory limit enforcement we implemented the memory measurement methods documented by Fivetran:
https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-memory-management?mkt_tok=MzUzLVVUQi00NDQAAAGiBL0vxDpyWUrBcpzTHxYUSMlGxbCpw5fUO-8nvYJTdfVGbGeSd0O_w7AoA1Sby9MX90zJ7ToVfa9pFk_IMFEti6IbppQMK_UlaMwqy5pG3sapUA
However, the in-code methods (psutil, tracemalloc, etc.) do not match the memory values used by Fivetran internal monitoring. Support confirmed that internal monitoring is container/pod-level and includes runtime overhead outside Python process memory.
This creates a visibility gap: We are expected to optimize for enforcement thresholds but cannot view or validate the same source-of-truth memory metric in the Dashboard.
Requested capability:
- Show the same memory metric used for warning/enforcement in the Dashboard for Connector SDK connections.
- Provide a breakdown of memory components where possible (for example: connector process memory vs runtime/container overhead).
- Provide time-series visibility for spikes with enough granularity to diagnose short-lived peaks.
- Provide API access to these memory metrics for automated monitoring and alerting.
Why this is critical:
Without access to the enforcement metric, optimization is trial-and-error and cannot be confidently validated before enforcement deadlines.
Reference [Fivetran Support] # 388002
Please sign in to leave a comment.
Comments
0 comments