Expose connector-level and container-level memory metrics in Dashboard for Connector SDK connections
As part of the upcoming Connector SDK memory limit enforcement we implemented the memory measurement methods documented by Fivetran:
https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-memory-management?mkt_tok=MzUzLVVUQi00NDQAAAGiBL0vxDpyWUrBcpzTHxYUSMlGxbCpw5fUO-8nvYJTdfVGbGeSd0O_w7AoA1Sby9MX90zJ7ToVfa9pFk_IMFEti6IbppQMK_UlaMwqy5pG3sapUA
However, the in-code methods (psutil, tracemalloc, etc.) do not match the memory values used by Fivetran internal monitoring. Support confirmed that internal monitoring is container/pod-level and includes runtime overhead outside Python process memory.
This creates a visibility gap: We are expected to optimize for enforcement thresholds but cannot view or validate the same source-of-truth memory metric in the Dashboard.
Requested capability:
- Show the same memory metric used for warning/enforcement in the Dashboard for Connector SDK connections.
- Provide a breakdown of memory components where possible (for example: connector process memory vs runtime/container overhead).
- Provide time-series visibility for spikes with enough granularity to diagnose short-lived peaks.
- Provide API access to these memory metrics for automated monitoring and alerting.
Why this is critical:
Without access to the enforcement metric, optimization is trial-and-error and cannot be confidently validated before enforcement deadlines.
Reference [Fivetran Support] # 388002
-
Official comment
Hi Tanmaya,
We are indeed working on showing memory usage in production. And rest assured, we will not start enforcing the memory limit until this feature is available and ample time is given to customers to adjust their code accordingly
Best,
Emrah
Please sign in to leave a comment.
Comments
1 comment