Connector Improvement: Add a parameter in api call to randomize base sync minute per connection
Answered
we have 208 connections to a snowflake destination
connections fails daily due to herding effect, i.e. all connections are trying to sync simultaneously instead of being spread out during an hour.
There is no feature to set sync minute at random.
Even on daily sync the only possible options are at 00 and 30 minutes.
At the moment all connections on the same destination have the same base minute.
There is a need to modify sync base minute per connector to avoid connection herding
-
Official comment
Thank you for submitting your suggestion—we really appreciate you taking the time to share your thoughts! To help us better understand your request, could you provide more details about the problem you’re trying to solve with randomization? For example, how are you currently creating and scheduling connectors? Any additional context will help us evaluate and prioritize this feature more effectively.
Looking forward to your response.
Best regards,
Please sign in to leave a comment.
Comments
1 comment