Other: Support running the Proxy Agent in Kubernetes / Containers
AnsweredWe have deployed the Proxy Agent (to pair with AWS RDS connectors) into our Kubernetes environment as that is where we have compute - we don't want to be maintaining VMs. Our Kubernetes cluster also has access to all of the Databases we run.
To do this, we downloaded the executable and bundled it into an image with a Dockerfile, but this has two major issues:
1. The agent often throws ECONNRESET Java Exceptions, and after this it completely loses connectivity with Fivetran. The agent doesn't recover from this and sits there failing until we restart it.
2. The download link for the Proxy Agent is a time-bound authenticated link, so we can't do this automatically from a Dockerfile. It's a manual process to download and build.
In the same way the HVR agent is Kubernetes supported, it would be great if you would be able to publish a container image supporting the Proxy Agent.
-
Official comment
Hi Dan,
This feature—support for running the Proxy Agent in Kubernetes/Containers—is currently in under development. To gain access, please reach out to your account manager, who can assist with enrollment in the preview program.
We will share updates on this thread as we get closer to general availability.
Thanks,
Pieter -
Hi Dan,
Thank you so much for sharing this request with us. Supporting the Proxy Agent in Kubernetes or containerized environments is a valuable idea, and I appreciate you taking the time to let us know about your needs. I’ve added your request to our feature improvements backlog so it can be considered as we plan future enhancements.
To better understand your use case, could you share a bit more about your specific requirements? For example, what challenges are you facing today with the current Proxy Agent deployment, and how would containerization or Kubernetes support improve your workflows? Any additional details you can provide will help us prioritize and scope this feature more effectively.
What requirements would you have about the container embedded operating system? CVE turnaround time?
What Kubernetes distro or service do you have in mind?
We’ll be sure to keep this thread updated with any developments or progress. Thank you again for your thoughtful suggestion!
Best,
Pieter Humphrey -
Hi Dan,
The feature to support running the Proxy Agent in Kubernetes/containers is planned for development commencing in Q1 CY 2027. If you're interested in joining the preview program, please reach out to your account manager for access, cc myself -- pieter dot humphrey at fivetran dot com.
We'll continue to post updates on this thread as we move closer to a general release. What kubernetes service or distro and version are you using?Thanks,
Pieter -
Hi Pieter,
Thank you for keeping us up to date on this! There are more of us who find it valuable to be able to run the proxy agent as a container. Just for the ease of not having to manage VMs.
Can I ask if the container support you are developing also includes support to run multiple replicas of the same proxy? In a kubernetes environment it makes sense to spread the load over multiple pods.
I don't believe this is supported today, right? Running multiple agents instances to one proxy ID on Fivetran side.
Kind regards,
Thomas Linderdahl -
Hi Thomas Linderdahl,
(1) Can I ask if the container support you are developing also includes support to run multiple replicas of the same proxy? In a kubernetes environment it makes sense to spread the load over multiple pods.
No, not yet, we will look into cluster aware proxy later this year though. I want to work on HA for Proxy Agent, which will require the ability to run as a cluster.(2) I don't believe this is supported today, right? Running multiple agents instances to one proxy ID on Fivetran side.
Correct, this is not supported today. -
Hi Thomas Linderdahl,
Just want to confirm that while we are working on Dockerizing, Kubernetes work is being deferred for now until we have more demand for it.
Also, we'd need to know what K8s distro or service you plan on using? -
Hi Pieter,
Thanks for the status update!
From our side, we'd be interested in running the proxy agent on Google Kubernetes Engine and/or Google Cloud Run.
Kr,
Thomas Linderdahl
Please sign in to leave a comment.
Comments
7 comments