Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Vendor Agnostic Data Sync HTTP Endpoint

Completed
Danny Myers User

Instead of requiring that we host Azure Functions, GCP, or AWS Lambda connectors to do custom data sync. It would be nice if we could provide our own HTTP endpoint instead. We have a fair amount of K8s infrastructure in place to build HTTP microservices, so it would be nice to leverage that instead of having to create vendor-specific sync endpoints/deployments.

BTW kudos on the general design of this functionality. Very easy to understand, build, and use.

Please sign in to leave a comment.

Comments

6 comments

Date Votes
  • Official comment
    Alison Fivetranner

    Hi Danny,

    I'm very happy to announce our connector SDK service which should allow you to create custom connectors for your HTTP microservices without having to depend on a third party function provider service.

    I'd love to hear any feedback you have after checking it out.

    Best regards

    Alison

  • Tom Strader User

    Agree!  We ended up creating a generic Azure function which simply works as a proxy to an HTTP route, but sure would be nice if that was just out-of-box functionality using the same request/response pattern as the Azure function.

    Here is the function code: https://github.com/tstrader/fivetran-generic-azure-connector

  • Alexander User

    Thanks Danny for the idea and thanks Tom for the helpful comment! That's a handy workaround.

    We are looking to expand functionality of our custom functions in the back half of this year. This is a good idea and I'll make sure we consider it as part of that effort.

  • Jonathon Lee User
    • Edited

    Was there ever any movement on this? How has it performed for you?

    Did you ever experiment with just removing the proxy cloud function and having the real endpoint exposed and just telling fivetran it was a cloud function.  I'm unclear as to how it would know the difference between one http endpoint vs another. (we are just getting started with this) 

  • Danny Myers User
    • Edited

    Hi Jonathan. We would love to be able to do that, but if you look at how the azure function connector is set up, it doesn't let you set a raw url (see below). Is there a generic HTTP connector?

     

  • Jonathon Lee User

    Interesting.  GCP cloud functions are just http triggers that are authenticated with a service account.  None of that is actually specific to the fact that it is deployed as function vs exposed via some other API.