Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Allow Cloud Function Connectors to return a "sync again in x minutes" property in the response

Answered
Marc Shiker User

Allow Cloud Function connectors to return a value to Fivetran in the response to indicate when the connector should execute a sync instead of waiting for the next scheduled sync.

For example, in a new AWS Lambda Cloud Function Connector I created, the source API has an API limit of 1,000 requests per hour. For the initial sync, there are over 2000 requests I need to make. After my function reaches the 1,000 limit, I return hasMore = false because my function won't be able to request more data until an hour later. The connector is configured to sync every six hours. Because of this, the next part of my initial sync to get the next 1,000 data items won't happen until 6 hours later. However, if I could return, for example, a value of 60 for a new "SyncInMinutes" property, then Fivetran could schedule my next sync in one hour instead of having to wait six hours.

 

Please sign in to leave a comment.

Comments

3 comments

Date Votes
  • Official comment
    Alison Fivetranner
    • Edited

    Hi Marc,

    Thank you for your response :-)

    I'm wondering if you have tried our new Connector SDK Custom connector offering. You have a lot more flexibility with how you manage the connector code, and therefore rate limits on your source side. You still have full control of the state variable. I'm thinking with the connector SDK, maybe more frequent sync frequency and possibly the Fivetran API you may get the flexibility you are looking for. For Connector SDK we manage the infrastructure allowing us to optimize for data connectors - if you are sending data we'll keep the connector running so pausing for a few seconds to allow a rate limit to reset is completely fine.

    Best regards

    Alison

  • Alison Fivetranner
    • Edited

    Dear Marc,

    Thank you for sharing a great feature enhancement idea. I'm wondering though if rather than a 'sync in x minuets' time it could be easier to use if we think about this as more of a request to handle rate limits better.

    Would a feature to enable you to report information about the rate limit you are encountering and then correctly set hasMore = TRUE be better? - then we can wait for the right time based on the rate limit configuration.

    Best regards
    Alison

  • Marc Shiker User

    Hi Alison.

    I had a call with Mead Brodeur today and I asked about any movement on this feature request and he pointed out to me that I hadn't responded to your question (woops!). Here is my very late response.

    I think the "sync in x minutes" is a better way to handle this because it allows flexibility on my side with the same result on the Fivetran side. Some thoughts: 1) Fivetran wouldn't know if it needs to or if it should sync again because it doesn't know how many remaining calls I have left in my rate limit window 2) Sometimes rate limits are for particular endpoints and not the API as a whole 3) I might want to request a sync in 5 minutes, based on some condition that is identified during the execution of the AWS Lambda Function code that is unrelated to a rate limit.

    I would be happy to discuss further and work through uses cases for your suggestion if this feature is something that could get picked up. I'm working on a new custom integration now that I would really like this (which is why I brought it up with Mead now).

    Thanks,

    Marc