Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Cloud functions hasMore behavior should not have to preserve state in the function

Answered
Michael Taluc User

When a cloud function (I'm working in Azure today) returns hasMore=true, the bookmark is not updated by Fivetran, so the same state is sent back to the function on the subsequent call.  This implies that the function has to somehow keep track of the state between calls.  It would be better if the function could just set hasMore=true, return the proper bookmark as the state, then the connector would send a new request with the updated state.

Please sign in to leave a comment.

Comments

4 comments

Date Votes
  • Official comment
    Craig User

    Hi! Please see our documentation regarding the `state` atttribute:

    https://fivetran.com/docs/functions#requestformat

    Your function is required to update `state` with whatever the next state should be on the subsequent function call. You are calling this your "bookmark," you can set `state` to whatever value(s) you need to be able to pick up where you left off on the previous run of the function. So for example, as part of your response,

    {
    "state": {
        "bookmark": "bookmark_value"
    },
    "insert": { /* insertions here */ },
    "delete": { /* deletions here */ },
    "schema" : { /* schema definition here */ },
    "hasMore" : true
}

    Keep in mind, if there is an error on the Fivetran side, we may need to call your function with the previously known good state. Your function should be able to pick up from whatever state is passed to it. However, this should be a very infrequent occurrence. 

  • Kaoru Kohashigawa User

    It would be awesome if yall could clarify this behavior in the documentation. It's not obvious (at least to me) that the `state` object will persist between calls when using `hasMore`.

  • Ambex Bantu User

    I agree with Kaoru. The documentation is not very clear.

  • Fraser User

    Hi Ambex - we added this disclaimer to the documentation:

    hasMore is a boolean indicator for Fivetran to make a follow-up call to fetch the next set of data. Fivetran will keep making calls while hasMore = true until it receives hasMore = false. In either case, your response should return an updated state to checkpoint the data fetched in that request and avoid an infinite loop.

    Is it still unclear?