Skip to main content

Community

Connector Improvement: Cloud functions hasMore behavior should not have to preserve state in the function

Answered

Please sign in to leave a comment.

Comments

4 comments

  • Official comment
    Craig User

    Hi! Please see our documentation regarding the `state` atttribute:

    https://fivetran.com/docs/functions#requestformat

    Your function is required to update `state` with whatever the next state should be on the subsequent function call. You are calling this your "bookmark," you can set `state` to whatever value(s) you need to be able to pick up where you left off on the previous run of the function. So for example, as part of your response,

    {
        "state": {
            "bookmark": "bookmark_value"
        },
        "insert": { /* insertions here */ },
        "delete": { /* deletions here */ },
        "schema" : { /* schema definition here */ },
        "hasMore" : true
    }

    Keep in mind, if there is an error on the Fivetran side, we may need to call your function with the previously known good state. Your function should be able to pick up from whatever state is passed to it. However, this should be a very infrequent occurrence. 

    It would be awesome if yall could clarify this behavior in the documentation. It's not obvious (at least to me) that the `state` object will persist between calls when using `hasMore`.

    I agree with Kaoru. The documentation is not very clear.

    Hi Ambex - we added this disclaimer to the documentation:

    hasMore is a boolean indicator for Fivetran to make a follow-up call to fetch the next set of data. Fivetran will keep making calls while hasMore = true until it receives hasMore = false. In either case, your response should return an updated state to checkpoint the data fetched in that request and avoid an infinite loop.

    Is it still unclear?