Question

How should I use the hasMore object in a cloud function?

Environment

Cloud function connectors.

Answer

• Ensure your function is maintaining an internal variable that determines whether or not the data returned from your source API is paginated
• Ensure your function utilizes an if/else statement to determine whether or not the ‘hasMore’ flag should be returned as ‘true’ or ‘false’

Example pseudo-code for this scenario:

 // Request
 
 var paginationCounter = 0 

 if (req.body.hasMore = false) { 

 hostname = endpoint.com 

 path = /giveMeEverything 

         } 

 else { 

 hostname = endpoint.com 

 path = /paginationCounter 

         } 

 if (res.moreData = true) { 

 function add() { 

 paginationCounter += 1; 

 } 

 else { 

 paginationCounter = 0 

 } 

 // Response 

 if (paginationCounter > 0) { 

 hasMore:true  

 } 

 else { 

 hasMore:false 

 } 

Context

The hasMore flag is the mechanism that Fivetran provides to allow a function to paginate through data that is returned in a paginated format.

For example (where we assume the source API provides paginated data):

• On the initial run of a function, the hasMore flag will be passed as ‘false’:
• {“hasMore”:false }
• We will expect the response from the request to the source API to be returned with an indicator as such; examples will include:
• {“nextCursor”:”yes”}
• {“page”:”1/10”}
• {“moreData”:true}
• During that initial run of the function, should a response come back from the source API that indicates there is more data to be pulled then you would pass the following back to Fivetran:
• {“hasMore”:true}
• Fivetran would not proceed to the next sync; Fivetran will write the returned data and asynchronously - within the context of the original sync - pass this same value back to the function:
• {“hasMore”:true}
• This will then indicate to the function to paginate the request and continuously work through the paginated data provided in the response
• Only when there is no more data in the paginated response will the function pass the following back to Fivetran:
• {“hasMore”:false }
• Only when the hasMore flag is ‘false’ will the function move to the next sync and a new invocation of the function

Considerations

• If the source API that you’re interacting with does not provide paginated data, then you can simply pass this back in the same manner to Fivetran:
• {“hasMore”:false }