Question
How should I use the hasMore
object in a cloud function?
Environment
Cloud function connectors.
Answer
- Ensure your function is maintaining an internal variable that determines whether or not the data returned from your source API is paginated
- Ensure your function utilizes an
if/else
statement to determine whether or not the ‘hasMore’ flag should be returned as ‘true’ or ‘false’
Example pseudo-code for this scenario:
// Request
var paginationCounter = 0
if (req.body.hasMore = false) {
hostname = endpoint.com
path = /giveMeEverything
}
else {
hostname = endpoint.com
path = /paginationCounter
}
if (res.moreData = true) {
function add() {
paginationCounter += 1;
}
else {
paginationCounter = 0
}
// Response
if (paginationCounter > 0) {
hasMore:true
}
else {
hasMore:false
}
Context
The hasMore
flag is the mechanism that Fivetran provides to allow a function to paginate through data that is returned in a paginated format.
For example (where we assume the source API provides paginated data):
- On the initial run of a function, the
hasMore
flag will be passed as ‘false’: - {“hasMore”:false }
- We will expect the response from the request to the source API to be returned with an indicator as such; examples will include:
- {“nextCursor”:”yes”}
- {“page”:”1/10”}
- {“moreData”:true}
- During that initial run of the function, should a response come back from the source API that indicates there is more data to be pulled then you would pass the following back to Fivetran:
- {“hasMore”:true}
- Fivetran would not proceed to the next sync; Fivetran will write the returned data and asynchronously - within the context of the original sync - pass this same value back to the function:
- {“hasMore”:true}
- This will then indicate to the function to paginate the request and continuously work through the paginated data provided in the response
- Only when there is no more data in the paginated response will the function pass the following back to Fivetran:
- {“hasMore”:false }
- Only when the
hasMore
flag is ‘false’ will the function move to the next sync and a new invocation of the function
Considerations
- If the source API that you’re interacting with does not provide paginated data, then you can simply pass this back in the same manner to Fivetran:
- {“hasMore”:false }