Scenario/Use Case:
You have read documentation on our cloud functions but don’t quite understand how best to utilize the ‘hasMore’ object.
Fivetran Recommendation:
- Ensure your function is maintaining an internal variable that determines whether or not the data returned from your source API is paginated
- Ensure your function utilizes an ‘if/else’ statement to determine whether or not the ‘hasMore’ flag should be returned as ‘true’ or ‘false’
Example pseudo-code for this scenario:
// Request
var paginationCounter = 0
if (req.body.hasMore = false) {
hostname = endpoint.com
path = /giveMeEverything
}
else {
hostname = endpoint.com
path = /paginationCounter
}
if (res.moreData = true) {
function add() {
paginationCounter += 1;
}
else {
paginationCounter = 0
}
// Response
if (paginationCounter > 0) {
hasMore:true
}
else {
hasMore:false
}
Context:
The ‘hasMore’ flag is the mechanism that Fivetran provides to allow a function to paginate through data that is returned in a paginated format.
For example (where we assume the source API provides paginated data):
- On the initial run of a function, the ‘hasMore’ flag will be passed as ‘false’:
- {“hasMore”:false }
- We will expect the response from the request to the source API to be returned with an indicator as such; examples will include:
- {“nextCursor”:”yes”}
- {“page”:”1/10”}
- {“moreData”:true}
- During that initial run of the function, should a response come back from the source API that indicates there is more data to be pulled then you would pass the following back to Fivetran:
- {“hasMore”:true}
- Fivetran would not proceed to the next sync; Fivetran will write the returned data and asynchronously - within the context of the original sync - pass this same value back to the function:
- {“hasMore”:true}
- This will then indicate to the function to paginate the request and continuously work through the paginated data provided in the response
- Only when there is no more data in the paginated response will the function pass the following back to Fivetran:
- {“hasMore”:false }
- Only when the ‘hasMore’ flag is ‘false’ will the function move to the next sync and a new invocation of the function
Considerations:
- If the source API that you’re interacting with does not provide paginated data, then you can simply pass this back in the same manner to Fivetran:
- {“hasMore”:false }