Connector Improvement: Exact Online connector takes way too long to sync and this is easy to fix
AnsweredDear Fivetran,
Currently the "Exact Online"-connector from Fivetran does every time a full data pull when it starts to sync. We (and I think many others) need for our operation a maximum delay in the datasync of 10 minutes but the way Fivetran handles this takes about around 10 hours and an incrediable amount of calls to Exact Online whole day long as takes every time "all" data (+/-10 GB in our case) from Exact and starts comparing it with the last sync. Very inefficient.
Exact Online is not happy with this amount of calls and especially as it can be easily avoided and we can not work with this long delay in data so this is a showstopper for us to use this current connector because of the extreme long delay.
This can be done much more efficient and in a lean way and Exact Online is also advising to do so.
Exact Online offers possibilitys for a very fast almost online sync as it supports:
1. Sync Endpoints (query data with a continuation token or timestamp to retrieve only the “latest changes” since your last sync)
2. Webhooks (subscribe to events that notify you of changes in near real-time, then fetch the changed records with the standard or sync REST endpoints)
Most of the REST API have a page size of 60. The bulk and sync endpoints have a pagesize of 1000. It is recommended by Exact to use the sync endpoints where possible. See this page and filter on the service "Bulk" and "Sync": https://start.exactonline.nl/docs/HlpRestAPIResources.aspx?SourceAction=10 .
It is possible to retrieve incremental changes (sometimes referred to as “delta” changes) from Exact Online by using their Sync Endpoints or by leveraging webhooks combined with the standard REST APIs.
Below is an overview of how you can set this up:
1. Use the Sync Endpoints
Exact Online offers dedicated sync endpoints that help you fetch data in an incremental (“delta”) manner. You can find these endpoints in the Bulk or Sync sections of the Exact Online API, typically under URLs like:
GET /api/v1/{division}/sync/<Module>/<Entity>
where:
- division is the Exact Online administration code / division number.
-
Module and Entity depend on what data you are trying to synchronize (e.g.
CRM/Accounts,Financial/GLAccounts, etc.).
How it works
-
Initial Full Load:
- You call the sync endpoint without providing a
timestamp(or “change pointer”) parameter. Exact Online returns a (potentially large) dataset. - Along with the dataset, Exact Online also returns a field named
@odata.nextLink,ContinuationToken, or atimestampfield. You store these so you know how far you’ve synchronized.
- You call the sync endpoint without providing a
-
Subsequent (Incremental) Calls:
- On the next call, you include the relevant continuation token or timestamp provided by Exact Online in the previous response.
- Exact Online will return only the data changed since your last call.
3. Paging:
- If the dataset is large, the response might be paginated. You follow the
@odata.nextLinkuntil no further pages are returned. - Each page also includes an updated
timestamporContinuationToken. Keep track of these carefully.
By storing this “last synchronization token” on your side, you can periodically query the Exact Online sync endpoints and only fetch records created or modified since the last token.
2. Use Webhooks (Optional / Advanced)
Exact Online provides webhooks to notify your application when certain events occur (e.g., when records are created, updated, or deleted). This can complement the incremental approach:
-
Webhook Subscription:
- You register a webhook in Exact Online specifying the event types or entities you’re interested in.
-
Receive Notifications:
- When a record changes in Exact Online, your webhook endpoint is called with the information on what changed.
-
Query the REST API:
- In response to the notification, you call the standard (or Sync) REST endpoints to retrieve the updated data.
- You could also maintain the “last known state” in your system and then merge the changes in near real-time.
Webhooks are especially useful when need near real-time updates instead of running a scheduled sync.
3. Additional Points
-
Handle Paging
- Even for incremental calls, responses can be paginated. Always check for an
@odata.nextLinkor equivalent field that tells you whether more data is available.
- Even for incremental calls, responses can be paginated. Always check for an
-
Store Checkpoints / Timestamps
- Make sure to persist the timestamp or continuation token from each successful call. If your synchronization job crashes or restarts, you’ll need that token so you do not lose track of your sync position.
-
Filtering by Modified Date (Alternative Approach)
- Some “bulk” endpoints support a
ModifiedSincefilter, letting you query records updated after a specified date-time. Check the Exact Online API documentation for which endpoints supportModifiedSince.
- Some “bulk” endpoints support a
Example Flow (Sync Endpoint)
Here is a high-level pseudocode outline for a synchronization job using the Accounts sync endpoint:
function syncExactOnlineAccounts(lastSyncToken):
url = "/api/v1/{division}/sync/CRM/Accounts"
if lastSyncToken exists:
url = url + "?$skiptoken=" + lastSyncToken // or use continuationToken
response = callExactOnlineAPI(url)
for account in response.data:
// Upsert data into your local database
// If there's paging, keep calling next pages
while response.@odata.nextLink exists:
response = callExactOnlineAPI(response.@odata.nextLink)
for account in response.data:
// Upsert data again into your local database
// Save the new sync token (e.g. next link's skiptoken or returned timestamp)
saveLastSyncToken(response.syncToken)We had contact with Mark Gaughran from Fivetran on this subject and he supporting this point but to get it activated he asked it to this portal so he has a connectionpoint to the supportteam. Please contact him on advise on this subject. We had contact with Mark Gaugran via https://support.fivetran.com/hc/en-us/requests/240669 so he knows how to contact us aswell.
As mentioned to Mark Gaughran we are open to invest money to get this connector guaranteed synchronizing all data with a maximum delay of 10 minutes so let us know if you want us to finance Fivetran for this.
For you to know: We have currently inhouse a same setup between Exact Online and a datalake with a maximum data delay of 10 minutes so we know it is possible but we want to outsource this and many of our other interface to a full service 3rd party like Fivetran but again we need to have a max delay of 10 minutes.
Hope you can help or let us know if we can help as we have also IT-people inhouse that maybe can asist.
Do you think you can get this implemented within a few weeks?
Regards.
-
Official comment
Hi Mat Jolic,
I'm pleased to inform you that the improved Exact Online connector is live and incorporates efficient data synchronization. If you have any questions or need assistance setting up, please feel free to reach out to our support team.
Best,
Sanjeevv Maniram T
Please sign in to leave a comment.
Comments
1 comment