Connector Improvement: Incremental sync improvements in Freshdesk connector
AnsweredHi, Freshdesk connector currently considers Conversation and associated email/attachment tables as re-import tables and creates unnecessary churn in the downstream consumers.
For incremental syncs, Fivetran connector uses Freshdesk Tickets API endpoint for a list of tickets that have shown any activity since the last sync. Connector then uses the Freshdesk Conversations API endpoint for each ticket and retrieves all conversations related to the ticket. This creates churn as the response could include several conversations that are not updated since the last sync and Fivetran connector updates all the conversations for the ticket that have shown activity since the last sync. Depending on number of tickets that have shown any activity since the last sync, connector makes several API calls (750+) during a sync and an incremental sync takes approximately 30 to 50 minutes (due to the number of API requests and limits).
This creates unnecessary churn in the downstream consumers (95% of the records with recent _fivetran_synced timestamp don't have any changes). Freshdesk maintains created_at and updated_at timestamps for each record. Most of the conversations were last updated several days or months ago but updated in the recent sync as the connector updates all conversations returned by the endpoint. After enumerating the conversations for each ticket, connector can check updated_at timestamp and ignore downstream database update if the update is not within the sync time window (i.e., last sync time to now). You can find additional details and examples in the support request (#120657).
Thanks for your help and support!
-
Official comment
Hi Varma,
Thanks for the request. I understand the idea here, but I think it still isn't quite possible. I don't believe the Freshdesk API allows extracting a limited selection of conversations based on created and updated timestamps. We need to extract them all even though they do have those fields in the response.
Perhaps some kind of filtering within Fivetran on the extracted data is possible so that we write fewer records to the destination. We will look into this as part of a wider improvement to Fivetran and update here if we have news to share.
Please sign in to leave a comment.
Comments
1 comment