Include source record identifiers in skip_record warnings
Answered
Add the source primary key to skip_record log entries so dropped records can be identified
Problem
When the connector exhausts its retries against the Gong API it emits a skip_record warning and moves on. That warning includes a Gong requestId but not the call ID of the record being skipped.
The result is that we can see records are being dropped , 1,432 skip_record events on our connection since 15 July but we have no way to determine which records. We cannot reconcile, cannot alert, and cannot tell a stakeholder which of their calls is missing.
Requested behaviour
Include the source record's primary key (call ID for the call endpoint, and equivalent for others) in the skip_record warning message, so it is available in the fivetran_log.log table.
-
Official comment
Hi Amanpreet,
I understand you're missing information that will help you and stakeholders determine which calls are not being synced. I can see it would be valuable to add this to our errors and I'll add this to our backlog. In the meantime I'd like to understand: how often does this happen for you? Is it possible to increase the Gong api limits on your end so you avoid missing data entirely?
Best,
Sadie Martin
Please sign in to leave a comment.
Comments
1 comment