Include source record identifiers in skip_record warnings
Add the source primary key to skip_record log entries so dropped records can be identified
Problem
When the connector exhausts its retries against the Gong API it emits a skip_record warning and moves on. That warning includes a Gong requestId but not the call ID of the record being skipped.
The result is that we can see records are being dropped , 1,432 skip_record events on our connection since 15 July but we have no way to determine which records. We cannot reconcile, cannot alert, and cannot tell a stakeholder which of their calls is missing.
Requested behaviour
Include the source record's primary key (call ID for the call endpoint, and equivalent for others) in the skip_record warning message, so it is available in the fivetran_log.log table.
Please sign in to leave a comment.
Comments
0 comments