Alerting and status visibility for skipped records
Surface skipped-record counts in connector status and support alerting on them
Problem
Records dropped after retry exhaustion are logged as warnings, not errors. The sync then completes with status SUCCESS. Nothing in the Fivetran UI, and no alert, indicates that data was lost.
On our connection this meant roughly 80% of calls were being dropped for two weeks while the dashboard showed nothing but successful syncs.
The status metrics actively obscured the problem. Every hourly sync reported approximately 2,534 loaded rows and a green "Successful sync". That figure turned out to be almost entirely the users table, which is fully rewritten on every sync.
Requested behaviour
- Surface a skipped-record count in the connector status view, per sync and per table
- Allow alerting to be configured on skipped-record counts, either absolute or as a proportion of expected volume
- Consider a distinct sync status such as "Completed with warnings" where records were skipped, so that a fully successful sync is distinguishable from a lossy one
- Optionally, break down loaded-row counts by table in the status view, so a single high-volume table cannot mask low throughput elsewhere
Please sign in to leave a comment.
Comments
0 comments