Ability to resume Github connector syncs from last known-good state
Which connector?: Github connector
Additional details: We had a long-running ticket with Fivetran support (Fivetran team - please see ticket 396873) due to a temporary break in our Github connection (certain tables were turned off for a couple of days, coinciding with a loss of webhook permissions causing the connector to enter an error state).
This break caused a resync to begin from the very beginning of Github history. This took 3 weeks, and caused a lot of pain when trying to get up-to-date internal metrics. We ultimately had to build a backup self-serve solution using the GraphQL API.
Concretely, what we are asking for:
- For some sort of checkpoint visibility to be provided so that we can know how far the connector has progressed
- When a connector breaks, but before triggering a full historical resync, show the last couple of known-good checkpoint states so that we can resume from there
Specifically in this case we would have been looking for resuming the sync from a certain date (as all data before that had settled).
Please sign in to leave a comment.
Comments
0 comments