Ability to resume Github connector syncs from last known-good state
AnsweredWhich connector?: Github connector
Additional details: We had a long-running ticket with Fivetran support (Fivetran team - please see ticket 396873) due to a temporary break in our Github connection (certain tables were turned off for a couple of days, coinciding with a loss of webhook permissions causing the connector to enter an error state).
This break caused a resync to begin from the very beginning of Github history. This took 3 weeks, and caused a lot of pain when trying to get up-to-date internal metrics. We ultimately had to build a backup self-serve solution using the GraphQL API.
Concretely, what we are asking for:
- For some sort of checkpoint visibility to be provided so that we can know how far the connector has progressed
- When a connector breaks, but before triggering a full historical resync, show the last couple of known-good checkpoint states so that we can resume from there
Specifically in this case we would have been looking for resuming the sync from a certain date (as all data before that had settled).
-
Official comment
Hi Ishaan,
This is a valuable idea and we will investigate this further. We'll keep the community updated on this thread with any progress or changes related to this feature request.
Thanks,
Frank
Please sign in to leave a comment.
Comments
1 comment