ServiceNow connector — translate row filters into sysparm_query instead of filtering after extraction
Which connector?: ServiceNow
Additional details:
Request: For tables with a row filter configured, append the filter to the sysparm_query parameter of the Table API call so non-matching records are never extracted. Today the filter is applied after extraction, so the connector pays the full API and transfer cost of every row it discards.
Proposed behavior — the connector already sends an encoded query for the incremental cursor, so this is an append:
Current: /api/now/table/em_alert?sysparm_query=sys_updated_on>=<cursor>
Proposed: /api/now/table/em_alert?sysparm_query=sys_updated_on>=<cursor>^u_suppress_escl=Normal
Must apply to both historical and incremental syncs, and fall back to post-extraction filtering if a predicate can't be expressed.
Why this matters:
- Filters give no performance benefit today — sync time scales with total table size, not the requested subset.
- ServiceNow tables often have Table Cleanup rules that delete records after N days. When extraction is slower than the source's growth, records are purged before the connector reads them — permanent, silent data loss.
- Historical syncs can fail to converge. If extraction throughput is below the table's row-generation rate, the sync never reaches the present and never switches to incremental, with no error raised.
- Sync frequency and rollback window cannot compensate, because the constraint is extraction volume, not scheduling.
Measured on our connector: 371,366 rows extracted to retain 1,249 (0.34% yield), at a throughput below the table's daily growth — so the sync could never converge.
Please sign in to leave a comment.
Comments
0 comments