Sync performance improvement for Tinuiti/Tinuiti_Datalake/partnerize/partnerize__wolverineworldwide
AnsweredWe are creating this FI ticket on behalf of the customer, as requested by the engineering team.
Additional details:
The customer has reached out to the source team and shared the details below to improve the sync performance of the Partnerize connection.
Parent Jira ticket: https://fivetran.atlassian.net/browse/RD-1284787
From Partnerize source team:
-
Looking at the architecture of the API request, the 503 Service Unavailable responses are certainly being caused by query-induced timeouts rather than a system outage. The /analytics/ endpoint becomes overloaded with highly complex requests, and encounters "cool off" periods which results in the dropped connection. Based on my checks, there were no incidents or maintenance windows on 10/6 that would result in this error.
-
The friction here lies in how the explode endpoint is being used vs. how it is designed to function. By using 30 group_by dimensions alongside 17 metrics, you're effectively forcing an aggregation endpoint to output raw, row-level granular data. Computing the permutations for 30 dimensions (especially on historical data from 2024 which falls into our 2+ year rate-limiting tier and is highly resource-intensive) forces our database to process a massive multidimensional cube. This exhausts the processing capacity and hits our internal timeout thresholds resulting in the 503 errors.
-
To pull granular, conversion-level data without hitting timeouts or capacity limits, you should use our dedicated /reporting/ and /export/ endpoints. These are designed to handle heavy, raw data extraction.
Option A:
GET /reporting/report_advertiser/campaign/{campaign_id}/conversion.json
-
This endpoint returns basket/item-level conversion data. Because historical pulls involve massive scale, the endpoint utilizes cursor-based pagination. Instead of computing everything at once, the system hands you a cursor to efficiently page through a bunch of rows without timing out.
Option B:
GET /reporting/export/export/conversion_item.csv
-
This endpoint provides a granular report where every single item sits on its own row. By passing the parameters
limit = 0andcomplete_email:{your_email}, you instruct Partnerize to process the data as a background job. This completely bypasses standard API timeout windows and is incredibly stable for warehouse ingestion.
By switching to cursor-paginated reporting or the asynchronous export, you will align your request with the indented API architecture, which will resolve the 503 cooling-off periods.
Listen to the API's built-in signals: Whenever you make a request, the API communicates its current capacity via response headers. To prevent your connector from failing, we recommend programming it to read and respect these headers.
-
X-RateLimit-Limit: Your total permitted token allowance. -
X-RateLimit-Remaining: The number of tokens you have left in the current window. -
X-RateLimit-Reset: The seconds remaining until your token allowance fully replenishes. -
X-RateLimit-Retry-After: The exact number of seconds your connector needs to wait before trying again.
Best Practice: Build backoff logic into your connector that listens for the X-RateLimit-Retry-After header. When a 429 is triggered, your system should naturally pause for the requested seconds and seamlessly resume, rather than forcefully retrying and compounding the errors.
Understand Analytics Query Costs: Our /analytics/ API operates on a dynamic token-cost model. The heavier your query, the more tokens it consumes from your allowance. This cost is calculated based on the specific endpoint you're hitting and the breadth of your date range. The breakdown of costs can be found here. As your date ranges expand, so do the costs.
Strategies for High-Volume Extraction:
-
Favor /timeseries/ over /explode/ - The explode endpoint requires heavy lifting to group across multiple dimensions. If you only need aggregated totals grouped by date, switch to timeseries endpoint. It is significantly faster, costs fewer tokens, and naturally zero-fills missing dates.
-
Adopt Cursor Pagination - If you're extracting massive lists of granular conversions, move away from standard limit/offset pagination and use cursor-based pagination (
cursor_id). Offset pagination compounds in processing overhead as you go deeper into the pages, whereas cursor pagination maintains a lightweight, consistent read of the dataset no matter how many pages you pull. -
Keep Date Ranges Tight - Only query the exact timeframes you require. Breaking massive historical pulls into smaller, incremental chronological chunks (<1 year at a time) will keep your token costs at their absolute minimum.
-
Implement Caching - Cache any relatively static data locally on your end so you aren't spending tokens on redundant API calls.
-
Official comment
Hi Jagadish,
Thank you for bringing this feature request to our attention. We are actively working on improving sync performance for the Partnerize connection, and I can confirm that this enhancement is planned. Based on the current roadmap, we anticipate completion within the next quarter.
Best,
Sanjeevv Maniram
Please sign in to leave a comment.
Comments
1 comment