Tradedesk Custom Report Grain Detection and Schema Discovery
AnsweredWhich connector?:
The Trade Desk
Additional details:
This request has two parts:
1. Currently the connector uses report_execution_id as the only primary key for upserting data, which means that all rows loaded for each report count as MAR, even if they restate data from previous reports. The TradeDesk GQL API exposes which columns of a custom report are 'GRAIN' and which are 'MEASURE' via querying "columnUsageType" on the "ReportingColumn" object, so Fivetran could detect the grain of the report and create a composite key from the combination of the GRAIN columns.
This would help reduce cost and bring the destination tables closer to a replication of the platform data. For reports that include conversion counts, the MAR can be 30x higher using only execution ID as key compared to using the logical grain of the report.
2. The update schema button for this connector does not discover new custom reports. This means that the only way to discover new reports is to run a sync. For our Trade Desk seat where there are many reports being created constantly that we don't want to ingest with fivetran, this means that we need to sync all those reports, then deselect all but the single new report we want to add any time we need to start ingesting a new report. The update schema function could query available report schedules without ingesting data from those reports.
Thanks for consideration!
-
Official comment
Hi Adam,
Thank you for the detailed request and for including the technical context. We've investigated both parts and confirmed they're feasible.
For the grain detection piece, the column usage type (GRAIN vs. MEASURE) looks like the right mechanism to use for building a composite primary key. This change is not straightforward because it would require migrating all existing connections' reports. We're going to weigh options for how to do this without massive impacts to existing users.
For schema discovery, you're right that the Update Schema button currently doesn't surface new custom reports without a full sync. We're exploring how we can have that UI action fetch the report configurations.
We've logged both as separate internal feature improvement tickets. We can't commit to a timeline, but both are now on our radar with your technical context attached. We'll update this thread when there's progress. Additional upvotes for these requests from other Fivetran users will help boost the priority of these improvements.
Thanks again for the thorough write-up.
Luke
Please sign in to leave a comment.
Comments
1 comment