Allow option to specify the partition column to use when scanning BigQuery table
Answered
Which connector?: Big Query
Additional details: Currently FiveTran does not provide any option to specify on which column a BigQuery table is partitioned to efficiently query the table. We have seen FiveTran is doing full scan on every sync and taking hours of time and costing us $$$. Whatever design principle used is clearly not helping. This connector should have been designed performance in mind. With this request I'm asking FiveTran to revisit heir design principles and improve performance and importantly allow customer to specify the partition column that should be used to scan the BQ table. Please refer to ticket#372576 about the painful experience customer is having with this connector. Thanks!
-
Hi Samir,
Thanks for sharing this feedback—we understand how impactful performance and cost can be, especially with BigQuery.
To better understand your use case, could you provide a bit more detail on:
- Which destination setup you’re using
- How frequently the connector is syncing
- The specific tables or data volumes involved
- The types of queries or workloads you’re running on the destination that are being impacted
This information will help us better understand the scenario and evaluate it further.
Best,
Please sign in to leave a comment.
Comments
1 comment