Destination Improvement: Implement a support for BigQuery table partitioning in Fivetran UI/API
For connectors with large volumetry especially upon initial sync. The Fivetran write process to BigQuery is very inefficient without partitioning. Scanning the whole table before writing it can cost thousands of USD, which can be easily avoided by applying table partitioning. Currently, the only way to do it with a very cumbersome manual process (https://fivetran.com/docs/destinations/bigquery/partition-table) which requires starting the initial sync stopping it cloning all table schemas manually with partitioning enabled deleting original ones and renaming the paritioned one to original ones. This can be avoided if partitioning would've been supported by default in Fivetran side and configurable before the initial sync. So the proposal is to:
- Implement a feature that allows to choose partitioning column in schema tab. Which when chosen is used to create the initial tables in BigQuery by Fivetran.
- Upon connector setup level enable partitioning for all schemas, if not specified on the schema column level it should fallback on using _fivetran_synced as the default time partition column.
Please sign in to leave a comment.
Comments
0 comments