BigQuery Overview

BigQuery is Google’s serverless, petabyte-scale data warehouse for analytics. With BigQuery, you can run analytics over large amounts of data using standard SQL queries.

Fivetran can sync to BigQuery as frequently as every five minutes.

Setup guidelink

Follow our step-by-step BigQuery setup guide to connect your BigQuery warehouse with Fivetran.

Type transformation mappinglink

The data types in your BigQuery warehouse follow Fivetran’s standard data type storage. We use the following data type conversions:

Fivetran represents nested JSON (including arrays) as STRING columns in BigQuery. This happens with semi-structured data sources (for example: webhooks, MongoDB, JSON files). The easiest way to query these columns is to define a user-defined function (UDF) to parse the text. Here is an example of this pattern. For different use cases, change the return type to match the data being parsed.

create temporary function structure(json string)
returns array<struct<x int64, y int64>>
language js as """
return JSON.parse(json)
""";

Example using this UDF:

with original_table as (select '[{"x":1,"y":1}, {"x":2,"y":2}]' as json_text),
structured_view as (select structure(json_text) as json_array from original_table)
select * from structured_view

More complicated UDFs can pick specific deeply nested fields and flatten arrays.

Flatten and query JSON datalink

Fivetran supports JSON flattening. By default, the feature is not enabled on your destination because Fivetran doesn’t support automatic data type promotion from STRING to JSON.

Contact our support team to enable the feature on your destination. After we enable the feature, use your connector’s dashboard to drop and re-sync the tables.

For example, if you sync a source column json_col with the following JSON format:

json_col {"id": 123, "nums" : [1, 2, 3, 4], "address": {"city":"ny", "pin":12345}}

We flatten the data and write the data in your destination as:

Although, this is a single column of JSON data, you can execute queries like they are separate columns. For example:

select json_col.id, json_col.address.city from <datasetId>.<tableId>;

You may observe longer sync durations for the tables because JSON columns consume more computation resources.

If we find non-JSON data type or corrupt data in the source column, we will convert the whole column to STRING type in the destination.

NOTE: Contact our support team if you want to disable JSON flattening on your destination.

Roles and permissionslink

The minimum role required for Fivetran to function is BigQuery User. This role provides permissions to:

• run, enumerate, and cancel our own jobs (updates)
• enumerate all datasets within a project
• create new datasets within the project and grant the bigquery.dataOwner role for these new datasets

These permissions are required because Fivetran creates its own datasets and modifies them as changes are made to the source. These permissions do not allow Fivetran to modify datasets it does not own. Use a separate project for datasets that you do not want Fivetran to be able to query.

Data load costslink

Fivetran uses SQL queries to merge data into the tables in your warehouse. Because BigQuery bills for query usage, they charge you when Fivetran loads data in your warehouse. Partioned tables significantly reduce costs because they allow Fivetran to quickly identify where to insert data in a table without scanning every row.

Column nameslink

Fivetran ignores the case of column names in your destination tables as BigQuery is case-insensitive

Partition modification limitationslink

BigQuery limits the maximum number of partition modifications. When we detect a weak primary key in a table, we update the data in the destination table using the following steps:

1. Create a copy table that contains all the data of the main table.

2. Delete the rows that are present in the staging table

3. Copy with append all the data from the staging table to the copy table.

4. Copy with truncate all the data from the copy table to the main table.

   NOTE: If the main table is a column partitioned table, the copy with truncate operation truncates the destination table and copies all data from the copy table to the main table. The operation modifies all the partitions in the main table.

If the sync fails with the QuotaExceededException error, we delete the main table and then copy it back from the copy table to reset the partition quota for the day. You can also resolve this error by:

• Increasing your BigQuery quota
• Reducing your sync frequency