BigQuery source connector: run query jobs in a customer-owned project instead of the source project
AnsweredWhich connector?: BigQuery (source)
Additional details:
Problem
The BigQuery source connector always creates its query jobs in the source project. The Fivetran service account therefore needs roles/bigquery.jobUser and roles/bigquery.readSessionUser on the source project, for both the setup test and ongoing syncs.
This blocks a common scenario: the source project is owned by a third party, such as a SaaS vendor exposing analytics data through a shared BigQuery dataset. The vendor will grant roles/bigquery.dataViewer on the shared dataset, but will not grant project-level roles like Job User to an external service account. Refusing project-level roles on a vendor-owned project is reasonable on their side.
BigQuery supports this pattern natively: a job running in project A can read tables in project B as long as the caller has read access on the dataset in B. Only the connector's design prevents it.
The existing "Quota project ID" setting looked like it would solve this, but it only affects quota and billing attribution. It does not change where jobs execute. Fivetran Support confirmed this and recommended filing a feature request.
Requested feature
Add a connector option, for example "Job execution project", so the connector creates all BigQuery jobs (setup test, SQL imports, temp tables, Storage Read API sessions) in a customer-owned project. The source project would then only need dataset-level read permissions.
- Source dataset: roles/bigquery.dataViewer (or bigquery.datasets.get, bigquery.tables.get, bigquery.tables.getData, bigquery.tables.list)
- Execution project: roles/bigquery.jobUser, roles/bigquery.readSessionUser, roles/serviceusage.serviceUsageConsumer
- Working / query-results dataset in the execution project: table create/update/read permissions as today
Alternatively, make the existing Quota project ID also act as the job execution project when set.
Why it matters
- Enables syncing vendor-shared BigQuery datasets without asking the vendor for project-level IAM grants.
- Least privilege: the service account gets only dataset-level read on data it does not own.
- Keeps cost and quota in the customer's project, where the data consumer wants it.
- Matches how BigQuery cross-project access already works.
Related: Support mentioned engineering is working on supporting a query-results dataset in a different project. That is complementary but does not cover job execution.
Workaround today
Ask the data owner to grant roles/bigquery.jobUser and roles/bigquery.readSessionUser on their project. This is often declined or slow, and not viable for most vendor-provided datasets.
-
Official comment
Hi there,
This is a useful idea. We have added the request to our feature improvements backlog and we will keep this thread updated with any progress.
Thanks,
Egidio
Please sign in to leave a comment.
Comments
1 comment