Connector Improvement: Google Cloud Function: Allow loading of repeated records
Not planned
It would be great if we could make use of the BigQuery functionality to load nested and repeated data.
Currently, it looks like it is not possible to create a response object with this kind of data structure, e.g.:
{
'insert': {
'buttons_test': {
'run_time': job_start,
'data': [
{
'record1_key1': 'value_a',
'record1_key2': 'value_b'
},
{
'record2_key1': 'value_c',
'record2_key2': 'value_d'
}
]
}
}
}
If I return a response object with the above format, I get an error with the following in the `data` object:
"data": {
"status": "FAILURE_WITH_TASK",
"reason" : "javax.ws.rs.client.ResponseProcessingException: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.util.ArrayList` out of START_OBJECT token\n at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 27] (through reference chain: com.fivetran.integrations.functions.common.Response[\"insert\"]->com.fivetran.integrations.functions.common.Response$Insert[\"<TABLE_NAME>\"])",
"taskType" : "response_processing_error"
}
-
Official comment
Hi Fabio,
Thank you for sharing a detailed description of the issue you are encountering. Unfortunately we don't support writing to BigQuery Struct Type because it interferes with some of our core value added functionality around Data Types, automatic inference and Schema Migrations.
I think structuring your data a little differently will allow you to deliver it relaibly to Big Query in a way you can easily transform it for later use ones its arrived.
Best regards
Alison
Please sign in to leave a comment.
Comments
1 comment