Connector Improvement: Add "sg_x_message_id" to SendGrid connector
AnsweredWe have a SendGrid connector and we'd like to have a column in the SendGrid output for "sg_x_message_id". This value would be extracted from the existing `sg_message_id`.
The existing "sg_message_id" column is a combination of SendGrid's `X-Message-ID`, concatenated with a string that represents "server data and the specific identification between a unique message and recipient" (see https://docs.sendgrid.com/
For example, in this Message ID:
XBg2anf2TqCy6WXKQFhieQ.filter0905p1mdw1-4434- 59E0C6FF-3.0
The `XBg2anf2TqCy6WXKQFhieQ` is the X-Message-ID, followed by a `.` and then the reset is the "server data and the specific identification...".
When we make an API request to SendGrid we receive an `X-Message-ID` in the response that we can store. We can then match that x-message-id to sendgrid messages by looking for rows in the SendGrid `events` table where `sg_message_id` starts with `<sg_x_message_id>.`.
Our destination is a Postgres DB. To make our application code easier to write and to make our postgres indexes more efficient and easy to use, we'd like to have the X-Message-ID broken out into a separate column.
E.g. the SQL to extract it could be:
split_part(sg_message_id, '.', 1)
It would be great if Fivetran provided this automatically.
Thanks!
-
Official comment
Hi Jordan,
As a general principle, we extract our data to match the source API as much as we can within reason. Is there a reason why you can't perform this transformation on your end in the destination after we load the data?
Thanks,
Frank
Please sign in to leave a comment.
Comments
1 comment