Skip to main content
Documentation
Feature Requests Trust Center Status Page

Community

Connector Improvement: Add support for JSON_SR data format in Confluent Cloud connector

Kirill Andreev User
  • Edited

The current Confluent Cloud connector supports 4 message types: JSON, Avro, Protobuf, and Text. This feature request is for this connector to also support JSON_SR message types. 

Relevant Fivetran reply from the original ticket diagnosing our Confluent connector issue: 

According to our engineering team, messages encoded in JSON_SR format will have schema_id and magic byte appended to the start of the message.
 
We currently do no support this.

Without JSON_SR support every user currently receiving JSON_SR message types has to default to syncing their data as Text. This means creating an additional custom ETL layer which dynamically determines the schema of the message and unpacks its contents. These are exactly the kind of extra steps we expect Fivetran to be capable of handling and, so, is why we are submitting this request.

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Official comment
    Alison Fivetranner

    Hi Kirill,

    Thank you for taking the time to request support for this file format type and for the provision of the details.

    Can you share a little about these JSON_SR message types - are these JSON messages that have been configured to work with the Confluent Schema Registry?

    When are you finding them to be the right selection for your data pipe? how is your data pipe configured to send data in this format? - I'm trying to understand more about how frequently this message type would be crucial and what behavior will be expected when using this type of data.

    Best regards

    Alison

  • Kirill Andreev User

    Hi Alison, with regards to your questions:

    are these JSON messages that have been configured to work with the Confluent Schema Registry?

    Yes, that is correct.

    When are you finding them to be the right selection for your data pipe? how is your data pipe configured to send data in this format?

    The JSON_SR message type allows us to specify the schema along with the JSON. In case it helps: relevant Confluent documentation on the JSON Serializer here: https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/serdes-json.html

    Thanks for following up!
    Kirill