Connector Improvement: Customizable Deserializing / decrypting functions
Currently trying to tap into a Kafka Broker to stream data into BigQuery.
Problem: the message value received from the Broker is encrypted with a customized function. As such, the encrypted received value is most of the time discarded, and sometimes transferred to BigQuery in it's encrypted form.
It would make sense to let the user select their own value-deserializer function/class and if needed, be able to load their own class as a jar file.
-
Official comment
Dear Samuel,
Thank you for sharing your experience - can you confirm which of our Kafka connectors you were using?
Also what messages type and encryption would you need to customize?
Conceptually the request makes sense, however I think we'd need a little more information to take it forward.Best regards
Alison
Please sign in to leave a comment.
Comments
1 comment