Connector Improvement: Add option to decompress compressed messages in azure service bus connector.
AnsweredThe connector should detect whether the incoming message payload is compressed (e.g., gzip, zlib/deflate, or base64-encoded binary) and transparently decompress it before any parsing or transformation occurs. If the payload is not compressed, the logic should safely fall back to using the raw body.
Additionally, if compressed data reaches the warehouse without being decompressed by the connector, converting the binary payload to a string makes the data permanently un-decompressable. To prevent this issue, the connector must perform decompression before converting the message body into UTF-8 text or JSON. This ensures the warehouse receives clean, readable, and properly decoded data.
-
Official comment
Hi Anuj,
This feature is not currently on our roadmap. We rely on feedback and upvotes to assess user demand; every upvote on this request helps build the case for implementation. Regarding the automatic detection of compressed message payloads, heuristic-based detection can be fragile and may not always reliably identify all compressed formats. To better understand your requirements and help us consider future development, please share more details about your specific use case and payload types.
Thanks,
Parmeet
Please sign in to leave a comment.
Comments
1 comment