Connector Improvement: Ablity to replicate multibyte characters
PlannedWe can insert Japanese characters on Sybase ASE where datatype of column is varchar. The same is replicated to MSSQL server without any issues when integrate is run in burst mode. The same doesn't works with continuous mode.
However, when we insert Japanese characters on MSSQL column of datatype varchar, during replication to ASE, data is garbled.
We can insert and retrieve the same data on MSSQL via Management Studio or Aqua data studio. We need HVR to do this during replication.
We need to have ability to replicate multibyte data in varchar from Sybase to MSSQL and vice versa in both continuous mode and burst mode.
We opened a support ticket for this with you and were provided a workaround of using univarchar on Sybase and to enable unicode conversion. However, that cannot be done due to following reasons -
-
Client applications may receive garbled text
If the client uses a different charset (e.g., UTF-8 or SJIS) than the server (iso_1), text will not be converted, resulting in unreadable characters. -
Replication issues may worsen
Replication Server or HVR often relies on conversions for multilingual data. Disabling conversions can cause:- Incorrect data in target MSSQL (e.g., mojibake for Japanese text).
- Possible replication errors if the target collation expects Unicode.
-
Stored procedures and triggers may fail
If they assume conversions for comparisons or sorting, behavior can change unexpectedly. -
Mixed-language environments break
Any client that isn’t using the same charset as ASE will see incorrect data. -
No fallback for Unicode
If you haveenable unicode conversionsturned on, disabling conversions overrides that.
Please help to include the ability to replicate Japanese characters / German Characters or any multibyte characters which can be stored on varchar datatype from Sybase to MSSQL and vice versa ASAP in both continuous and burst integrate mode.
-
Hi Ankur,
Thank you for the detailed explanation and for highlighting the challenges with multibyte character replication from MSSQL to Sybase.
We’re actively looking at how we can enhance HVR’s capabilities in this area to solve this issue. Specifically, we’re exploring the option to override the default database encoding with a user-specified encoding to ensure accurate replication of multibyte characters, such as Japanese and German.
This improvement is currently under feasibility assessment by our engineering team, and while we don’t yet have an estimated timeline, we’ll keep you updated as we make progress.
Best regards,
Edwin -
Hi Edwin/FT engineering,
Post the active ongoing discussion between all parties including FiveTran and new findings here is a high level summary ask :
Support HVR Replication of non-english/international characters in varchar fields from :
1. Sybase ASE - ISO 8859-1 (Latin-1) to Microsoft SQL Server -Latin1_General_100_BIN2_UTF8
2. Microsoft SQL Server -Latin1_General_100_BIN2_UTF8 to Sybase ASE - ISO 8859-1 (Latin-1)
3. Sybase ASE - ISO 8859-1 (Latin-1) to Snowflake (UTF8 default collation)
Based on our in-house tests/repro, none of the above scenarios work.
Please sign in to leave a comment.
Comments
2 comments