CODEWITHSUNDEEP
javaunicodeutf-8redischaracter-encoding
User
User

Reputation: 4211

Character encoding/decoding in redis

I'm retrieving some special characters from Redis to Java program, but these are not encoded/decoded properly. These characters have been accessed from an api, inserted into redis cache and then used in java program.

On Postman (REST Client):

[^!-~°-µ\\\\s±–—‐“”’¼-¾]

On Redis Cache console (redis-cli.exe):

[^!-~\xef\xbf\xbd\xef\xbf\xbd-\xef\xbf\xbd\xef\xbf\xbd\\\\\\\\s\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd-\xef\xbf\xbd\xef\xbf\xbd]

In Java Program:

[^!-~��-��\s����������������������-��]

Don't know what's going on here! Why these characters have been converted in that xxx format. Anybody please help!

Upvotes: 1

Views: 5283

Answers (2)

Serge Ballesta
Serge Ballesta

Reputation: 148880

I can explain what you see, but I cannot fix the problem.

° is unicode character DEGREE SIGN or U+00B0 and is encoded in UTF-8 as \xc2\xb0

μ is unicode character GREEK SMALL LETTER MU or U+03BC and is encoded in UTF-8 as \xce\xbc

� is unicode character REPLACEMENT CHARACTER or U+FFFD and is encoded in UTF-8 as \xef\xbf\xbd

Your posts suggests that each of the non ascii character was replaced (I cannot know how) with two replacement characters in Redis console and Java program. The Java program correctly outputs the character while the Redis console outputs its UTF-8 representation.

I suspect that the characters were originally passed in UTF-8 and used 2 bytes over 0x7F. But when this was converted to Java 16 bits characters, the charset used for decoding was plain ascii and all were converted into the unicode replacement character. I cannot say more without a clear explaination of the exact worklow.

Upvotes: 2

qwerty
qwerty

Reputation: 2512

I have observed similar issues related to character set on windows machines. Use Charset.forName("windows-1252") (if this doesn't work check the desired encoding) wherever you are dealing with this input stream.

Upvotes: 0

Related Questions