getBytes() With UTF-8 Doesn't Work for Upper-Case German Umlauts

Question

For development I'm using ResourceBundle to read a UTF-8 encoded properties-file (I set that in Eclipse' file properties on that file) directly from my resources-directory in the IDE (native2ascii is used on the way to production), e.g.:

menu.file.open.label=&Öffnen...
label.btn.add.name=&Hinzufügen
label.btn.remove.name=&Löschen

Since that causes issues with the character encoding when using non-ASCII characters I thought I'd be happy with:

ResourceBundle resourceBundle = ResourceBundle.getBundle("messages", Locale.getDefault());
String value = resourceBundle.getString(key);
value = new String(value.getBytes(), "UTF-8");

Well, it does work nicely for lower-case German umlauts, but not for the upper-case ones, the ß also doesn't work. Here's the value read with getString(key) and the value after the conversion with new String(value.getBytes(), "UTF-8"):

&Löschen => &Löschen
&Hinzufügen => &Hinzufügen

&Ã?ber => &??ber
&SchlieÃ?en => &Schlie??en
&Ã?ffnen... => &??ffnen...

The last three should be:

&Ã?ber => &Über
&SchlieÃ?en => &Schließen
&Ã?ffnen... => &Öffnen...

I guess that I'm not too far away from the truth, but what am I missing here?

Google found something similar, but that remained unanswered.

EDIT: a little more code

Answer 1

Today I was talking to one of my colleagues and he was pretty much on the same path as the other answers have mentioned. So I tried to achieve what Jon Skeet had mentioned, meaning creating the same file as in production. Since rebuilding the project after each change of a resource is out of question and I hadn't done any of what solved this (and I guess this will be new to some) let me line it out (even if it may be just for personal reference ;) ). In short this uses Eclipse' project builders.

1. Create an Ant-style build.xml

   <?xml version="1.0" encoding="UTF-8"?>
   <project>
       <property name="dir.resources" value="src/main/resources" />
       <property name="dir.target" value="bin/main" />
   
       <target name="native-to-ascii">
           <delete dir="${dir.target}" includes="**/*.properties" />
           <native2ascii src="${dir.resources}" dest="${dir.target}" includes="**/*.properties" />
       </target>
   </project>
   

   Its intention is to delete the properties-files in the target directory and use native2ascii to recreate them. The delete is necessary as native2ascii won't overwrite existing files.

2. In Eclipse go to the project properties and select "Builders", click "New...", pick "Ant Builder" (that's the slightly enhanced editor for run configurations)
3. In "Main" let "Buildfile" point to the Ant-script, set "Base Directory" to ${project_loc}
4. In "Refresh" tick "Refresh resources upon completion" and pick "The project containing the selected resource"
5. In "Targets" click "Set Targets" next to the "Auto Build" and pick native-to-ascii there (note that for some reason I had to do this later again)
6. This might not be necessary for everybody, but in "JRE" pick a proper execution environment
7. In "Build Options" tick off "Allocate Console" (however, you may want to keep this ticked on until you see that it's all working)
8. "Apply", "OK"
9. I was told that the newly created builder should be somewhere underneath the Java Builder (use Up/Down-button)
10. In the "Java Build Path" select the source folder with the resources (src/main/resources for me) and add an exclusion for **/*.properties

That should have been it. If you edit a properties-file and save it, it should automatically be converted to ASCII in the output folder. You can try with entering ü, which should end up as \u00fc.

Note that if you have a lot of properties-files, this may take some time. Just don't save after every keypress. :)

Answer 2

Some notes:

• If it is a String it is UTF-16 and if it isn't it is a corrupt string (and too late to fix.)
• new String(value.getBytes(), "UTF-8"); - this code will (at best) do nothing on a system that uses UTF-8 as the default encoding; otherwise it will corrupt the string.
• .properties files must be ISO 8859-1 (the Properties type supports other formats and encodings, but I don't know how you would tell ResourceBundle that.)
• System.out can introduce its own transcoding bugs (the PrintStream encodes UTF-16 strings to the default encoding; the receiving device must decode the bytes using the same encoding.)

I suspect you are trying to fix your problems in the wrong place.

Answer 3

The problem is you're calling String.getBytes() without specifying an encoding - which will use the default platform encoding. You're then using the binary result of that operation as if it were in UTF-8.

If you use UTF-8 in both directions, it'll be fine:

// Should be a round-trip
value = new String(value.getBytes("UTF-8"), "UTF-8");

... but if you were trying to use this to read a UTF-8-encoded property file without telling the code which is performing the initial read, that won't work.

The code you've presented is basically always the wrong approach. Your "Since that causes issues with the character encoding" suggests that you'd already run across an earlier problem - so I'd go back to that, instead of trying to apply a broken fix. If you've already lost data when constructing the ResourceBundle, it's too late to go back later... you need to make sure the ResourceBundle itself is loaded correctly.

Please tell us exactly what problems you had with the ResourceBundle, and we can see if we can fix the root cause.

EDIT: It's not clear how you're running native2ascii. The fix may be as simple as changing to use:

native2ascii -encoding UTF-8 input.properties output.properties

Answer 4

You are encoding the text with a different encoding to the one you are decoding with.

Try instead using the same character set for encoding and decoding.

value = new String(value.getBytes("UTF-8"), "UTF-8");

---

String s = "ßßßßß";
s += s.toUpperCase();
s = new String(s.getBytes("UTF-8"), "UTF-8");
System.out.println(s);

prints

ßßßßßSSSSSSSSSS