Controller I18n encoding

Question

In a Rails controller, I'd like to compute an i18n message to be sent to front-end. I proceed as follow:

flash[:notice] = I18n.t 'programs.update.program_saved'

In my translation file (fr.yml), the translation is the following one: 'Programme sauvegardé'.

When setting a breakpoint on this line, and then typing it in my console, I have encoding issues:

0> I18n.t 'programs.update.program_saved'
=> "Programme sauvegardé"

I have implemented the AJAX pattern to handle flash messages, and on the front side, I can see the same encoding issue.

Apart from that, when I type the same thing in a rails console, I have no encoding issue.

I'm in Ruby 2.4.4 and Rails 5.2.1.

What can cause this encoding issue, and how to get rid of it?

EDIT: Add additional details

I develop using RubyMine 2018.2. My rails server is running under a WSL (Windows Subsystem for Linux) with an Ubuntu 16. I run my rails server from RubyMine, from Windows side. The rails SDK that is used is a Linux one. The rails server runs on Linux side.

The oigin of my problem was encoding issues in my browser, when showing flash messages set in HTTP request response. These flash message are computed as explained, i.e.: I18n.t 'programs.update.program_saved'.

The problem is the same when I launch my rails server from my RubyMine, or directly from Linux terminal.

To investigate, I wanted to debug and use the RubyMine console. When executing this command from RubyMine debugger console, there still were encoding issues: I18n.t 'programs.update.program_saved'. When doing it from rails console in Linux or Windows (in RubyMine, the rails console executes on Linux side), I have no encoding issue.

Furthermore, the issue is still present when running the app on a heroku instance, so I'm wondering whether this is related to my local config or not.

Answer 1

It is apparent somewhere down the line your String is interpreted as "ISO-8859-1" despite the fact it actually is "UTF-8". You can check the fact with the following code snippet in irb or Rails console:

s=[0x64,0xc3,0xa9].pack('c*')  # => "d\xC3\xA9" ("dé" if UTF-8)
s.encoding    # => #<Encoding:ASCII-8BIT>
s.encode "UTF-8", "UTF-8"      # => "dé"  ("de'")
s.encode "UTF-8", "ISO-8859-1" # => "dé" ("d~A(c)")

I can think of two possibilities for what went wrong.

Case 1

Your terminal where Rails console is run is either incapable of interpreting UTF-8 string or wrongly set up.

Try the following code snippet (nb., it can be run by any one, even if the translation is undefined):

s2 = I18n.t('programs.update.program_saved', :default => nil)
s2 ||= [0x64,0xc3,0xa9].pack('c*').encode("UTF-8", "UTF-8")  # => "dé"  ("de'")
p s2[-2,2].bytes  # => [100, 195, 169]  if the object is in UTF-8
                  # => [100, 233]       if the object is in ISO-8859-1

and you can see how the (internal) encoding of the String object is. If it is [100, 195, 169], then the encoding is UTF-8, and hence both Ruby and Rails treat the translated String object correctly as UTF-8, and so the problem is in you terminal. Your terminal wrongly interprets the byte-string [100, 195, 169] that it received from Rails as ISO-8859-1 and chooses the characters and font to display accordingly.

In Rails consolei on your terminal, you can try this; it should display the characters correctly if terminal is UTF-8 compatible:

[0x64,0xc3,0xa9].pack('c*').force_encoding('UTF-8')
  # => "dé" ("de'") should be displayed.

Chack out your terminal is really capable of displaying UTF-8 strings (most modern terminals should be able to, but old ones may not). Also, check out your terminal settings. This answer to "How to input Unicode character in Rails console?" may be of help.

Case 2

It is Ruby that interpretes the input string as "ISO-8859-1" and internally converted it to "UTF-8" (though it should not happen in the default setting). In that case, a possibility is your yml file may contain some characters that look like "ISO-8859-1"; then Rails might interpret the entire file as "ISO-8859-1" (though very unlikely).

You can check if the file you read (config/locales/fr.yml) is indeed in UTF-8 as follows:

fn = 'config/locales/fr.yml'
IO.binread(fn).force_encoding('UTF-8').valid_encoding?  # => should be true
IO.binread(fn).force_encoding('ISO-8859-1').valid_encoding?  # => false

Unfortunately there is a bit of flaw. Some UTF-8 characters can be legitimately interpreted as ISO-8859-1, and in such cases how a code (Rails) interprets it might vary. If you suspect that is the case, you can look at the output of the above command like IO.binread(fn).force_encoding('UTF-8') and see if every character is as expected.

If the file contains some non-UTF-8 characters, fix it, and hopefully everything will be fine.

Or, in your particular case, you can perhaps fix it as a botch job like

I18n.t('programs.update.program_saved').encode('UTF-8', 'ISO-8859-1')

Note

So long as you want to make the Rails defult as UTF-8 (strongly advised), make sure the default encoding of your app is UTF-8. Check it out via

MyApp::Application.config.encoding  # => #<Encoding:UTF-8>

(Reference: Configuring Rails Applications)

Also, if you are using Heroku, set the default charset to UTF-8. See the answer to "Set UTF-8 as default string encoding in Heroku".

---

Note a major update was made on 2018-11-05 to add Case 1.

Answer 2

Be sure that you open fr.yml file in utf-8 so that what you write is properly saved in utf-8. Probably this is the encoding that your browser is using.

You can go to your Linux console and look for your current configuration by looking the value of the LANG variable. I have for example LANG="ca_ES.UTF-8". Maybe you also may check the encoding properties of your terminal window.

Also check in which encoding you're viewing your site. In Firefox for example, check the view/encoding option.

At the end you need to view the content in the same encoding that you persisted it.