CODEWITHSUNDEEP
ruby-on-railsrubyjsonencoding
Brian
Brian

Reputation: 2619

JSON encoding/decoding with unicode in rails

I upgraded downgraded to rails 2.3.17 due to the security bugs, but now I can't decode json strings that I have saved down to a DB if they have unicode in them :(. Is there a way to process the string such that it decodes properly?

e = ActiveSupport::JSON.encode({'a' => "Hello Unicode \u2019"})
ActiveSupport::JSON.decode(e)

gives me

RangeError: 8217 out of char range
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:314:in `unquote'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:251:in `strtok'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:215:in `tok'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:178:in `lex'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:46:in `decode'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:612:in `decode'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/decoding.rb:14:in `decode'
from (irb):30
from /usr/local/bin/irb:12:in `<main>'

I can't change the first line since it's coming from the DB like that.

This used to work.

Upvotes: 1

Views: 4012

Answers (3)

roo
roo

Reputation: 7196

You can change the backend JSON provider in ActiveSupport.

Add ActiveSupport::JSON.backend = "JSONGem" into an application initialiser (I added it to application.rb). This fixed the unicode parsing issues I had after I upgraded activesupport to 3.0.20.

See the vulnerability notice which caused this update - It mentions that this workaround should apply to 2.3.16 as well.

From rails console:

> ActiveSupport::VERSION::STRING
 => "3.0.20" 
> ActiveSupport::JSON.decode('{"test":"string\u2019"}')
RangeError: 8217 out of char range
> ActiveSupport::JSON.backend = "JSONGem" 
> ActiveSupport::JSON.decode('{"test":"string\u2019"}')
 => {"test"=>"string’"}

Upvotes: 3

Brian
Brian

Reputation: 2619

  • Switch to 2.3.15 which should be fine because that's when the fixes landed.
  • Curse the developer who started this project in rails
  • Begin work on porting to python post haste

Upvotes: -1

tadman
tadman

Reputation: 211580

The JSON gem will handle this correctly.

As a note, the gem is much more strict than the other JSON parsers out there. For example:

{ 'test' : 'value' }

This is not valid JSON even though it looks okay.

For whatever reason the non-UTF-8 savvy JSON parser shipped as part of the 2.3.16 patch which is really sloppy on the part of the maintainer.

Upvotes: 0

Related Questions