CODEWITHSUNDEEP
rubyutf-8
Artem Kalinchuk
Artem Kalinchuk

Reputation: 6652

gsub :: ArgumentError (invalid byte sequence in UTF-8)

This code uses the Hpricot gem to get HTML that contains UTF-8 characters.

# <div>This is a test<a href="">测试</a></div>
div[0].to_html.gsub(/test/, "")

When that is run, it spits out this error (pointing at gsub):

ArgumentError (invalid byte sequence in UTF-8)

How can we fix this issue?

Upvotes: 3

Views: 6024

Answers (2)

Artem Kalinchuk
Artem Kalinchuk

Reputation: 6652

Figured out the issue. Hpricot's to_html calls methods that trigger the error so to get rid of that we need to make the Hpricot document encoding UTF-8, not just that one string. We do that like this:

ic = Iconv.new("UTF-8//IGNORE", "UTF-8")
doc = open("http://example.com") {|f| Hpricot(ic.iconv(f.read)) }

And then we can call other Hpricot methods but now the whole document has UTF-8 encoding and it won't give us any errors.

Upvotes: 3

lcoq
lcoq

Reputation: 10726

The to_html looks to return a non-utf8 string in this case.

I had same problem with file containing some non-utf8 characters. The fix I found is not really beautiful, but it could also works for your case :

the_utf8_string = the_non_utf8_string.unpack('C*').pack('U*')

Be careful, I'm not sure there is no one data lost.

Upvotes: 0

Related Questions