CODEWITHSUNDEEP
ruby-on-railsruby
loganathan
loganathan

Reputation: 6176

How to check whether the character is utf-8

How to check whether the character set is in utf-8 encoding,through ruby|ror ?

Upvotes: 13

Views: 21716

Answers (3)

Andreas Rayo Kniep
Andreas Rayo Kniep

Reputation: 6562

Check UTF-8 Validity

For most multi-byte encodings it is possible to programmatically detect invalid byte-sequences. As Ruby by default treats all strings to be UTF-8, you can check if a string is given in valid UTF-8:

# encoding: UTF-8
# -------------------------------------------
str = "Partly valid\xE4 UTF-8 encoding: äöüß"

str.valid_encoding?
   # => false

str.scrub('').valid_encoding?
   # => true

Convert Encoding

Additionally, if a string is not valid UTF-8 encoding, but you know the actual character-encoding, you can convert the string to UTF-8 encoding.

Example
Sometimes, you end up in a situation, in which you know that the encoding of an input-file is either UTF-8 or CP1252 (a.k.a. Windows-1252).
Check which encoding it is and convert to UTF-8 (if necessary):

# encoding: UTF-8
# ------------------------------------------------------
test = "String in CP1252 encoding: \xE4\xF6\xFC\xDF"
File.open( 'input_file', 'w' ) {|f| f.write(test)}

str  = File.read( 'input_file' )

unless str.valid_encoding?
  str.encode!( 'UTF-8', 'CP1252', invalid: :replace, undef: :replace, replace: '?' )
end #unless
   # => "String CP1252 encoding: äöüß"

=======
Notes

  • It is programmatically possible to detect most multi-byte encodings like UTF-8 (in Ruby, see: #valid_encoding?) with pretty high reliability. After only 16 bytes, the probability of a random byte-sequence being valid UTF-8 is only 0.01%. (Compare this with relying on the UTF-8 BOM)

  • However, it is NOT easily possible to programmatically detect (in)validity of single-byte-encodings like CP1252 or ISO-8859-1. Thus the above code snippet does not work the other way around, i.e. detecting if a String is valid CP1252 encoding.

  • Even though UTF-8 has become increasingly popular as the default encoding in the web, CP1252 and other Latin1 flavors are still very popular in the Western countries, especially in North America. Be aware that there a several single-byte encodings out there that are very similar, but slightly vary from CP1252 (a.k.a. Windows-1252). Examples: ISO-8859-1, ISO-8859-15

Upvotes: 20

Mladen Jablanović
Mladen Jablanović

Reputation: 44110

There's no definite way to do this, in Ruby nor anywhere else:

str = 'foo' # start with a simple string
# => "foo" 
str.encoding
# => #<Encoding:UTF-8> # which is UTF-8 encoded
str.bytes.to_a
# => [102, 111, 111] # as you can see, it consists of three bytes 102, 111 and 111
str.encode!('us-ascii') # now we will recode the string to 8-bit us-ascii encoding
# => "foo" 
str.encoding
# => #<Encoding:US-ASCII> 
str.bytes.to_a
# => [102, 111, 111] # see, same three bytes
str.encode!('windows-1251') # let us try some cyrillic
# => "foo" 
str.encoding
# => #<Encoding:Windows-1251> 
str.bytes.to_a
# => [102, 111, 111] # see, the same three again!

Of course, you can employ some statistical analysis on the text, and eliminate encodings which the text is not valid for, but theoretically, this is not solvable problem.

Upvotes: 13

sawa
sawa

Reputation: 168269

"your string".encoding
 # => #<Encoding:UTF-8>

Or if you want it progmatically,

"your string".encoding.name == "UTF-8"
 # => true

Upvotes: 1

Related Questions