CODEWITHSUNDEEP
rubycsvcharacter-encodingoutputputs
Assad Ebrahim
Assad Ebrahim

Reputation: 6361

Wierd output characters (Chinese characters) when using Ruby to read / write CSV

I'm trying to print the first 5 lines from a set of large (>500MB) csv files into small headers in order to inspect the content more easily.

I'm using Ruby code to do this but am getting each line padded out with extra Chinese characters, like this:

 week_num   type    ID  location    total_qty   A_qty   B_qty   count਍㌀㐀ऀ猀漀爀琀愀戀氀攀ऀ㄀㤀㜀ऀ䐀䔀开伀渀氀礀ऀ㔀㐀㜀㈀　㌀ऀ㔀㐀㜀㈀　㌀ऀ　ऀ㤀㄀㈀㔀㌀ഀ
 44 small   14  A   907859  907859  0   550360਍㐀㄀ऀ猀漀爀琀愀戀氀攀ऀ㐀㈀㄀ऀ䐀䔀开伀渀氀礀ऀ㌀　㈀㄀㜀㐀ऀ㌀　㈀㄀

The first few lines of input file are like so:

 week_num   type    ID  location    total_qty   A_qty   B_qty   count
 34 small   197 A   547203  547203  0   91253
 44 small   14  A   907859  907859  0   550360
 41 small   421 A   302174  302174  0   18198

The strange characters appear to be Line 1 and Line 3 of the data.

Here's my Ruby code:

num_lines=ARGV[0]  
fh = File.open(file_in,"r")
fw = File.open(file_out,"w")
until (line=fh.gets).nil? or num_lines==0
    fw.puts line if outflag
    num_lines = num_lines-1
end

Any idea what's going on and what I can do to simply stop at the line end character?

Looking at input/output files in hex (useful suggestion by @user1934428)

Input file - each character looks to be two bytes.

enter image description here

Output file - notice the NULL (00) between each single byte character...

enter image description here

Ruby version 1.9.1

Upvotes: 1

Views: 1092

Answers (1)

Assad Ebrahim
Assad Ebrahim

Reputation: 6361

The problem is an encoding mismatch which is happening because the encoding is not explicitly specified in the read and write parts of the code. Read the input csv as a binary file "rb" with utf-16le encoding. Write the output in the same format.

num_lines=ARGV[0]  

# ****** Specifying the right encodings  <<<< this is the key
fh = File.open(file_in,"rb:utf-16le")
fw = File.open(file_out,"wb:utf-16le")

until (line=fh.gets).nil? or num_lines==0
    fw.puts line
    num_lines = num_lines-1
end

Useful references:

Upvotes: 1

Related Questions