CODEWITHSUNDEEP
rubyencodingsqlite
Andrzej Gis
Andrzej Gis

Reputation: 14306

Encoding issue with Sqlite3 in Ruby

I have a list of sql queries beautifully encoded in utf-8. I read them from files, perform the inserts and than do a select.

# encoding: utf-8 
def exec_sql_lines(file_name)
  puts "----> #{file_name} <----"
  File.open(file_name, 'r') do |f|
    # sometimes a query doesn't fit one line
    previous_line=""
    i = 0
    while line = f.gets do
      puts i+=1

      if(line[-2] != ')')
        previous_line += line[0..-2]
        next
      end

      puts (previous_line + line)  #  <----  (1)

      $db.execute((previous_line + line))
      previous_line =""
    end
    a = $db.execute("select * from Table where _id=6")
    puts a <---- (2)
  end
end

$db=SQLite3::Database.new($DBNAME)
exec_sql_lines("creates.txt")
exec_sql_lines("inserts.txt")
$db.close

The text in (1) is different than the one in (2). Polish letters get broken. If I use IRB and call $db.open ; $db.encoding is says UTF-8.

Why do Polish letters come out broken? How to fix it?

I need this database properly encoded in UTF-8 for my Android app, so I'm not interested in manipulating with database output. I need to fix it's content.

EDIT

Significant lines from the output:

6
INSERT INTO 'Leki' VALUES (NULL, '6', 'Acenocoumarolum', 'Acenocumarol WZF', 'tabl. ', '4 mg', '60 tabl.', '5909990055715', '2012-01-01', '2 lata', '21.0, Leki przeciwzakrzepowe z grupy antagonistów witaminy K', '8.32', '12.07', '12.07', 'We wszystkich zarejestrowanych wskazaniach na dzień wydania decyzji', '', 'ryczałt', '5.12')

out:

6
6
Acenocoumarolum
Acenocumarol WZF
tabl. 
4 mg
60 tabl.
5909990055715
2012-01-01
2 lata
21.0, Leki przeciwzakrzepowe z grupy antagonistĂł[<--HERE]w witaminy K
8.32
12.07
12.07
We wszystkich zarejestrowanych wskazaniach na dzieĹ[<--HERE] wydania decyzji

ryczaĹ[<--HERE]t
5.12

Upvotes: 0

Views: 513

Answers (1)

knut
knut

Reputation: 27855

There are three default encoding.

In you code you set the source encoding. Perhaps there is a problem with External and Internal Encoding?

A quick test in windows:

#encoding: utf-8
File.open(__FILE__,'r'){|f|
  p f.external_encoding
  p f.internal_encoding
  p f.read.encoding
}

Result:

#<Encoding:CP850>
nil
#<Encoding:CP850>

Even if UTF-8 is used, the data are read as cp850.

In your case: Does File.open(filename,'r:utf-8') help?

Upvotes: 2

Related Questions