Rails migration "invalid byte sequence in UTF-8"

Question

I'm trying to run a migration, but whenever I try to I get the error "invalid byte sequence in UTF-8". Here is the migration:

class ChangeSchoolEmailToApplicationEmail < ActiveRecord::Migration
  def self.up
    rename_column :users, :school_email, :application_email
  end

  def self.down
    rename_column :users, :application_email, :school_email
  end
end

If I run just rake db:migrate I get the message that I need to use bundle exec. So the command that I use in the end is bundle exec rake db:migrate which produces the following error:

C:\Rails\qsn>bundle exec rake db:migrate
==  ChangeSchoolEmailToApplicationEmail: migrating ============================
-- rename_column(:users, :school_email, :application_email)
rake aborted!
An error has occurred, this and all later migrations canceled:

invalid byte sequence in UTF-8

Tasks: TOP => db:migrate
(See full trace by running task with --trace)

Help appreciated!

Edit: With --trace the output looks like:

C:\Rails\qsn>bundle exec rake db:migrate --trace
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:migrate
==  ChangeSchoolEmailToApplicationEmail: migrating ============================
-- rename_column(:users, :school_email, :application_email)
rake aborted!
An error has occurred, this and all later migrations canceled:

invalid byte sequence in UTF-8
C:/Ruby/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.4-x86-mingw32/lib/sqlite3/database.rb:48:in `gsub'
C:/Ruby/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.4-x86-mingw32/lib/sqlite3/database.rb:48:in `quote'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/sqlite_adapter.rb:114:in `quote_string'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/abstract/quoting.rb:21:in `quote'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/sqlite3_adapter.rb:45:in `quote'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/sqlite_adapter.rb:372:in `block (2 levels) in copy_table_con
tents'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/sqlite_adapter.rb:372:in `map'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/sqlite_adapter.rb:372:in `block in copy_table_contents'
C:/Ruby/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.4-x86-mingw32/lib/sqlite3/database.rb:142:in `block (2 levels) in execute'
C:/Ruby/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.4-x86-mingw32/lib/sqlite3/statement.rb:110:in `block in each'
C:/Ruby/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.4-x86-mingw32/lib/sqlite3/statement.rb:107:in `loop'
C:/Ruby/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.4-x86-mingw32/lib/sqlite3/statement.rb:107:in `each'
C:/Ruby/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.4-x86-mingw32/lib/sqlite3/database.rb:140:in `block in execute'
C:/Ruby/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.4-x86-mingw32/lib/sqlite3/database.rb:95:in `prepare'
C:/Ruby/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.4-x86-mingw32/lib/sqlite3/database.rb:134:in `execute'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/sqlite_adapter.rb:370:in `copy_table_contents'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/sqlite_adapter.rb:334:in `copy_table'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/sqlite_adapter.rb:311:in `move_table'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/sqlite_adapter.rb:304:in `block in alter_table'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/abstract/database_statements.rb:139:in `transaction'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/sqlite_adapter.rb:303:in `alter_table'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/sqlite_adapter.rb:275:in `rename_column'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:385:in `block in method_missing'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:361:in `block in say_with_time'
C:/Ruby/lib/ruby/1.9.1/benchmark.rb:295:in `measure'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:361:in `say_with_time'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:381:in `method_missing'
C:/Rails/qsn/db/migrate/20120606013213_change_school_email_to_application_email.rb:3:in `up'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:314:in `block in migrate'
C:/Ruby/lib/ruby/1.9.1/benchmark.rb:295:in `measure'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:314:in `migrate'
C:in `migrate'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:539:in `block (2 levels) in migrate'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:613:in `call'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:613:in `block in ddl_transaction'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/connection_adapters/abstract/database_statements.rb:139:in `transaction'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/transactions.rb:207:in `transaction'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:613:in `ddl_transaction'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:538:in `block in migrate'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:525:in `each'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:525:in `migrate'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:435:in `up'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/migration.rb:417:in `migrate'
C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.0.9/lib/active_record/railties/databases.rake:142:in `block (2 levels) in <top (required)>'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:205:in `call'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:205:in `block in execute'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:200:in `each'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:200:in `execute'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
C:/Ruby/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:144:in `invoke'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:112:in `invoke_task'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:90:in `block (2 levels) in top_level'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:90:in `each'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:90:in `block in top_level'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:129:in `standard_exception_handling'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:84:in `top_level'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:62:in `block in run'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:129:in `standard_exception_handling'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:59:in `run'
C:/Ruby/lib/ruby/gems/1.9.1/gems/rake-0.9.2/bin/rake:32:in `<top (required)>'
C:/Ruby/lib/ruby/gems/1.9.1/bin/rake:19:in `load'
C:/Ruby/lib/ruby/gems/1.9.1/bin/rake:19:in `<main>'
Tasks: TOP => db:migrate

Edit:

The type of school_email is String.

Also, I just had a thought- is it possible that the error is a result of a problem in the database? Such as, it could be partially corrupt somehow? The database type is SQLite3.

Answer 1

When you rename a column in SQLite (which doesn't support renaming), the table needs to be recreated, and the data needs to be reinserted. Apparently, some data that it currently has is not UTF-8 encoded. The most likely reason is data corruption, although I'm not sure if there are exceptions to the UTF-8 rule that would allow such data to pass in under odd circumstances.

You'll need to fix the data before you perform the migration, so try searching for the invalid data and fix:

User.all.each do |user|
  email = user.school_email
  email.force_encoding "utf-8"
  unless email.valid_encoding?
    puts "Invalid email: #{email} in user id: #{user.id}" 
    email.encode!("utf-8", "utf-8", :invalid => :replace)
    user.update_attribute(:school_email, email)
  end
end

It's entirely possible that the invalid data is not in the email field, in which case you'll need to do something similar for each string field to determine the culprit