rake invalid byte sequence in UTF-8

Question

Out of the blue I am getting a weird error for any rake command on my webserver running rails 3.2 and Ruby 1.9.3p125 and the stack trace is the same no matter what rake task. There is nothing but what exists in ascii in Rakefile and lib/tasks.

The stack trace:

rake --trace
rake aborted!
invalid byte sequence in UTF-8
/usr/local/lib/ruby/1.9.1/rake/application.rb:183:in `glob'
/usr/local/lib/ruby/1.9.1/rake/application.rb:183:in `block in have_rakefile'
/usr/local/lib/ruby/1.9.1/rake/application.rb:181:in `each'
/usr/local/lib/ruby/1.9.1/rake/application.rb:181:in `have_rakefile'
/usr/local/lib/ruby/1.9.1/rake/application.rb:468:in `find_rakefile_location'
/usr/local/lib/ruby/1.9.1/rake/application.rb:486:in `raw_load_rakefile'
/usr/local/lib/ruby/1.9.1/rake/application.rb:82:in `block in load_rakefile'
/usr/local/lib/ruby/1.9.1/rake/application.rb:133:in `standard_exception_handling'
/usr/local/lib/ruby/1.9.1/rake/application.rb:81:in `load_rakefile'
/usr/local/lib/ruby/1.9.1/rake/application.rb:65:in `block in run'
/usr/local/lib/ruby/1.9.1/rake/application.rb:133:in `standard_exception_handling'
/usr/local/lib/ruby/1.9.1/rake/application.rb:63:in `run'
/usr/local/bin/rake:32:in `<main>'

The offending method is

def have_rakefile
      @rakefiles.each do |fn|
        if File.exist?(fn)
          others = Dir.glob(fn, File::FNM_CASEFOLD)
          return others.size == 1 ? others.first : fn
        elsif fn == ''
          return fn
        end
      end
      return nil
    end

Since the stack trace was unhelpful to me I inserted a puts "#{fn} #{File::FNM_CASEFOLD}" at the beginning of the block and got this:

rakefile 8
Rakefile 8
rake aborted!
invalid byte sequence in UTF-8
/usr/local/lib/ruby/1.9.1/rake/application.rb:184:in `glob'
/usr/local/lib/ruby/1.9.1/rake/application.rb:184:in `block in have_rakefile'
/usr/local/lib/ruby/1.9.1/rake/application.rb:181:in `each'
/usr/local/lib/ruby/1.9.1/rake/application.rb:181:in `have_rakefile'
/usr/local/lib/ruby/1.9.1/rake/application.rb:469:in `find_rakefile_location'
/usr/local/lib/ruby/1.9.1/rake/application.rb:487:in `raw_load_rakefile'
/usr/local/lib/ruby/1.9.1/rake/application.rb:82:in `block in load_rakefile'
/usr/local/lib/ruby/1.9.1/rake/application.rb:133:in `standard_exception_handling'
/usr/local/lib/ruby/1.9.1/rake/application.rb:81:in `load_rakefile'
/usr/local/lib/ruby/1.9.1/rake/application.rb:65:in `block in run'
/usr/local/lib/ruby/1.9.1/rake/application.rb:133:in `standard_exception_handling'
/usr/local/lib/ruby/1.9.1/rake/application.rb:63:in `run'
/usr/local/bin/rake:32:in `<main>'

The rakefile is just the default one rails generates

# Add your own tasks in files placed in lib/tasks ending in .rake,
# for example lib/tasks/capistrano.rake, and they will automatically be available to Rake.

require File.expand_path('../config/application', __FILE__)
require 'rake/dsl_definition'
require 'rake'

MyApp::Application.load_tasks

The only task file in lib/tasks is

 desc "Resets the help files in the db by deleting all existing and rereading the yaml files"
    task :help_reset => :environment do
      HelpSystem.delete_all
      HelpSystem.seed_help
    end

I have no idea where to go next, any help is greatly appreciated.

Answer 1

Based on GeorgeMillo's idea but without the need to debug, one can do:

  def add_by_file_path(file_path)
    File.open(file_path) do |f|
      self.add_by_io(f, file_type(file_path))
    end
  rescue Exception => e
    puts "Exception raised while processing: #{file_path}: #{e.message}"
  end

Errors will be ignored and a trace with the offending file will be printed.

Answer 2

Okay, my problem was slightly different to yours, but I'll post how I solved it in case it helps a future Googler.

My issue was that I was getting the following error every time I tried to run rake stats:

rake aborted!
ArgumentError: invalid byte sequence in UTF-8
/Users/george/.rvm/gems/ruby-2.1.5/gems/railties-4.1.6/lib/rails/code_statistics_calculator.rb:61:in `=~'
/Users/george/.rvm/gems/ruby-2.1.5/gems/railties-4.1.6/lib/rails/code_statistics_calculator.rb:61:in `add_by_io'
/Users/george/.rvm/gems/ruby-2.1.5/gems/railties-4.1.6/lib/rails/code_statistics_calculator.rb:43:in `block in add_by_file_path'
... # more stacktrace

So I opened up code_statistics_calculator.rb (the file at the top of the stacktrace and changed:

def add_by_file_path(file_path)
  File.open(file_path) do |f|
    self.add_by_io(f, file_type(file_path)) # <- this line is raising the error
  end
end

to:

def add_by_file_path(file_path)
  File.open(file_path) do |f|
    begin
      self.add_by_io(f, file_type(file_path))
    rescue ArgumentError
      debugger
      puts # An extra statement is needed between 'debugger' and 'end' or debugger screws up.
    end
  end
end

Running rake stats again, I entered debugger, at which point I could see that file_path at this moment was pointing to a particular file in app/models which it couldn't parse as utf-8.

Sure enough, I opened up that file in vim and when I typed :set fileencoding? it returned latin-1. So I set it to utf-8 (set fileencoding=utf-8 then save the file) and sure enough, rake stats worked again! Voila.

(Note that in your case there could be more than one file that's not in utf-8. Also, when you're done make sure you don't forget to change code_statistics_calculator.rb back to its original form!)

Answer 3

Try saving the offending file (could be anything that rake is trying) in UTF-8 WITH BOM.