CODEWITHSUNDEEP
ruby-on-railsdatabasepostgresql
Topher Hunt
Topher Hunt

Reputation: 4804

Rails app: ArgumentError (invalid byte sequence in UTF-8) when connecting to production database

I'm setting up a small Rails intranet app, and running into a confusing bug during deploy. The staging app deploys fine, and I can hit various API endpoints which display basic information about the contents of our Postgres database (a COUNT on a certain table, etc.)

But when I deploy the same app to production, on the first attempt to run any database query, it triggers a 500 response with this exception logged:

ArgumentError (invalid byte sequence in UTF-8)

I assume this error is telling me that there's a problem connecting to the database, or maybe a problem parsing the data returned by a database query, but no stacktrace is shown and I'm having trouble figuring out what's wrong. I've double- and triple-checked the connection settings in database.yml, which (in both staging and production) sets the database URL which is fetched from a a Rails encrypted secret.

The database URL is in the standard format, same as what I'm using on several other Rails apps: postgres://username:password@hostname:5432/dbname

I can use psql to directly connect to either of these database URLs with no problem. The problem doesn't lie with the encrypted secrets file, because it persists even when I hard-code the database URL into database.yml.

Where should I go from here? What are best practices for figuring out the source of this error?

Upvotes: 1

Views: 251

Answers (1)

Topher Hunt
Topher Hunt

Reputation: 4804

After some flailing around, I figured out that the production DB password contains special characters like % and #, which must be escaped in order for Rails to correctly parse the database URL. The staging DB password does not contain such characters, so I could include the password in the database URL without any escaping. But the production password does, so I needed to escape it and put that value into the DB URL, rather than the raw password.

URI.encode("abc123#%")
=> "abc123%23%25"
database_url = "postgres://username:abc123%23%25@hostname:5432/dbname"

When Rails attempts to parse the connection credentials in database.yml, if it runs into any such unescaped characters, it might give you one of several unhelpfully low-level exceptions such as ArgumentError (invalid byte sequence in UTF-8), often without a stacktrace logged, so you may need to manually catch this error and print out the stacktrace in order to confirm that it is indeed an issue with parsing the connection URL.

The lesson I've learned: Anytime you use postgres:// URLs, be sure to escape your DB passwords with URI.escape()! And if you hit weird string-parsing-related errors when your app first attempts to connect to the database, double-check your postgres:// URLs.

Upvotes: 2

Related Questions