Thinking Sphinx delta indexing fails in production

Question

Here's what I've determined:

1. Delta indexing works fine in development
2. Delta indexing does not work when I push to the production server, and no action is logged in searchd.log
3. I'm running Phusion Passenger, and, as recommended in the basic troubleshooting guide, have confirmed that:
   1. www-data has permission to run indexing rake tasks (ran them from command line manually)
   2. the path to indexer and searchd are correct (/usr/local/bin)
   3. there are no errors in production.log

What on earth could I possibly be missing? I'm running Ruby Enterprise 1.8.6, Rails 2.3.4, Sphinx 0.9.8.1, and Thinking Sphinx 1.2.11.

Thanks!

Answer 1

As above, same issues were faced our side on two machine. The first one we had an issue with mysql which showed in apache2 log. Only seemed to affect local OSX machine..

Second time when we deployed to Ubuntu server, we had same issue. Rails c production was fine, no errors, bla bla bla.

Ended up being a permissions problem. Couldn't figure this out as there were no problems starting, although I guess I was doing so as root.

Using capistrano and passenger, we did this:

1. Create a passenger user and added to www-data group
2. Changed user in deploy.rb to be passenger
3. Manually changed all the /current files to be owned by above
4. Logged in as passenger user.
5. Ran rake ts:rebuild RAILS_ENV="production"

Worked a treat for us...

Good luck

Answer 2

I had the same problem. Works on the command line, not inthe app.

Turns out that we still had a slave database that we were using for the indexing, but the slave wasn't getting updated.

Answer 3

I was having this problem and found the "bin_path" solution mentioned above. When it didn't seem to work, it took me a while to realize that I'd pasted in the example code for "production" when I was testing on "stagiing" environment. Problem solved!

This was after making sure that the rake tasks that config, index, and start sphinx are all running as the same user as your passenger instance. If you log into the server as root to run these tasks, they will work in the console but not via passenger.

Answer 4

Last night as I slept it hit me. Unsurprisingly, it was a stupid issue involving bad configuration, though I am rather surprised that it produced the results it did. I guess I don't know much about Thinking Sphinx internals.

Recently I migrated servers. sphinx.yml looked like this:

production:
  bin_path: '/usr/local/bin'
  host: mysql.mysite.com

On the new server, MySQL was just a local service, but I had forgotten to remove that line. Interestingly, manual rake reindexing still worked just fine. I'm intrigued that Thinking Sphinx didn't throw an error when trying to reload the deltas, since mysql.mysite.com no longer exists, even though that was clearly the source of the issue.

Thanks for all your help, and sorry to have brought up such a silly problem.

Answer 5

Here's the next troubleshooting step I would take. Open up the file for the delta indexing strategy you are using (presumably lib/thinking_sphinx/deltas/default_delta.rb). Find the line where it actually generates the indexing command. In mine (v1.1.6) it's line 20:

output = `#{config.bin_path}indexer --config #{config.config_file} #{rotate} #{delta_index_name model}`

Change that so you can log the command itself, and maybe log the output as well:

command = `#{config.bin_path}indexer --config #{config.config_file} #{rotate} #{delta_index_name model}`
RAILS_DEFAULT_LOGGER.info(command)
output = `#{command}`
RAILS_DEFAULT_LOGGER.info(output)

Deploy that to production and tail the log while modifying a delta-indexed model. Hopefully that will actually show you the problem. Of course maybe the problem is elsewhere in the code and you won't even get to this point, but this is where I would start.

Answer 6

Are there any clues in Apache/Nginx's error log?