Passenger on Apache: "doesn't seem to be running", app page gives 403; what next?

Question

I'm having trouble getting my web server to talk to an app in Ruby on Rails. This is especially frustrating because I had it working last year for several months; I assume that some setting changed in the course of an upgrade in the meantime, but I can't find it.

The symptom: when I navigate to the page that should be serving the app (a sub-URI of the main domain), Apache gives a 403 error: "There is either no index document or the directory is read-protected.". The bottom of this error page gives the config string "Apache/2.4.9 (Unix) OpenSSL/1.0.1g Phusion_Passenger/4.0.37", which seems to indicate that Passenger is loaded and running. If I navigate directly to the public directory, or to the physical directory where the app lives, Apache happily displays the file listing of that directory (and if I click on any of the files, they display), so it doesn't seem to be that kind of permissions problem, at least.

In the httpd.conf file, there were the following lines to load Passenger:

LoadModule passenger_module modules/mod_passenger.so
PassengerRoot /usr/local/rvm/gems/ruby-2.0.0-p247/gems/passenger-4.0.14
PassengerDefaultRuby /usr/local/rvm/wrappers/ruby-2.0.0-p247/ruby
PassengerEnabled Off
PassengerLogLevel 3

and, later, the following lines to tell Apache about the app:

<Location "/pathtoapp/f13">
  Options +Indexes +ExecCGI +FollowSymLinks -MultiViews
  Order allow,deny
  Require all granted

  PassengerEnabled On
  RackBaseURI /pathtoapp/f13
  #RailsEnv production
  RailsEnv development
</Location>

The app itself lives in /pathtoapp/f13app, and there was a symlink in the filesystem from /pathtoapp/f13 to /pathtoapp/f13app/public. If I created a symlink /pathtoapp/test to /pathtoapp/f13app/public, it worked fine (in that it showed a file listing for the public directory, so again, no permissions problems).

I tried changing around a few things based on trawling the net for possible solutions. Most didn't appear to affect anything, so I reverted them; I did note that changing Location to Directory meant that navigating to /pathtoapp/f13 in my browser gave me the file listing of the public directory.

I tried (re-)running sudo passenger-install-apache2-module even though the Apache error seemed to indicate passenger was already installed. It (re-?)installed and prompted some changes to httpd.conf, which now has:

LoadModule passenger_module /usr/lib/ruby/gems/2.1.0/gems/passenger-4.0.37/buildout/apache2/mod_passenger.so
<IfModule mod_passenger.c>
  PassengerRoot /usr/lib/ruby/gems/2.1.0/gems/passenger-4.0.37
  PassengerDefaultRuby /usr/bin/ruby
</IfModule>

(I'm not sure if passenger did the ruby upgrade into /usr/lib or if that came in some earlier system update I'd done, but the indicated files are indeed there.) Later in httpd.conf, based on the current text of http://www.modrails.com/documentation/Users%20guide%20Apache.html#deploying_rack_to_sub_uri , I now have

Alias /pathtoapp/f13 /pathtoapp/f13app/public
<Location "/pathtoapp/f13">
  PassengerBaseURI /pathtoapp/f13
  PassengerAppRoot /pathtoapp/f13app
  PassengerEnabled On

  RailsEnv development
</Location>
<Directory "/pathtoapp/f13app/public">
  Options +Indexes +ExecCGI +FollowSymLinks -MultiViews
  Allow from all
  Require all granted
</Directory>

I then removed the in-filesystem symlink from f13 to f13app/public. And, of course, I've restarted httpd.

passenger-status still reports "ERROR: Phusion Passenger doesn't seem to be running.", and navigating to the app's page (/pathtoapp/f13) still gives the same 403 error. I'm now completely stuck as to what to try next, or even what diagnostic tool I should be using to help triage the problem. Any ideas?

(Extra system info: uname -r is "3.14.1-1-ARCH"; apachectl -v is "Apache/2.4.9 (Unix)"; passenger -v is "Phusion Passenger version 4.0.37")

Answer 1

After a lot of mucking about, I finally discovered that the problem was with the Apache configuration itself: Apache was never getting far enough in to even talk to Passenger. Specifically, on the lines

Alias /pathtoapp/f13 /pathtoapp/f13app/public

and

PassengerAppRoot /pathtoapp/f13app

it's important that the pathtoapp be relative to the filesystem, not the URI. So those two lines should really be more like

Alias /pathtoapp/f13 /srv/http/htdoc/pathtoapp/f13app/public

and

PassengerAppRoot /srv/http/htdoc/pathtoapp/f13app

though of course the exact details will vary by system. I'm not sure if this was an issue with Apache 2.2-2.4 upgrade or something else (I have root access but am not the primary sysadmin on this system), but now that I've found the problem, I didn't want to leave the question hanging. I'll also add that the Apache config line

LogLevel debug

was invaluable in helping me track down which system component the problem was in.