mod_perl "use" randomly fails

Question

We have a web interface for our software using perl and Apache with mod_perl. On a recent Ubuntu 14 installation (Apache 2.4.7, perl version 5.18.2) we've had problems with it randomly stopping working with the error below. This will happen after a random amount of time (eg a few hours or days), has not happened on any previous Ubuntu or CentOS installation, and we've only been able to temporarily resolve it by restarting Apache.

The difficulty in debugging it stems from the fact that it runs fine for a while, processing hundreds or thousands of requests, and we've not been able to identify any particular trigger for it to stop working.

Would anyone have ideas on how to debug and resolve it? Thank you.

The following is the error message. It is repeated with every web request until Apache is restarted. The Utils.pm mentioned is part of our software, and is "use"d in line 2 of index.pl. Utils.pm itself "use"s quite a few other modules.

[Sun Jul 27 19:26:18.110765 2014] [:error] [pid 26316:tid 139927794730752] Attempt to reload Utils.pm aborted.\nCompilation failed in require at /path/to/index.pl line 2.\nBEGIN failed--compilation aborted at /path/to/index.pl line 2.\n

Answer 1

The problem seems to be resolved by disabling Apache2::Reload with the following in apache2.conf:

PerlInitHandler Apache2::Reload
PerlSetVar ReloadAll Off

Answer 2

I approach these problems through process of elimination. If a deployment isn't fully successful, I want to know how it differs from fully successful deployments.

I would begin by examining the versions of Perl dependencies. Different distributions might include different versions of non-core modules. If your deployment procedure includes pulling dependencies from CPAN, you may have newer versions of modules on the recent deploy than on previous deploys.

If I found that the dependencies differed, I'd deploy the same versions from an acceptable deployment to the new deployment. If it fixed the problem, I'd know that I have to refine the process for future deployments. If it didn't fix the problem, I'd move on to other identifiable differences between acceptable and unacceptable deployments. "Identifiable" depends on what resources I have at my disposal. I'm an almost-adequate system administrator, so I'd be likely to consult with a colleague to help me identify inconsistencies at that level.

We know that there are environments where the software works as expected. We might not be able to isolate the precise root-cause, but it is reasonable to expect that we can make the environment more hospitable to the application.