Random session data loss in PHP

Question

Here is the problem we have been facing for the past few weeks.

1/ Our setup

• PHP 5.4 + MySQL
• 2 dedicated servers, load-balanced
• Sessions are replicated between the 2 servers using memcached
• 3 applications running on these servers :
  • One custom-developped application, using default php session settings
  • Another custom-developped application, using different session settings (cookie name, path)
  • One Wordpress CMS

2/ The problem

The problem occurs on our first application.

Some of our users reported that they sometimes get disconnected after a few minutes (when the session is setup to last 3 hours). It can happen to them several time in the same day, then no disconnection for a few days, but the problem always comes back. So far the fraction of users impacted is small, but I would like to solve this before it "spreads" to other users.

The problem seems to occur in different places of the application, though we have identified 3 scenarii where most of the errors occur :

• Some involve submitting a form ($_SESSION variable is modified)
• Other simply involve opening a popup page, with no modification of the session data

We have tried to reproduce the different scenarii described by the users : sometimes we have been able to, but most of the time we don't have any problem, which makes it hard to debug.

Other notes :

• The problem is recent, this application had been running for years without any problem.
• It doesn't seem to be related to our server load, because the problem still occured during the summer break when our trafic was low
• It only affects one session/users at a time: all the other users logged in at the same time don't experience this problem
• The problem occured on all the different browsers (IE, Firefox, Chrome)

3/ Technical analysis

When a disconnect occurs, the user is redirected to a page "Your session has expired or you don't have the right to view". When this page is loaded, we get a technical email with a dump of the $_SESSION variable.

When a session expires the normal way, the email we get shows that the $_SESSION variable is empty (normal behavior). When an unexpected disconnect occurs, what is interesting is that the $_SESSION is not entirely empty : out of the ~20 elements the array contained, only one is left (always the same).

So this would mean the session is not expired, but not enough data is left to "identify" the user, hence the "no rights" page displayed. As a confirmation when this occurs, we can check in memcached that this session still holds some data.

These are the potential problem causes we have identified so far, and what we have done to rule them out :

• Memcached indicates between 70 et 80% freespace, so we don't think it is the problem.
• We removed Memcached and went back to using a NFS shared directory for session files: the problem actually got worse. This would point to an applicative bug, because NFS being slower to write data, session loss would occur more often.
• We have browsed all the different forums (including SO) talking about PHP session data loss, and reviewed our code accordingly. The code base is big, but we have used automated tools and scripts to avoid missing a file.
  • session_start() is called at the beginning of each page.
  • exit() is called after each header("Location...")
  • register_globals is Off
• We have tested the possible interractions between our 2 other applications and the problematic one, though they don't share any code, database or session handling. Nothing identified there.
• We have analyzed our access logs around the times of the disconnections, to check for behavior patterns : no luck here either.

So we have no idea what causes this problem, as it seems to occur randomly, so my questions are :

• The problem could come from our code: did we miss anything to check ? This solutions seems unlikely as the code works most of the time for all our users, but I am still considering it.
• The problem could come from another application/process that would "empty" part of the session variable array. We have also reviewed the code from the other applications, but didn't find anything that could cause this. And if another process is doing this, why would it only empty some sessions and not all of them ?

Thanks for your help.

Answer 1

It could be as simple as a WordPress plugin that uses sessions and calls either session_name() or session_id() with a different value, overlapping your custom applications with default session settings.

Since WordPress itself does not use sessions, plugins are often written from the perspective of having free rein with sessions. I just did a search on a WordPress test site and found sessions used in a gallery plugin, a plugin for putting a background image on the page, a shopping cart plugin, and a plugin I was writing that needed to carry an uploaded file from one admin page to another.

Answer 2

If this problem "suddenly" occurred, check what has changed. Did you do any work on the application? If so check committed code (you talked about automated tools so I expect there to be a repository which would allow for accurate finding of code changes). Did you change anything on the server? Like upgrade software, upgrade/change hardware, make changes to the other two applications ? One thing that popped to mind, did you check the drives you use for caching? It could be a corrupted part of the file system. Which would explain the random user part.

I couple of things I always to is:

• Try to determine the moment of first occurrence as accurate as possible. At my work this occasionally triggers someone saying "oh yeah that might have to do with when I changed/updated/created this or that" so this might help. On the other hand it can sometimes takes days, weeks or more before something gets noticed so start expanding that time-frame if nothing comes up.
• You have already a couple of scenario, find the common factor in these. If they don't share any code, stop looking there. If they DO share code search there. Of course sharing (part of) it here might allow us to help you search.
• Do an organised search. I usually do the main application check when I am the one working most on the application (or even better when I created it). A colleague will check surrounding applications that might have influence on it. In your case those 2 other applications. Finally our sysadmin will check for newly installed or updated software on the server(s) and he will also check with our network guys if anything changed hardware wise or network related (for other people this could be the hosting provider).

Answer 3

I don't think you'll get a definitive answer to your question. There are too many probable causes and you haven't shown any code.

Still, my guess is that you have memcached.sess_locking turned Off, or if you have a custom session implementation - that it doesn't implement locking at all.
Eventually, this leads to a race condition between two simultaneous HTTP requests.

My guess is based on the often seen bad advice to turn off locks or free them as soon as possible, in order to achieve higher performance.