CODEWITHSUNDEEP
phpsessionamazon-web-servicesmemcachedamazon-elasticache
Osushi
Osushi

Reputation: 107

Fixed: PHP Session Store Error Using Memcached

The error message displays on my server:

$ sudo tail /var/log/php-fpm/5.5/www-error.log
PHP Warning:  session_start(): Unable to clear session lock record

When my server stores session, it uses the memcached on ElastiCache(AWS).
Here is my environments:

*Memcached(ElastiCache)
-version 1.4.14
-1 Node
-cache.t2.micro

$ nginx -v
nginx version: nginx/1.6.2
# use unix domain socket  
(fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;)

$ php-fpm -v
PHP 5.5.24 (fpm-fcgi) (built: Apr 17 2015 18:04:41)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2015 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2015, by Zend Technologies

$ pecl list
Installed packages, channel pecl.php.net:
=========================================
Package   Version State
...
memcached 2.2.0   stable
...

Here is data:

$ cat /etc/php-5.5.ini
...
...
; http://php.net/session.save-handler
; session.save_handler = files
session.hash_bits_per_character = 5
session.hash_function = 0
session.use_trans_sid = 0
session.cache_expire = 180
session.cache_limiter = nocache
session.referer_check =
session.bug_compat_warn = Off
session.bug_compat_42 = Off
session.gc_maxlifetime = 1440
session.gc_divisor = 1000
session.gc_probability = 1
session.serialize_handler = php
session.cookie_httponly =
session.cookie_path = /
session.cookie_lifetime = 0
session.auto_start = 0
session.name = PHPSESSID
session.use_only_cookies = 1
session.use_cookies = 1
...
...

$ cat /etc/php-5.5.d/memcached.ini
extension=memcached.so

session.save_handler=memcached
session.save_path="*******.cache.amazonaws.com:11211"
memcached.sess_locking = On
memcached.sess_lock_wait = 150000
memcached.sess_lock_max_wait = 0;
memcached.sess_lock_expire = 0;
memcached.sess_prefix = "memc.sess.key."
memcached.sess_consistent_hash = Off
memcached.sess_remove_failed = 0
memcached.sess_number_of_replicas = 0
memcached.sess_binary = Off
memcached.sess_randomize_replica_read = Off
memcached.sess_connect_timeout = 1000
memcached.sess_sasl_username = NULL
memcached.sess_sasl_password = NULL
memcached.compression_type = "fastlz"
memcached.compression_factor = "1.3"
memcached.compression_threshold = 2000
memcached.serializer = "igbinary"
memcached.use_sasl = Off
memcached.store_retry_count = 2

$ php -i | grep memcached
/etc/php-5.5.d/memcached.ini,
memcached
memcached support => enabled
libmemcached version => 1.0.8
memcached.compression_factor => 1.3 => 1.3
memcached.compression_threshold => 2000 => 2000
memcached.compression_type => fastlz => fastlz
memcached.serializer => igbinary => igbinary
memcached.sess_binary => no value => no value
memcached.sess_connect_timeout => 1000 => 1000
memcached.sess_consistent_hash => no value => no value
memcached.sess_lock_expire => 0 => 0
memcached.sess_lock_max_wait => 0 => 0
memcached.sess_lock_wait => 150000 => 150000
memcached.sess_locking => no value => no value
memcached.sess_number_of_replicas => 0 => 0
memcached.sess_prefix => memc.sess.key. => memc.sess.key.
memcached.sess_randomize_replica_read => no value => no value
memcached.sess_remove_failed => 0 => 0
memcached.sess_sasl_password => no value => no value
memcached.sess_sasl_username => no value => no value
memcached.store_retry_count => 2 => 2
memcached.use_sasl => no value => no value
Registered save handlers => files user memcached
session.save_handler => memcached => memcached

$ yum list installed | grep memcached
libmemcached.x86_64                  1.0.8-2.6.amzn1               @amzn-main
memcached.x86_64                     1.4.15-9.12.amzn1             @amzn-main
php55-pecl-memcached.x86_64          2.2.0-5.14.amzn1              @amzn-updates

$ cat /etc/php-fpm-5.5.d/www.conf
php_value[session.save_handler] = memcached
php_value[session.save_path] = "*******.cache.amazonaws.com:11211"

$ memcached-tool *******.cache.amazonaws.com:11211 display
  #  Item_Size  Max_age   Pages   Count   Full?  Evicted Evict_Time OOM
  1      96B         0s       1       0     yes        0        0    0
  2     120B     55039s       1       4     yes        0        0    0
  4     192B      1439s       2    2226     yes        0        0    0
  5     240B         0s       1       0     yes        0        0    0
  6     304B     14477s       1       1     yes        0        0    0
  7     384B      9557s       1       1     yes        0        0    0
  8     480B       268s       1       1     yes        0        0    0
  9     600B      2014s       1       1     yes        0        0    0

$ memcached-tool *******.cache.amazonaws.com:11211 stats
         accepting_conns           1
               auth_cmds           0
             auth_errors           0
                   bytes      470256
              bytes_read  5006739226
           bytes_written  2668634300
              cas_badval           0
                cas_hits           0
              cas_misses           0
          cmd_config_get      489750
          cmd_config_set           1
               cmd_flush           1
                 cmd_get    15601270
                 cmd_set    33168188
               cmd_touch           0
             conn_yields           0
   connection_structures         260
             curr_config           1
        curr_connections           9
              curr_items        2581
               decr_hits           0
             decr_misses           0
             delete_hits    15438339
           delete_misses       10047
       evicted_unfetched           0
               evictions           0
       expired_unfetched     7474894
                get_hits     8113028
              get_misses     7488242
              hash_bytes      524288
       hash_is_expanding           0
        hash_power_level          16
               incr_hits           0
             incr_misses           0
                libevent 1.4.13-stable
          limit_maxbytes   477102080
     listen_disabled_num           0
                     pid           1
            pointer_size          64
               reclaimed     7476127
            reserved_fds           5
           rusage_system 1539.028000
             rusage_user  838.464000
                 threads           1
                    time  1436772530
       total_connections    15600239
             total_items    31043112
              touch_hits           0
            touch_misses           0
                  uptime     7346275
                 version      1.4.14

Strangely, this error happens about once time per weekly, but yesterday, it happens about 2000 times.
When this error is happens, the server also happens busy status.
It disappears over time.

■Cloud Watch Log: cloud_watch

■New Relic Application Log: new_relic

I don't know what it's happening. If you have good solution, please let me know.

Update at 2020/10/8

It was long time, I solved this issue:

When user access came many, it seems that the DNS can't solve on AWS EC2.

https://aws.amazon.com/jp/premiumsupport/knowledge-center/dns-resolution-failures-ec2-linux/

Here is fixed patch.

$ wget https://raw.githubusercontent.com/awslabs/aws-support-tools/master/EC2/AutomateDnsmasq/AutomateDnsmasq.sh
$ chmod +x AutomateDnsmasq.sh
$ sudo ./AutomateDnsmasq.sh
-> Finally, you must restart php server.

Upvotes: 4

Views: 13758

Answers (2)

terry zhang
terry zhang

Reputation: 309

I am using below config fixed the problem.

[session]
session.lazy_write = 0

[memcached]
memcached.sess_locking = On
memcached.sess_lock_wait = 150000
memcached.sess_lock_max_wait = 30;
memcached.sess_lock_expire = 0;
memcached.sess_consistent_hash = Off
memcached.sess_remove_failed = 1
memcached.sess_number_of_replicas = 0
memcached.sess_binary = Off
memcached.sess_randomize_replica_read = Off
memcached.sess_connect_timeout = 1000
memcached.sess_sasl_username = NULL
memcached.sess_sasl_password = NULL
memcached.compression_type = "fastlz"
memcached.compression_factor = "1.3"
memcached.compression_threshold = 2000
memcached.serializer = "igbinary"
memcached.use_sasl = Off
memcached.store_retry_count = 2

Upvotes: 1

Will
Will

Reputation: 24689

I believe the solution here is to call session_write_close() as soon as you're done writing data to the session. It will still be available for reading, but no more writes can be made. This explains it a bit.

Long-running requests will cause the biggest problem. PHP won't close/unlock the session until the request completes if session_write_close() is never called.

You can also try implementing locking manually, locking only the fields you need to lock. There have been numerous bugs and issues with session locking via Memcache over the yeard.

Upvotes: 3

Related Questions