CODEWITHSUNDEEP
phpmemcached
lordstyx
lordstyx

Reputation: 785

Memcached slow gets, high CPU usage

I've got a memcached instance running on a machine to take the stress of the database. Currently there are about 350 requests per second via PHP, which should be perfectly doable according to the memcached docs, but I'm seeing ridiculously slow get() times. Averaging around 60 ms, with spikes in both ways (0.1 ms and 250 ms).

The memcached process is also using around 80% CPU all the time. It's getting really problematic as with all the gets combined it takes over 5 seconds for the page to complete.

I'm pretty sure it's the get command as I've commented out in the code and the database takes over, making the memcached process use 0 CPU.

Here are the stats:

stats
STAT pid 617
STAT uptime 855901
STAT time 1370358572
STAT version 1.4.5
STAT pointer_size 32
STAT rusage_user 15472.778988
STAT rusage_system 38712.971409
STAT curr_connections 175
STAT total_connections 4423163
STAT connection_structures 252
STAT cmd_get 319670822
STAT cmd_set 48996864
STAT cmd_flush 0
STAT get_hits 233440856
STAT get_misses 86229966
STAT delete_misses 11025386
STAT delete_hits 11131141
STAT incr_misses 27702934
STAT incr_hits 19471007
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 25484001864
STAT bytes_written 77617943971
STAT limit_maxbytes 201326592
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 0
STAT bytes 47135355
STAT curr_items 539471
STAT total_items 21293860
STAT evictions 3183365
STAT reclaimed 3222011
END

And the settings:

stats settings
STAT maxbytes 201326592
STAT maxconns 1024
STAT tcpport 11211
STAT udpport 11211
STAT inter 127.0.0.1
STAT verbosity 0
STAT oldest 0
STAT evictions on
STAT domain_socket NULL
STAT umask 700
STAT growth_factor 1.25
STAT chunk_size 48
STAT num_threads 4
STAT stat_key_prefix :
STAT detail_enabled no
STAT reqs_per_event 20
STAT cas_enabled yes
STAT tcp_backlog 1024
STAT binding_protocol auto-negotiate
STAT auth_enabled_sasl no
STAT item_size_max 1048576
END

Now did I configure memcached wrong? Or is there something else going on?

EDIT:

On request, here's the code with the get (there's not much to it):

function getItem($memcached, $key, $id) {
    $md5key = md5($key.":".$id);
    $v = $memcached->get($md5key); // changing this to $v = false made the memcached CPU usage go to 0
    if ($v === false) {
        //code that fetches the correct data for each key, stores it in memcached and in $v.
    }
    return $v;
}

And in the main script there's the following:

$memcached = new Memcached;
$memcached->addServer('localhost', 11211) or die ("Could not connect to memcached server");
$memcached->setOption(Memcached::OPT_COMPRESSION, false);

$myItem = getItem($memcached, "key", "123");

EDIT2: For some reason I'm still noticing a high load on the database. Now when I manually check for the data to be present in the cache via telnet it's just fine. Could it be that the memcached client thinks the connection to the cache is timing out and hence goed to the database? That would leave me with the question why on earth the connection is timing out...

Upvotes: 4

Views: 13170

Answers (1)

lordstyx
lordstyx

Reputation: 785

Well, I've found the problem! To get an idea of the requests per second I used the memcache.php file that's available out there. It told me that there were 350 requests per second. The thing is that there has been quite an increase of use in the past few days, and the requests/second is really just an average over the entire uptime. Calculated by (hits+missed)/uptime. Now after restarting memcached this average returns more correct values and there are actually 4000 requests per second.

tl;dr: Wrong stats in first post. Correct stats are: 4000 requests/second.

I suppose my hardware simply can't cope with that.

Upvotes: 1

Related Questions