CODEWITHSUNDEEP
pythonmemcachedlimit
cfischer
cfischer

Reputation: 24912

Maximum size of object that can be saved in memcached

I need to return a rather big file (11MB) to the user. For certain reasons, I can't just provide a direct url to the file (http://www.sample.com/mybigfile.exe); instead it must be accessed through code.

Instead of having to read it from disk over and over, I thought of saving it in memcached (if this is not a good idea, let me know). Everything seems to work, fine (no errors), but when I try to retrieve the file from memcached I always get None, as if the file wasn't cached.

Is there a size limit to what can be saved?

Here's the code:

def download_demo():
    """
    Returns the demo file
    """
    KEY = "xyz"
    TIME = 86400 #24 hours

    buff = memc.get(KEY)
    if not buff: 
        file = open(FILENAME, 'r')
        buff = file.read()
        memc.set(KEY, buff, TIME)

    print "Content-Type:application/x-download\nContent-Disposition:attachment;filename=%s\nContent-Length:%s\n\n%s" %    (os.path.split(FILENAME)[-1], len(buff), buff)

Upvotes: 37

Views: 77111

Answers (8)

ns15
ns15

Reputation: 8734

2022 Update - For open source Memcached,

  • the minimum value for max-item-size(-I, --max-item-size) is 1KiB
  • the maximum value is 1 GiB
  • the default value is 1 MiB

https://github.com/memcached/memcached/blob/597645db6a2b138710f01ffe5e92e453117b987a/memcached.c#L5213

if (unit == 'k' || unit == 'K')
    size_max *= 1024;
if (unit == 'm' || unit == 'M')
    size_max *= 1024 * 1024;

https://github.com/memcached/memcached/blob/597645db6a2b138710f01ffe5e92e453117b987a/memcached.c#L5730

if (settings.item_size_max > (ITEM_SIZE_MAX_UPPER_LIMIT)) {
    fprintf(stderr, "Cannot set item size limit higher than a gigabyte.\n");
    exit(EX_USAGE);
}

In case of cloud memcache variant this value will differ - For example here is the limit on google cloud memcache - gcloud docs

Note: If you are planning to increase the max-item-size value then inspect/increase the amount of memory allocated for the cache(maxbytes : this is the overall Memcached memory). use the -m option to specify the amount of RAM to be allocated (in megabytes). Default is 64MB.

Both settings are available to inspect in memcached statistics (size in bytes):

stats settings
STAT maxbytes 67108864
STAT item_size_max 1048576

Upvotes: 1

Alex
Alex

Reputation: 18522

To summarise the necessary steps:

1) Update Memcache to version 1.4.2 or later.

2) Add the flag -I 15M (or however many megabytes) to your memcache run command.

That's either command line, or in Ubuntu, add the line

-I 15M

to anywhere in /etc/memcached.conf and restart the service.

3) Add the necessary flag to the client in memcache.

import memcache
memc = memcache.Client(['localhost'], server_max_value_length=1024*1024*15)
memc.set(KEY, buff, TIME)

If you don't have direct access to the memcache client (i.e. working through a framework), then just hack the memcache code directly.

On Ubuntu, it's /usr/local/lib/python2.7/dist-packages/memcache.py. Change the line:

SERVER_MAX_ITEM_LENGTH = 1024 * 1024

to 

SERVER_MAX_ITEM_LENGTH = 1024 * 1024 * 15

Obviously, you'll need to do this hack over again if you update memcache, but it's a very simple and quick fix.

Upvotes: 24

nsanders
nsanders

Reputation: 12636

As of memcache 1.4.2, this is a user-configurable parameter: ReleaseNotes142 * memcached @ Github

Configurable maximum item size

Many people have asked for memcached to be able to store items larger than 1MB, while it's generally recommended that one not do this, it is now supported on the commandline.

A few enlightened folk have also asked for memcached to reduce the maximum item size. That is also an option.

The new -I parameter allows you to specify the maximum item size at runtime. It supports a unit postfix to allow for natural expression of item size.

Examples:

memcached -I 128k # Refuse items larger than 128k. 
memcached -I 10m  # Allow objects up to 10MB

Upvotes: 44

Pascal MARTIN
Pascal MARTIN

Reputation: 401022

There are two entries about that in the memcached FAQ :

The answer to the first one is (quoting, emphasis mine) :

The maximum size of a value you can store in memcached is 1 megabyte. If your data is larger, consider clientside compression or splitting the value up into multiple keys.

So I'm guessing your 11MB file is quite too big to fit in one memcached entry.

Increasing the size of an object is possible, as per other answers.

Upvotes: 41

Jason Smith
Jason Smith

Reputation: 469

Nobody seems to address this part of the question:

Instead of having to read it from disk over and over, I thought of saving it in memcached (if this is not a good idea, let me know).

This is not a good idea. All modern file systems have efficient caching mechanisms. Reading a file from disk that was read a few seconds earlier will typically still be in memory cache. This is much faster than accessing memcached over a network.

If you are running memcached on 127.0.0.1, the memory it claimed will only be used for that single file, wereas if you simply rely on the caching of your operating system the memory can be used for various purposes depending on the jobs at hand.

Upvotes: 1

Pari Rajaram
Pari Rajaram

Reputation: 432

If you are using python memcached client, please also make sure to increase the limit in client memcache.py file along with the memcached server.

~/anaconda/lib/python2.7/site-packages/memcache.py

SERVER_MAX_KEY_LENGTH = 250 
SERVER_MAX_VALUE_LENGTH = 1024 * 1024 * 15

you can see in memcached console ( telnet localhost 11211 )

stats slabs STAT 48:chunk_size 3677344

Upvotes: 1

The Disintegrator
The Disintegrator

Reputation: 4187

The maximum data size is 1mb per item

http://code.google.com/p/memcached/wiki/FAQ#What_is_the_maximum_data_size_you_can_store?_%281_megabyte%29

Update: This is a 2009 answer. And in that date the info was accurate and the source was official. Now in 2014 memcached can store 128mb instead of 1mb (but the developers didn't bother to update the official FAQ). The only reference someone could find id an obscure page that probably will be dead in one year.

Upvotes: 2

Eric
Eric

Reputation: 24920

Following article explained why it limit max size of a single file to 1M.

http://www.mikeperham.com/2009/06/22/slabs-pages-chunks-and-memcached/

Basicly, there are small pieces of memory pages in memcache, inside which objects reside.

And the default page size seems to be 1M, so the max object a page can contain is limited to 1M.

Even though you can config it to increase the size, but I think this might have some kind of performance trade-off.

Upvotes: 2

Related Questions