CODEWITHSUNDEEP
pythonmongodbubuntu
Shuai Zhang
Shuai Zhang

Reputation: 2061

MemoryError in MongoDB

I use the python library 'mongoengine' to query MongoDB. Here is the code:

mblog_list = MicroBlog.objects(omid__exists=False) # MicroBlog is a class I defined using mongoengine
data = []
for mblog in mblog_list:
    weibo_created = mblog.created

Note there about 30000 items in mblog_list and it uses about 2.5 G disk space. In the above code, the for loop runs several times and raises the "MemoryError".

The MongoDB server shows no error msg but says one connections is closed.

I also use the official js interpreter to issue the same query and it works fine.

Run the 'ulimit -a' command and I get:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31723
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31723
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

MongoDB version: 2.4.12

OS: Ubuntu 12.04

Upvotes: 0

Views: 847

Answers (1)

Shuai Zhang
Shuai Zhang

Reputation: 2061

It is not a MongoDB problem but the library mongoengine's.

By default, when iterating the returned queryset, mongoengine keeps all the returned doc in memory for reuse in future!

So changing the code like this will fix this problem:

mblog_list = MicroBlog.objects(omid__exists=False).non_cache() # this should be the default behavior

Thanks!

Upvotes: 1

Related Questions