CODEWITHSUNDEEP
linuxmemorymemory-managementmemory-leakspmap
Satish
Satish

Reputation: 17457

pmap high memory usage by python script

I have a python script base daemon (listen on network for client connection) which is using lots of memory, its 23G almost. following is my pmap output:

[root@example ~]# pmap -x 9766 | grep anon
0000000001f64000   31140   31140   31140 rw---    [ anon ]
0000000003dcd000 24265388 24265220 24265220 rw---    [ anon ]
000000308d875000      16      12      12 rw---    [ anon ]
000000308e409000     184       0       0 rw---    [ anon ]
0000003659a18000       8       0       0 rw---    [ anon ]
0000003989021000       4       4       4 rw---    [ anon ]
000000398998f000      20      16      16 rw---    [ anon ]
0000003989c19000      16       4       4 rw---    [ anon ]
000000398d017000       8       0       0 rw---    [ anon ]
0000003e49c1e000       4       4       4 rw---    [ anon ]
0000003e4a1df000      16      16      16 rw---    [ anon ]
0000003e4a585000       8       0       0 rw---    [ anon ]
0000003e4d02b000       4       0       0 rw---    [ anon ]
00007fc7d8000000     132       8       8 rw---    [ anon ]
00007fc7d8021000   65404       0       0 -----    [ anon ]
00007fc7e0000000     132       8       8 rw---    [ anon ]
00007fc7e0021000   65404       0       0 -----    [ anon ]
00007fc7e6bfe000       4       0       0 -----    [ anon ]
00007fc7e6bff000   10240      12      12 rw---    [ anon ]
00007fc7e75ff000       4       0       0 -----    [ anon ]
00007fc7e7600000   10240    2048    2048 rw---    [ anon ]
00007fc7e8000000     132       8       8 rw---    [ anon ]
00007fc7e8021000   65404       0       0 -----    [ anon ]
00007fc7ec000000     132       8       8 rw---    [ anon ]
00007fc7ec021000   65404       0       0 -----    [ anon ]
00007fc7f0000000     132       8       8 rw---    [ anon ]
00007fc7f0021000   65404       0       0 -----    [ anon ]
00007fc7f4179000    3076    3076    3076 rw---    [ anon ]
00007fc7f44b8000    1452    1440    1440 rw---    [ anon ]
00007fc7f466a000     908     884     884 rw---    [ anon ]
00007fc7f47f1000       4       0       0 -----    [ anon ]
00007fc7f47f2000   10240      12      12 rw---    [ anon ]
00007fc7f51f2000       4       0       0 -----    [ anon ]
00007fc7f51f3000   10240      12      12 rw---    [ anon ]
00007fc7f5bf3000       4       0       0 -----    [ anon ]
00007fc7f5bf4000   10240      12      12 rw---    [ anon ]
00007fc7f8503000     260     256     256 rw---    [ anon ]
00007fc7f8b56000     520     512     512 rw---    [ anon ]
00007fc7f8ddb000     260     256     256 rw---    [ anon ]
00007fc7f903d000     260     256     256 rw---    [ anon ]
00007fc7f9495000     520     512     512 rw---    [ anon ]
00007fc7f972c000    1292    1284    1284 rw---    [ anon ]
00007fc7fa08f000     260     256     256 rw---    [ anon ]
00007fc7fa4d9000     260     256     256 rw---    [ anon ]
00007fc7fb360000     260     256     256 rw---    [ anon ]
00007fc7fbfd6000     260     256     256 rw---    [ anon ]
00007fc801ea8000     520     512     512 rw---    [ anon ]
00007fc801f5c000     540     532     532 rw---    [ anon ]
00007fc8023ae000      60      60      60 rw---    [ anon ]
00007fc8023c5000       4       4       4 rw---    [ anon ]
00007fc8023c6000       4       4       4 rwx--    [ anon ]
00007fc8023c7000       4       4       4 rw---    [ anon ]
00007fffd45ff000       4       4       0 r-x--    [ anon ]
ffffffffff600000       4       0       0 r-x--    [ anon ]

I have notice it is growing, thank my system has 64G memory so it is still surviving but i am afraid it will crash once it reach to max point.

0000000003dcd000 24265388 24265220 24265220 rw---    [ anon ]

is above output looks normal? I am not expert but i need suggestion to know what is going on, How do i clean dirty memories?

following memory usage:

[root@example ~]# free -m
             total       used       free     shared    buffers     cached
Mem:         64389      46304      18085         22        242      12892
-/+ buffers/cache:      33170      31219
Swap:         1027          0       1027

Upvotes: 2

Views: 3132

Answers (1)

Burak Cetin
Burak Cetin

Reputation: 62

Looking at this it looks like most of your usage is here:

0000000003dcd000 24265388 24265220 24265220 rw--- [ anon ]

This is likely Python run-time's heap. It's impossible to guess what's in it by looking at pmap output in this case.

I recommend you attach the gdb debugger to this process. This gives you direct access to the process memory, and dump this segment to a file to examine it.

First you need to find the memory offset start and end (pmap doesn't give you that as far as I can see). You can dump smaps for the process for this:

cat /procs/9766/smaps

Find the corresponding block that starts at 0000000003dcd000. Should read something like:

03dcd000 -070be000 rw-p 00000000 00:00 0 Size: 60 kB Rss: 60 kB Pss: 51 kB Shared_Clean: 0 kB Shared_Dirty: 12 kB Private_Clean: 0 kB Private_Dirty: 48 kB Referenced: 60 kB Anonymous: 60 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB

Here the start/end of the data segment is 0x03dcd000 / 0x070be000

Attach the debugger:

gdb -p 9766

Then dump this memory to a file:

dump binary memory /root/swap_problem_raw 0x03dcd000 0x070be000

You can then view this file in a hex editor to understand what's going on. It should hopefully give you a clue what kind of data is being passed around if/when it's clear text. I currently use vim and works ok.

Summary information for /proc/$id/smaps can be found in proc man page.

Upvotes: 5

Related Questions