Linux: memory usage and releasing swap memory

Question

I have observed the below stats on one of my servers.

1. From the top command, only one process is using 15% of RAM and there are no others
2. There are all 0's under swap-in & swap-out columns of vmstat output

But still I am seeing that Swap and RAM both are fully occupied in "free -m" output

top output

Mem:  16413804k total, 16390264k used,    23540k free,    59604k buffers
Swap:  2040244k total,  2040244k used,        0k free,   584688k cached

PID USER        PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+     COMMAND                                                                                                                                                                         
10984 mysql     15   0 3100m 2.4g 5472 S    0 15.5   1129:44    mysqld                                                                                                                                                                          
12773 root      16   0 18440 7916 1064 S    0  0.0   65:46.67   IPremoted                                                                                                                                                                       
3108 ntp        16   0 18984 5720 4652 S    0  0.0   54:35.78   ntpd                                                                                                                                                                            
19694 root      16   0 48996 5708 3656 S    0  0.0   0:00.03    sshd                                                                                                                                                                            
11084 hpsmh     17   0  371m 3892 2532 S    0  0.0   0:00.00    hpsmhd                                                                                                                                                                          

free -m output

    total  used    free   shared   buffers   cached
Mem:16029  15983   46     0        52        546
-/+ buffers/cache:      
               15384   644
Swap:1992  1992    0

vmstat output

 swap usage
 si   so    
 0    0
 0    0
 0    0
 0    0
 0    0
 0    0
 0    0

Can some one explain this case ?

Thanks in advace.

Answer 1

Swap is not a problem here. But something using more than half of RAM is.

Swap is full with totally useless data, so you have 2 GB more of RAM to do the important stuff. Good for you! How I know swap contents are useless? Well, zero in si. No process ever wants to read this back into memory.

Now the problem is:

• 16 GB total RAM
• minus 3 GB or so used by all processes (I'm guessing you sorted top output properly)
• minus 644 MB for cache/buffers/free
• leaves over 12 gigabytes used by what? Either kernel or some slab or shared memory or some other anonymous pages. Weird. I would examine /proc/meminfo carefully to find out the cause.

Also don't look at inactive memory statistic as suggested in comments. It's useless.

Answer 2

First you find which process is utilizing the swap space while using below script.

#!/bin/bash
# Get current swap usage for all running processes
# Erik Ljungstrom 27/05/2011
SUM=0
OVERALL=0
for DIR in `find /proc/ -maxdepth 1 -type d | egrep "^/proc/[0-9]"` ; do
PID=`echo $DIR | cut -d / -f 3`
PROGNAME=`ps -p $PID -o comm --no-headers`
for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'`
do
let SUM=$SUM+$SWAP
done
echo "PID=$PID - Swap used: $SUM - ($PROGNAME )"
let OVERALL=$OVERALL+$SUM
SUM=0

done
echo "Overall swap used: $OVERALL"

then do the below swapoff -a once swap cleared then swapon -a

Thanks -Arun