CODEWITHSUNDEEP
filesystemsmountdiskspacedu
AKS
AKS

Reputation: 17306

df -kvh not showing the free space - Where is the remaining space

On one of Linux development server that I use, I'm seeing 100% space usage at root "/" level. Root folder has a folder called "production" which is on a different mount / filer than the filer where root is. Root has close to 7.7/8 GB disk space.

I'm doing a search "du -chs" on all folder which are in MB or GB and also finding the total space used at Root "/" (excluding /production) but I'm not getting where the other space is and why it's 100% used when all folders (excluding /production) is showing that, it's only 2.8GB in total.

  1. How can I get the missing space back?

  2. Why df -kvh and du command doesn't reflect all the space getting used / not used correctly (when we say 100% used and du saying 2.8GB at Root "/" level)? Where is the approximate (7.7 allocated - 2.8 used) GB space?

Any idea! what I may be missing here.

[root@linux-server /]# df -kvh / && echo && echo && df -kvh /production
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-root
                      7.7G  7.3G     0 100% /


Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-production
                      258G  231G   14G  95% /production
[root@linux-server /]#


[root@linux-server /]# ls -1|grep -v "production" | xargs du -chs 2>/dev/null|egrep "total|[0-9][0-9]*[MG]"
7.8M    bin
20M     boot
106M    etc
320M    lib
25M     lib64
60M     root
39M     sbin
1.9G    usr
364M    var
2.8G    total
[root@linux-server /]#

Upvotes: 2

Views: 4854

Answers (2)

Mani
Mani

Reputation: 441

Technically, The file descriptor is used by some running program. Once the file is closed by the application or when the applications comes to an end, the free space will be reflected correctly.

Upvotes: 0

AKS
AKS

Reputation: 17306

OK. I don't know who down voted this question but it was important for me to find what was causing this behavior. I would like to share the answer with the community.

So, even though there was close to 5 GB free space at root "/", df command was not showing it, may be because some OS (bug / whatever by design).

It seems like if you kill a process and if it leaves some child process(es) behind OR if you kill a process which is using a folder and if you delete that folder (while the process is in progress), then this issue may occur.

Highlevel, if system does not somehow kill a child process/pid (which is not-required) process, your space won't be reflected correctly by "df" commmand (as that child/process still somehow holds the resources).

Now, how I resolved.

I saw there were some Jenkins (java) processes running, I killed them. That still didn't get my space back (I saw only 420KB free), then I did "ps -eAf" and saw the following process running.

c123456  23019     1 99 Oct21 ?        3-07:57:35 readelf -Wa /tmp/Goslr0qbOe/content2/jdk16-solaris-1.6.0.3.tar/jdk1.6.0_03/jre/lib/sparc/libioser12.so

(This above process was initiated by some utility, may be "pkgdiff" or some child step/process initiated by pkgdiff utility. One of my team member was comparing JDK 1.6 vs JDK 1.7 tar files and somehow this above process was still running even after I killed the main pkgdiff / related PIDs and also deleted /tmp/Goslr0qb0e two days back).

As soon as I killed the above PID (23019)... shenzi! I'm back in business.

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-root
                      7.7G  2.5G  4.9G  34% /

Even after I restarted Jenkins instance(s), df now correctly shows it's 4.9G free for root "/" partition/mount/filer and that confirms that the above process was holding up the resources (which now I assume, is out of control/scope of "df" command's output).

Upvotes: 3

Related Questions