How to cleanup disk space on openshift when 'rhc tidy' has not enough disk space?

Question

My quota on openshift has exceeded:

 Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
 /dev/mapper/EBSStore01-user_home01
            1048572       0 1048576            6890       0   80000        

I found a different stackoverflow questions that the disk space can be cleaned by 'rhc app-tidy'

But when i run this command i get the following error:

Warning: Gear xxx is using 100.0% of disk quota
Failed to execute: 'control start' for /var/lib/openshift/xxx/mysql

When i run the following command do see which files are using the most space

du -h * | sort -rh | head -50

I get the following ouput:

605M    wildfly
320M    git/mythings.git/objects
320M    git/mythings.git
320M    git
317M    wildfly/usr/lib/jvm/jdk1.8.0_05
317M    wildfly/usr/lib/jvm
317M    wildfly/usr/lib
317M    wildfly/usr
166M    wildfly/usr/lib/jvm/jdk1.8.0_05/jre
165M    wildfly/usr/lib/jvm/jdk1.8.0_05/jre/lib
136M    wildfly/versions/8
136M    wildfly/versions
118M    wildfly/versions/8/modules
118M    wildfly/usr/lib/jvm/jdk1.8.0_05/lib
118M    wildfly/modules
116M    wildfly/versions/8/modules/system/layers/base
116M    wildfly/versions/8/modules/system/layers
116M    wildfly/versions/8/modules/system
116M    wildfly/modules/system/layers/base
116M    wildfly/modules/system/layers
116M    wildfly/modules/system
95M wildfly/versions/8/modules/system/layers/base/org
95M wildfly/modules/system/layers/base/org
59M wildfly/usr/lib/jvm/jdk1.8.0_05/jre/lib/i386
52M mysql/data
52M mysql
49M wildfly/usr/lib/jvm/jdk1.8.0_05/lib/missioncontrol
43M wildfly/usr/lib/jvm/jdk1.8.0_05/lib/missioncontrol/plugins
37M wildfly/versions/8/modules/system/layers/base/org/jboss
37M wildfly/modules/system/layers/base/org/jboss
32M wildfly/usr/lib/jvm/jdk1.8.0_05/lib/visualvm
28M git/mythings.git/objects/76
26M wildfly/usr/lib/jvm/jdk1.8.0_05/jre/lib/ext
26M git/mythings.git/objects/pack
26M git/mythings.git/objects/f6
21M wildfly/usr/lib/jvm/jdk1.8.0_05/lib/visualvm/platform
21M app-root/runtime/repo
21M app-root/runtime
21M app-root
21M app-deployments/2014-08-04_14-34-47.614/repo
21M app-deployments/2014-08-04_14-34-47.614
21M app-deployments
20M wildfly/versions/8/modules/system/layers/base/org/jboss/as
20M wildfly/modules/system/layers/base/org/jboss/as
20M git/mythings.git/objects/ac
20M git/mythings.git/objects/8a
20M git/mythings.git/objects/6e
20M git/mythings.git/objects/69
20M git/mythings.git/objects/21
20M git/mythings.git/objects/12

As you can see the git directory takes a lot of space, but when i try to do 'git gc', i get the following error:

Counting objects: 226, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (192/192), done.
dm-0: write failed, user block limit reached.
fatal: sha1 file '/var/lib/openshift/xxx/git/mythings.git/objects/pack/tmp_pack_ipHspL' write error: Disk quota exceeded
error: failed to run repack

So now i am out of ideas, what can i do to cleanup some space?

Thanks!

Answer 1

As mentioned above you need to ssh to your cartridge (you will find code in web console - next to delete button) and then cd to app-root/logs and then run du -sh to check the dimension of the current dir, in my case the the file php-error.log was 542M. You can safe delete it with rm -f php-error.log

Answer 2

ssh to the application directory using ssh clients (i use putty) or command prompt then try to find out unnecessary files and delete them.

In my case deleting the content of app-log folder worked like charm.

Give attention to temp and log folders

Answer 3

Delete whatever you can easily restore from a backup or from a package, like e.g. the JDK. That should give you enough space to do cleanup. Then restore whatever you deleted.

Or pay for more storage. :-)