CODEWITHSUNDEEP
amazon-s3storageceph
nig007
nig007

Reputation: 33

Ceph s3 bucket space not freeing up

I been testing Ceph with s3

my test ENV is a 3node with an datadisk of 10GB each so 30GB its set to replicate 3 times. so i have "15290 MB" space available.

I got the S3 bucket working and been uploading files, and filled up the storage, tried to remove the said files but the disks are still show as full

cluster 4ab8d087-1802-4c10-8c8c-23339cbeded8
 health HEALTH_ERR
        3 full osd(s)
        full flag(s) set
 monmap e1: 3 mons at {ceph-1=xxx.xxx.xxx.3:6789/0,ceph-2=xxx.xxx.xxx.4:6789/0,ceph-3=xxx.xxx.xxx.5:6789/0}
        election epoch 30, quorum 0,1,2 ceph-1,ceph-2,ceph-3
 osdmap e119: 3 osds: 3 up, 3 in
        flags full,sortbitwise,require_jewel_osds
  pgmap v2224: 164 pgs, 13 pools, 4860 MB data, 1483 objects
        14715 MB used, 575 MB / 15290 MB avail
             164 active+clean

I am not sure how to get the disk space back?

Can any one advise on what i have done wrong or have missed

Upvotes: 1

Views: 3514

Answers (4)

Jeremias Märki
Jeremias Märki

Reputation: 1747

I just resolved my problem of bucket space not being freed up by using the "Tools/Uncompleted Multipart Uploads" function of "S3 Browser". I found multiple years worth of "invisible" incomplete uploads.

Upvotes: 0

Alex
Alex

Reputation: 11

You can list all buckets to be processed by GC (garbage collection) with:

radosgw-admin gc list --include-all

Then you can check that GC will run after specified time. Manually you can run:

radosgw-admin gc process --include-all

It will start process of garbage collection and with "--include-all" will process all entries, including unexpired.

Then you can check the progress of clean-up with:

watch -c "ceph -s"

or simply check the result with "ceph -s" that all buckets supposed to be deleted are gone. Documentation regarding GC settings you can find here:

https://docs.ceph.com/en/quincy/radosgw/config-ref/#garbage-collection-settings

Upvotes: 1

nig007
nig007

Reputation: 33

Thanks to Julien on this info

you be right with steps 1 and 2

when you run

radosgw-admin gc list --include-all

you see an example like

[ { "tag": "17925483-8ff6-4aaf-9db2-1eafeccd0454.94098.295\u0000", "time": "2017-10-27 13:51:58.0.493358s", "objs": [ { "pool": "default.rgw.buckets.data", "oid": "17925483-8ff6-4aaf-9db2-1eafeccd0454.24248.3__multipart_certs/boot2docker.iso.2~UQ4MH7uZgQyEd3nDZ9hFJr8TkvldwTp.1", "key": "", "instance": "" }, { "pool": "default.rgw.buckets.data", "oid": "17925483-8ff6-4aaf-9db2-1eafeccd0454.24248.3__shadow_certs/boot2docker.iso.2~UQ4MH7uZgQyEd3nDZ9hFJr8TkvldwTp.1_1", "key": "", "instance": "" }, ....

if you notice the time

2017-10-27 13:51:58.0.493358s

when running

radosgw-admin gc process

It will only clear/remove parts that are older then the time feild

eg i can run "radosgw-admin gc process" over and over again but the files wont be removed till till after "2017-10-27 13:51:58.0.493358s"

but you are also right

rados purge default.rgw.buckets.data --yes-i-really-really-mean-it

works as well

Upvotes: 0

Julien
Julien

Reputation: 66

I'm beginning with ceph and had the same problem.

  1. try running the garbage collector

list what will be deleted

radosgw-admin gc list --include-all

then run it

radosgw-admin gc process
  1. if it didn't work (like for me with most of my data)

find the bucket with your data :

ceph df

Usually your S3 data goes in the default pool default.rgw.buckets.data

purge it from every object /!\ you will loose all your data /!\

rados purge default.rgw.buckets.data --yes-i-really-really-mean-it

I don't know why ceph is not purging this data itself for now (still learning...).

Upvotes: 5

Related Questions