CODEWITHSUNDEEP
ceph
Turtle V. Rabbit
Turtle V. Rabbit

Reputation: 313

Deleting files in Ceph does not free up space

I am using Ceph, uploading many files through radosgw. After, I want to delete the files. I am trying to do that in Python, like this:

bucket = conn.get_bucket(BUCKET)
for key in bucket.list():
    bucket.delete_key(key)

Afterwards, I use bucket.list() to list files in the bucket, and this says that the bucket is now empty, as I intended.

However, when I run ceph df on the mon, it shows that the OSDs still have high utilization (e.g. %RAW USED 90.91). If I continue writing (thinking that the status data just hasn't caught up with the state yet), Ceph essentially locks up (100% utilization).

What's going on?

Note: I do have these standing out in ceph status:

   health HEALTH_WARN
            3 near full osd(s)
            too many PGs per OSD (2168 > max 300)
            pool default.rgw.buckets.data has many more objects per pg than average (too few pgs?)

From what I gather online, this wouldn't cause my particular issue. But I'm new to Ceph and could be wrong.

I have one mon and 3 OSDs. This is just for testing.

Upvotes: 2

Views: 4409

Answers (1)

hjwsm1989
hjwsm1989

Reputation: 83

You can check if the object is really deleted by rados -p $pool list, I knew for cephfs, when you delete a file, it will return ok when mds mark it as deleted in local memory and then do real delete by sending delete messages to related osd. Maybe radosgw use the same design to speed up delete

Upvotes: 1

Related Questions