CODEWITHSUNDEEP
pythonpickleshelve
ScientiaEtVeritas
ScientiaEtVeritas

Reputation: 5278

shelve gives KeyError and "cannot add item to database"

My following code:

shelve_object = shelve.open('data/shelve_data', writeback=True)
print([key for key in shelve_object])
for key in shelve_object:
    print(shelve_object[key])

gives me the following output:

['251784238314029058', '259650808973492225', '493076669914152965', '525293044375879690', '272814395237859330']

Traceback (most recent call last):
  File "/usr/lib/python3.6/shelve.py", line 111, in __getitem__
    value = self.cache[key]
KeyError: '251784238314029058'

I also noticed the following errors in my logs:

  File "/usr/lib/python3.6/shelve.py", line 113, in __getitem__
    f = BytesIO(self.dict[key.encode(self.keyencoding)])
_dbm.error

  File "/usr/lib/python3.6/shelve.py", line 125, in __setitem__
    self.dict[key.encode(self.keyencoding)] = f.getvalue()
_dbm.error: cannot add item to database

How is it possible that shelve returns keys that doesn't seem to exist?

Upvotes: 0

Views: 1377

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1125368

Either the dbm file backing your shelve is corrupted, or you have some kind of incompatibility between your file and the dbm backend library used for your shelve file.

shelve is backed by the dbm module, and it depends on your OS exactly what specific implementation is used, but generally speaking a dbm file is an on-disk hash table, a data structure that uses a sparse table structure to efficiently find keys again later on.

To show you all the keys in a file, a dbm library would scan over the table and give you each key as it finds a non-empty slot. But to find the value for a given key, it would hash the key to come to a number between zero and the maximum size of the table, look at the corresponding row in the table and get you the value.

Between those two techniques there is a discrepancy; you appear to have keys in the wrong location in the table. How that happened is hard to say, it could be that file corruption changed the bookkeeping and the actual table size differs from what the bookkeeping info states it should be. It could be that you changed machines and the hashing implementation is now wrong. Or something else, but we can't really help there, sorry.

Upvotes: 1

Related Questions