CODEWITHSUNDEEP
pythonsqlitedictionary
S E Clark
S E Clark

Reputation: 423

Convert python dictionary of lists to SQLite database

I have a python dictionary that is currently pickled using cPickle, but is too large (~8Gb on disk) to load in at once. Since I don't necessarily need the whole dictionary at once, I'd like to use SQLite or another database system to store and access this data.

The current dictionary format is {id: (list)}, where the id is always an integer and the list is always 165 floats. The order of the floats is important, so each of them should be stored in a column.

Only I will be accessing this data, and only using python. How can I most easily write this into a database and then retrieve the list from the id?

Upvotes: 0

Views: 1385

Answers (1)

mechanical_meat
mechanical_meat

Reputation: 169274

Below is a demonstration using 3 floats. You can extrapolate for 165.

>>> import sqlite3
>>> conn = sqlite3.connect(':memory:')
>>> c = conn.cursor()
>>> c.execute('create table t (id,a,b,c);')
<sqlite3.Cursor object at 0x01418420>
>>> conn.commit()
>>> d = {0:[1.0,2.0,3.0]}
>>> import itertools as it
>>> c.execute('insert into t values (?,?,?,?)',[i for i in it.chain(d.keys(),*d.values())])
<sqlite3.Cursor object at 0x01418420>
>>> conn.commit()
>>> results = c.execute('select * from t where id = 0').fetchall()
>>> results
[(0, 1.0, 2.0, 3.0)]

Upvotes: 1

Related Questions