CODEWITHSUNDEEP
pythonscrapy
john
john

Reputation: 1280

Storing Data Locally in Python

I'm running a Scrapy project and I'm looking for the best method for storing already scraped data locally. Currently I'm using AnyDBM but i keep getting the following error after a while of running:

bsddb.db.DBRunRecoveryError: (-30973, 'DB_RUNRECOVERY: Fatal error, run database recovery -- PANIC: fatal region error detected; run recovery')

It my be something I'm doing wrong as I'm pretty new to Python, but i was wondering if there is a better solution other than Anydbm anyway.

I'm storing numeric ID's of the pages I have crawled, and will be storing around 500,000 records with plans for a possible 3-4 million for future projects.

Is AnyDBM what i should be sticking with or should i change to something more suitable for the job.

Upvotes: 0

Views: 246

Answers (2)

Serdalis
Serdalis

Reputation: 10489

By default python comes with sqlite3 which is a very good data base system.

Here is a pretty good tutorial on it. To put the table in memory use:

conn = sqlite3.connect(":memory:")
conn.isolation_level = None
cur = conn.cursor()

Upvotes: 1

Tim Pietzcker
Tim Pietzcker

Reputation: 336258

Looks like a good fit for sqlite, already part of Python's standard library.

Upvotes: 1

Related Questions