Reputation: 27
I'm really having the damndest time trying to discern the cause of an error message I'm getting.
I'm writing a web scraper that scrapes data from Procycing Stats into a MySQL database using Python with BeautifulSoup for scraping and Peewee for database interaction. The web scraper works totally fine, but I'm having a bit of trouble with inserting the data into MySQL tables.
First off, I created the tables in an empty database by using peewee's create_tables()
function. Underneath I've pasted the code for my Peewee models, which is contained in a file I call peewee_lib.py
.
from peewee import *
from mysql_login_info import *
results_database = MySQLDatabase(mysql_db_name, user=mysql_uname, password=mysql_pw, host='localhost')
class BaseModel(Model):
class Meta:
database = results_database
class Rider(BaseModel):
pcsid = IntegerField()
name = CharField()
class Race(BaseModel):
name = CharField()
class Result(BaseModel):
name = CharField()
year = IntegerField()
date = DateField()
position = IntegerField()
points_pcs = IntegerField()
race = ForeignKeyField(Race, backref='results')
rider = ForeignKeyField(Rider, backref='results')
Next, I use a file scrape_to_peewee.py
to create classes that "bind" together my class definitions from my scraping library scraper_lib.py
and the aforementioned peewee library, peewee_lib.py
.
Here is the code from scrape_to_peewee.py
:
import scraper_lib as pylib
import peewee_lib as pw
class Sheet_bind:
def __init__(self, rider_obj, sheet):
self.year = sheet.year
self.rider = sheet.rider
self.rows = []
for row in sheet.rows:
if row.row_type == "tour_header":
pass
else:
temp_query = pw.Race.select().where(pw.Race.name == row.race)
if not temp_query.exists():
temp_query = pw.Race(name=row.race)
temp_query.save()
else:
pass
temp_res = pw.Result(name=row.name,\
year=sheet.year,\
position=row.result,\
points_pcs=row.points_pcs)
if row.row_type in ["stage", "classification"]:
temp_res.name = row.race + ' ' + row.name
temp_res.race=temp_query
temp_res.rider=rider_obj
temp_res.save()
temp_query = None
temp_res = None
class Rider_bind:
def __init__(self, rider_id):
self.rider_py = pylib.Rider(rider_id)
self.rider_pw = pw.Rider(pcsid=self.rider_py.url_id, name=self.rider_py.name)
self.rider_pw.save()
def load_sheets(self, start_year, end_year):
for year in xrange(start_year, end_year + 1):
if year not in self.rider_py.sheets:
self.rider_py.load_sheets(year, year)
loaded_sheet = Sheet_bind(self.rider_pw, self.rider_py.sheets[year])
loaded_sheet.save()
def main():
pw.results_database.connect()
main()
After loading this final file into the interpreter, I attempted to load an example rider into the database. Initiating the Rider_bind
class worked fine, and I double checked to make sure that a row had actually been written to my rider
table in MySQL, which it had. When I attempt to load results into the database with Rider_bind.load_sheets()
however, I get the following error:
$ python
Python 2.7.15rc1 (default, Nov 12 2018, 14:31:15)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from scrape_to_peewee import *
>>> olly = Rider_bind("oliver-naesen")
>>> olly.load_sheets(2018, 2018)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "scrape_to_peewee.py", line 55, in load_sheets
loaded_sheet = Sheet_bind(self.rider_pw, self.rider_py.sheets[year])
File "scrape_to_peewee.py", line 33, in __init__
temp_res.race=temp_query
File "/home/trenza/.local/lib/python2.7/site-packages/peewee.py", line 3848, in __set__
if obj != fk_value and self.name in instance.__rel__:
File "/home/trenza/.local/lib/python2.7/site-packages/peewee.py", line 726, in __ne__
return not (self == other)
File "/home/trenza/.local/lib/python2.7/site-packages/peewee.py", line 723, in __eq__
return self._hash == other._hash
AttributeError: 'NoneType' object has no attribute '_hash'
It appears that the problem relates to assigning one of the peewee models to a foreignkey field. When I reversed the order of the calls so that temp_res.rider = rider_obj
came first, it gave me the same error with the traceback pointing to that call.
From the peewee docs it seems that ForeignKey fields should be as simple as just assigning another peewee class to them as a value. Does anyone know what I'm getting wrong here? any help would be much appreciated.
Thank you!
Edit:
Not a duplicate of this question because it is (as far as I know) not related to the return value from a select
call (the problem in the aforementioned question).
Upvotes: 1
Views: 935
Reputation: 26235
You need to resolve the "temp_query" into an object when assigning to an attribute.
if not temp_query.exists():
temp_query = pw.Race(name=row.race)
temp_query.save()
else:
temp_query = temp_query.get() # fixed.
Upvotes: 2