Sqlalchemy: OperationalError with pycharm but not at command line

Question

I'm working with scrapy and have had a functioning pipeline which uses sql-alchemy to insert records into a sqllite db. This was working but then I added 2 new fields: name and address. Now I'm getting:

2016-06-14 10:12:23 [scrapy] ERROR: Error processing {'account': u'X6',
 'address': u' the address',
 'name': u'my name'}
Traceback (most recent call last):
  File "C:\envs\virtalenvs\teat\lib\site-packages\twisted\internet\defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "C:\envs\r2\tutorial\tutorial\pipelines.py", line 54, in process_item
    self.connection.execute(ins_query)
  File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\sql\elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 1146, in _execute_context
    context)
  File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\util\compat.py", line 202, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 1139, in _execute_context
    context)
  File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) table accounts has no column named name [SQL: u'INSERT INTO accounts (account, name, address) VALUES (?, ?, ?)'] [parameters: (u'X6', u' my name', u' the address')]

Here's the pipeline code:

class MYPipeline(object):

    def __init__(self):
        _engine = create_engine("sqlite:///data.db")
        _connection = _engine.connect()
        _metadata = MetaData()
        _stack_items = Table("accounts", _metadata,
                             Column("id", Integer, primary_key=True),
                             Column("account", Text),
                             Column("name", Text),
                             Column("address", Text))
        _metadata.create_all(_engine)
        self.connection = _connection
        self.stack_items = _stack_items

    def process_item(self, item, spider):
        is_valid = True
        for data in item:
            if not data:
                is_valid = False
                raise DropItem("Missing %s!" % data)
        if is_valid:
            ins_query = self.stack_items.insert().values(
                account=item["account"],
                name=item["name"],
                address=item["address"])
            self.connection.execute(ins_query)
        return item

Whats weird is that if I run this using :

scrapy crawl myspider

using GIT-Bash (I'm using win7) this works normally. However with pycharm I get the errors above.

for pycharm the run config is in the screenshot, and man.py:

from scrapy import cmdline    
cmdline.execute("scrapy crawl myspider".split())

What am I doing wrong?

Answer 1

I suspect you're running with two different working directories. Try configuring PyCharm with the working directory you're using on the command line. Alternatively, specify an absolute path for the SQLAlchemy URL.