CODEWITHSUNDEEP
pythonscrapyeggscrapyd
The.Anti.9
The.Anti.9

Reputation: 44718

Scrapyd deploy "Attribute Error: NoneType object has no attribute 'module_name'"

I'm trying to using scrapy to deploy my crawler project to a scrapyd instance but calling the commend returns the following error:

Server response (200): {"status": "error", "message": "AttributeError: 'NoneType' object has no attribute 'module_name'"}

Here's my setup.py to build the python egg submitted during deploy:

from setuptools import setup, find_packages

setup(
    name = 'mycrawler',
    version = '0.1',
    packages = find_packages(),
    install_requires = [
        'scrapy',
        'PyMongo',
            'simplejson',
            'queue'
    ]
)

My scrapy.cfg:

[settings]
default = mycrawler.settings

[deploy:scrapyd_home_vm]
url = http://192.168.1.2:6800/
project = mycrawler

[deploy:scrapyd_local_vm]
url = http://192.168.38.131:6800/
project = mycrawler

I get the feeling that this has to do with the way the egg is being built but I'm not sure. I know that python throws an error like this when an you access an attribute on what should be an object but for whatever reason is actually null. I also do not have anything with the "module_name" attribute or anything that tries to reference it in my own code. Running the crawler from scrapy locally works just fine but deploying the egg does not.

Upvotes: 2

Views: 2985

Answers (3)

Vinícius OA
Vinícius OA

Reputation: 357

Are you using scrapyd-client package? If yes, then you don't even need a setup.py. I've came across that AttributeError because I already had a setup.py, so I deleted.

Upvotes: 1

Darian Moody
Darian Moody

Reputation: 4084

A rather late answer but I came across this same issue and found the solution.

The problem for me could be found by looking at the traceback emanating from scrapyd itself:

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/vagrant/venv/lib/python2.7/site-packages/scrapyd/runner.py", line 39, in <module>
    main()
  File "/home/vagrant/venv/lib/python2.7/site-packages/scrapyd/runner.py", line 34, in main
    with project_environment(project):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/vagrant/venv/lib/python2.7/site-packages/scrapyd/runner.py", line 22, in project_environment
    activate_egg(eggpath)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/scrapyd/eggutils.py", line 13, in activate_egg
    settings_module = d.get_entry_info('scrapy', 'settings').module_name

AttributeError: 'NoneType' object has no attribute 'module_name'

As you can see it is trying to load the scrapy project's settings_module; the module_name attribute doesn't exist because d.get_entry_info is returning the None object.

The solution is to check your setup.py that is being used to generate the egg and check that the call to setup contains the following lines:

packages=find_packages(),
entry_points={'scrapy': ['settings = scraper.settings']},

Here, scraper.settings is the Python module path to the scrapy project's settings file. Change this to one which reflects your project layout and you should be dandy..

..if not, the key here is to check the output from scrapyd (enabling debug allows you to see that in the server response) to find the solution.

Upvotes: 9

isedev
isedev

Reputation: 19631

This is a coding error, probably in your mycrawler module:

AttributeError: 'NoneType' object has no attribute 'module_name'

This means you are trying to access the attribute module_name in an object that was returned by some function or method, but the return value was None as opposed to being an object (probably the way in which the function or method indicates that an error occurred).

Check your code for places where you reference module_name name in a returned value.

Or it could be that scrapy requires that one of the objects you define and pass to it has to have the module_name attribute defined and you've forgotten to do so.

Finally, it could be a bug in scrapy.

But it very unlikely to be a problem with setuptools.

Upvotes: -1

Related Questions