CODEWITHSUNDEEP
pythonweb-crawler
boeing
boeing

Reputation: 302

How to use spider.py python module

I've downloaded the spider.py 0.5 module from here. Inside the spider.py file there are lots of functions, one of them is:-

def webspider(self, b=None, w=200, d=5, t=None):

    '''Returns two lists of child URLs and paths
    b -- base web URL (default: None)
    w -- amount of resources to crawl (default: 200)
    d -- depth in hierarchy to crawl (default: 5)
    t -- number of threads (default: None)'''

    if b: self.weburls(b, w, d, t)
    return self.webpaths(), self.urls

I've created a new file in the same directory called run.py with the following code:-

import spider

webspider(b='http://example.com', w=200, d=5, t=5)

When i execute run.py i'm getting the following message:

NameError: name 'webspider' is not defined

Any ideas on how I can correctly use this module? I would like all links found to be saved into a file called urls.txt.

Upvotes: 2

Views: 2119

Answers (1)

scriptmonster
scriptmonster

Reputation: 2771

You should call it like this:

import spider
spider.webspider(b='http://example.com', w=200, d=5, t=5)

Or you can only import webspider:

from spider import webspider
webspider(b='http://example.com', w=200, d=5, t=5)

You can rename imported method:

from spider import webspider as myspider
myspider(b='http://example.com', w=200, d=5, t=5)

Upvotes: 4

Related Questions