CODEWITHSUNDEEP
pythonscrapy
Hanu
Hanu

Reputation: 119

spider from command line arguments

The following spider code output files in the form k.1375093834.0.txt. What I want is a filename of the form kickstarter.com.1375093834.0.txt

Any suggested code changes would be very helpful

class shnurl(CrawlSpider):
    name = "shnurl"
    #start_urls = [
    # "http://www.blogger.com"
    # ]
    rules = [
        Rule(SgmlLinkExtractor(),follow=True, callback="parse")
    ]

    def __init__(self, *args, **kwargs):

        #Initialize the parent class.
        super(shnurl, self).__init__(*args, **kwargs)

        #Get the start URL from the command line.
        self.start_urls = [kwargs.get('start_url')]

        #Create a results file based on the start_url + current time.
        self.fname = '{0}.{1}.{2}'.format(self.start_url[12], time.time(),'txt')
        self.fileout = open(self.fname, 'w+')

        #Create a logfile based on the start_url + current time.
        #Log file stores the errors, debug & info prints.
        logfname = '{0}.{1}.{2}'.format(self.start_url[12], time.time(),'log')
        #log.start(logfile='./runtime.log', loglevel=log.INFO)
        log.start(logfile=logfname, loglevel=log.INFO)
        self.log('Output will be written to: {0}'.format(self.fname), log.INFO)
        #End of constructor

Usage:- 

scrapy crawl shnurl -a start_url="https://www.kickstarter.com"

Upvotes: 0

Views: 209

Answers (1)

Talvalin
Talvalin

Reputation: 7889

Assuming I've understand the issue, you want to do a slice on the start_url but you've defined it incorrectly. Put a colon after the 12 in square brackets as per the following, and that will fix the issue:

    self.fname = '{0}.{1}.{2}'.format(self.start_url[12:], time.time(),'txt')
    logfname = '{0}.{1}.{2}'.format(self.start_url[12:], time.time(),'log')

Upvotes: 1

Related Questions