CODEWITHSUNDEEP
python-2.7scrapy
skittles
skittles

Reputation: 23

Splitting variables in scrapy spider

forgive me, I am a total programming noob.

I am trying to extract a record id from a url with the following code and Im running into trouble. If I run it through the shell it seems to work fine (no errors) but when I run it through scrapy the framework generates errors

Example:
if the url is http://domain.com/path/to/record_id=1599
then record_link = /path/to/record_id=1599
therefore record_id should be = 1599

   for site in sites:

      record_link = site.select('div[@class="description"]/h4/a/@href').extract()
      record_id = record_link.strip().split('=')[1]

      item['link'] = record_link
      item['id'] = record_id
      items.append(item)

any help is greatly appreciated

EDIT::

Scrapy errors like something like this:

   root@web01:/home/user/spiderdir/spiderdir/spiders# sudo scrapy crawl spider
   2012-02-23 09:47:16+1100 [scrapy] INFO: Scrapy 0.13.0.2839 started (bot: spider)
   2012-02-23 09:47:16+1100 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, MemoryUsage, SpiderState
   2012-02-23 09:47:16+1100 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
   2012-02-23 09:47:16+1100 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
   2012-02-23 09:47:16+1100 [scrapy] DEBUG: Enabled item pipelines:
   2012-02-23 09:47:16+1100 [spider] INFO: Spider opened
   2012-02-23 09:47:16+1100 [spider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
   2012-02-23 09:47:16+1100 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6031
   2012-02-23 09:47:16+1100 [scrapy] DEBUG: Web service listening on 0.0.0.0:6088
   2012-02-23 09:47:19+1100 [spider] DEBUG: Crawled (200) <GET http://www.domain.com/path/to/> (referer: None)
   2012-02-23 09:47:21+1100 [spider] DEBUG: Crawled (200) <GET http://www.domain.com/path/to/record_id=2> (referer: http://www.domain.com/path/to/)
   2012-02-23 09:47:21+1100 [spider] ERROR: Spider error processing <GET http://www.domain.com/path/to/record_id=2>
   Traceback (most recent call last):
      File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 778, in runUntilCurrent
        call.func(*call.args, **call.kw)
      File "/usr/lib/python2.6/dist-packages/twisted/internet/task.py", line 577, in _tick
        taskObj._oneWorkUnit()
      File "/usr/lib/python2.6/dist-packages/twisted/internet/task.py", line 458, in _oneWorkUnit
        result = self._iterator.next()
      File "/usr/lib/pymodules/python2.6/scrapy/utils/defer.py", line 57, in <genexpr>
        work = (callable(elem, *args, **named) for elem in iterable)
    --- <exception caught here> ---
      File "/usr/lib/pymodules/python2.6/scrapy/utils/defer.py", line 96, in iter_errback
        yield it.next()
      File "/usr/lib/pymodules/python2.6/scrapy/contrib/spidermiddleware/offsite.py", line 24, in process_spider_output
        for x in result:
      File "/usr/lib/pymodules/python2.6/scrapy/contrib/spidermiddleware/referer.py", line 14, in <genexpr>
        return (_set_referer(r) for r in result or ())
      File "/usr/lib/pymodules/python2.6/scrapy/contrib/spidermiddleware/urllength.py", line 32, in <genexpr>
        return (r for r in result or () if _filter(r))
      File "/usr/lib/pymodules/python2.6/scrapy/contrib/spidermiddleware/depth.py", line 56, in <genexpr>
        return (r for r in result or () if _filter(r))
      File "/usr/lib/pymodules/python2.6/scrapy/contrib/spiders/crawl.py", line 66, in _parse_response
        cb_res = callback(response, **cb_kwargs) or ()
      File "/home/nick/googledir/googledir/spiders/google_directory.py", line 36, in parse_main
        record_id = record_link.split("=")[1]
    exceptions.AttributeError: 'list' object has no attribute 'split'

`

Upvotes: 1

Views: 2959

Answers (2)

skittles
skittles

Reputation: 23

I think what I'm after is something like this:

for site in sites:

      record_link = site.select('div[@class="description"]/h4/a/@href').extract()
      record_id = [i.split('=')[1] for i in record_link]

  item['link'] = record_link
  item['id'] = record_id
  items.append(item)

Upvotes: 0

Sjaak Trekhaak
Sjaak Trekhaak

Reputation: 4966

Kind of a long shot since you didn't post your errors, but I'm guessing you will have to change this line:

record_id = record_link.strip().split('=')[1]

to

record_id = record_link[0].strip().split('=')[1]

Since HtmlXPathSelector always returns a list of selected items.

Upvotes: 3

Related Questions