CODEWITHSUNDEEP
pythonscrapyweb-crawler
simon s
simon s

Reputation: 135

When using scrapy, crawled 0 pages (at 0 pages/min) scraped 0 items (at 0 items/min)

I just began to learn Python and Scrapy.

My first project is to crawl information on a website containing web security information. But when I run that using cmd, it says that

crawled 0 pages (at 0 pages/min) scraped 0 items (at 0 items/min)

and nothing seems to come out. I'd be grateful if someone kind could solve my problem.

Following is my spider file:

from ssl_abuse.items import SslAbuseItem
import scrapy

class SslAbuseSpider(scrapy.Spider):
    name='ssl_abuse'
    start_urls=['https://sslbl.abuse.ch/']
    def parse(self, response):
        for sel in response.xpath('/table//tr'):
            item=SslAbuseItem()
            item['date']=sel.xpath('/td/text()')[0].extract()
            item['name']=sel.xpath('/td/text()')[2].extract()
            item['type']=sel.xpath('/td/text()')[3].extract()
            yield item

Following is the website I'm about to crawl:

https://sslbl.abuse.ch/

I wish to get all element of that chart except SHA1 fingerprint..

After I changed my code like Will said, there is an error coming up:

`2017-01-04 09:31:40 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-04 09:31:40 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-01-04 09:31:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sslbl.abuse.ch/robots.txt> (referer: None)
2017-01-04 09:31:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sslbl.abuse.ch/> (referer: None)
2017-01-04 09:31:53 [scrapy.core.scraper] ERROR: Spider error processing <GET https://sslbl.abuse.ch/> (referer: None)
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
    yield next(it)
  File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output
    for x in result:
  File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 22, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "V:\work\ssl_abuse\ssl_abuse\spiders\ssl_abuse_spider.py", line 11, in parse
    item['date']=sel.xpath('/td/text()')[0].extract()
  File "c:\python27\lib\site-packages\parsel\selector.py", line 58, in __getitem__
    o = super(SelectorList, self).__getitem__(pos)
IndexError: list index out of range`

My updated code: `

from ssl_abuse.items import SslAbuseItem
import scrapy
class SslAbuseSpider(scrapy.Spider):
    name='ssl_abuse'
    start_urls=['https://sslbl.abuse.ch/']
    def parse(self, response):
        for sel in response.xpath('//table//tr'):
            item=SslAbuseItem()
            item['date']=sel.xpath('/td/text()')[0].extract()
            item['name']=sel.xpath('/td/text()')[2].extract()
            item['type']=sel.xpath('/td/text()')[3].extract()
            yield item`

Upvotes: 0

Views: 2802

Answers (1)

Will
Will

Reputation: 200

I did a quick test with scrapy shell. It seems there is something wrong with the xpath locator. The response.body looks like:

...
<table class="sortable">
<tr><th>Listing date (UTC)</th><th>SHA1 fingerprint</th><th>Common Name</th><th>Listing reason</th></tr>
<tr bgcolor="#D8D8D8" onmouseover="this.style.backgroundColor='#3371A3';" onmouseout="this.style.backgroundColor='#D8D8D8';"><td>2016-12-30 07:54:19</td><td><a href="/intel/1d05c6fef14d2671d759a05b496464b831c650e8" target="_parent" title="Show more information about this SSL certificate">1d05c6fef14d2671d759a05b496464b831c650e8</a></td><td>host/emailAddress=web@host</td><td>Gootkit C&amp;C</td></tr>
<tr bgcolor="#ffffff" onmouseover="this.style.backgroundColor='#3371A3';" onmouseout="this.style.backgroundColor='#ffffff';"><td>2016-12-28 10:03:54</td><td><a href="/intel/a82dd258544acf0a109296493421262397741db7" target="_parent" title="Show more information about this SSL certificate">a82dd258544acf0a109296493421262397741db7</a></td><td>google.com/[email protected]</td><td>Gootkit C&amp;C</td></tr>
<tr bgcolor="#D8D8D8" onmouseover="this.style.backgroundColor='#3371A3';" onmouseout="this.style.backgroundColor='#D8D8D8';"><td>2016-12-27 19:19:35</td><td><a href="/intel/df6f665e91d2fe8a338f778ad53c1921fcab3d8f" target="_parent" title="Show more information about this SSL certificate">df6f665e91d2fe8a338f778ad53c1921fcab3d8f</a></td><td>CN=p.fmsacademy.it</td><td>Gozi MITM</td></tr>
...

the first item is the table head, real content get started from the second row. For example:

# scrapy shell 'https://sslbl.abuse.ch/'
>>> rows = response.xpath('//table//tr')
>>> head = rows[0]

>>> head.xpath('th/text()').extract()
[u'Listing date (UTC)', u'SHA1 fingerprint', u'Common Name', u'Listing reason']

>>> td1 = rows[1]
>>> td1.xpath('td')
[<Selector xpath='td' data=u'<td>2016-12-30 07:54:19</td>'>, <Selector xpath='td' data=u'<td><a href="/intel/1d05c6fef14d2671d759'>, <Selector xpath='td' data=u'<td>host/emailAddress=web@host</td>'>, <Selector xpath='td' data=u'<td>Gootkit C&amp;C</td>'>]

>>> td1.xpath('td/text()').extract()
[u'2016-12-30 07:54:19', u'host/emailAddress=web@host', u'Gootkit C&C']

So, the xpath to locate tr should be:

for sel in response.xpath('//table//tr'):

the xpath to locate td text is:

item['date']=sel.xpath('td/text()')[0].extract()

Upvotes: 0

Related Questions