CODEWITHSUNDEEP
pythonparsingscrapy
ijj
ijj

Reputation: 321

scrapy output titles and related links

My scrapy spider show me the titles of all web pages. Tell me please how to display a title and link related to that title? I want parsing this page. My code:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from probe1.items import SpiderItem

class SpiderSpider(CrawlSpider):
    name = "spider"
    allowed_domains = ["WEB_PAGE"]
    start_urls = [
    "http://www.WEB_PAGE"
    ]

    rules = (
        Rule(
            SgmlLinkExtractor(allow_domains=("WEB_PAGE",)),
            callback='parse_page', follow=True
        ),
    )


    def parse_page(self, response):
      hxs = HtmlXPathSelector(response)
      print hxs
      sites = hxs.select('//title')
      items = []
      for s in sites:
      item = SpiderItem()
          item['title'] = s.select('//title').extract
          items.append(item)
      return items

Upvotes: 1

Views: 94

Answers (1)

alecxe
alecxe

Reputation: 473873

response.url contains what you need:

url

A string containing the URL of the response.

Upvotes: 1

Related Questions