CODEWITHSUNDEEP
pythonscrapy
Tank
Tank

Reputation: 521

Scrapy informatio is not JSON serializable

I am scraping information from some event websites I got this error when I ran my spider, I was wondering if anyone new how to solve it.

  File "/usr/lib/python3.4/json/encoder.py", line 173, in default raise TypeError(repr(o) + " is not JSON serializable")
  TypeError: <Selector xpath='.//a[@target="_top"]/text()' data='Artist Development Fellowship Informatio'> is not JSON serializable

I can scrape the text with:

scrapy shell http://ofa.fas.harvard.edu/events

for event in response.xpath('.//article[@class="node node-event node-teaser article event-start clearfix"]'):
event.xpath('.//a[@target="_top"]/text()')

spider:

import scrapy


class FAS(scrapy.Spider):
    name ='fas'
    start_urls = [
        'http://ofa.fas.harvard.edu/events',
    ]

    def parse(self, response):
        for event in response.xpath('.//article[@class="node node-event node-teaser article event-start clearfix"]'):
            yield {
            'title' : event.xpath('.//a[@target="_top"]/text()'),  
             }

Upvotes: 3

Views: 2495

Answers (1)

eLRuLL
eLRuLL

Reputation: 18799

as the error says: anything.xpath('...') is a selector, not a string you are missing to add the .extract_first() method.

anything.xpath('...').extract_first()

Upvotes: 7

Related Questions