CODEWITHSUNDEEP
pythonscrapy
bullybear17
bullybear17

Reputation: 889

Scrapy - trying to get text of a tag from inside another tag

I'm trying to get everything inside my p tag including a nested b tag, however I'm getting everything else besides the b. Can someone help me with this?

Here is my code:

def parse(self, response):
    images = response.css("#mw-content-text > div > table > tbody  > tr > td > a > img::attr(src)").extract_first()
    text1 = response.css("#mw-content-text > div > p::text").extract() 
    text2 = ''.join(text1)
    yield {'text2': text2, 'image_urls':[PROTOCOL+ images]}
    for next_page in response.css('#mw-content-text > div > ul > li > b > a::attr(href)').extract():
        yield Request(BASE_URL + next_page, callback=self.parse)

Upvotes: 0

Views: 118

Answers (1)

Apalala
Apalala

Reputation: 9244

You must use a space before ::text to have the selector retrieve all the text below the last tag:

text1 = response.css("#mw-content-text > div > p ::text").extract()

Upvotes: 1

Related Questions