CODEWITHSUNDEEP
pythonscrapy
romankk
romankk

Reputation: 51

Scrapy - use "normalize-space" with a list of items

Trying to remove escape symbols and spaces from html list. I'm using normalize-space() but was not able to apply it to whole list. I'm testing my code using scrapy shell

scrapy shell https://universalmotors.ru/boardmotors/suzuki/suzuki-df-4-s/

<tr itemprop="additionalProperty" itemscope="" itemtype="http://schema.org/PropertyValue">
              <td class="label_table" itemprop="name">Мощность двигателя (л.с.)</td>
              <td class="value_table">
                <span itemprop="value">4</span>
              </td>
            </tr>
<tr itemprop="additionalProperty" itemscope="" itemtype="http://schema.org/PropertyValue">
              <td class="label_table" itemprop="name">Тип масла в двигателе</td>
              <td class="value_table">
                <span itemprop="value">10W-30 10W-40</span>
              </td>
            </tr>

Here what I tried

[item.normalize-space() for item in response.xpath('//tr[@itemprop="additionalProperty"]').extract()]

But i'm getting an error

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "<console>", line 1, in <listcomp>
AttributeError: 'str' object has no attribute 'normalize'

It is only working for

[item.strip() for item in response.xpath('//tr[@itemprop="additionalProperty"]').extract()]

then I get folowing

['<tr itemprop="additionalProperty" itemscope="" itemtype="http://schema.org/PropertyValue">\n              <td class="label_table" itemprop="name">Мощность двигателя (л.с.)</td>\n              <td class="value_table">\n                <span itemprop="value">4</span>\n              </td>\n            </tr>', '<tr itemprop="additionalProperty" itemscope="" itemtype="http://schema.org/PropertyValue">\n              <td class="label_table" itemprop="name">Тип масла в двигателе</td>\n              <td class="value_table">\n

My goal is to get flowing:

Мощность двигателя (л.с.) 4
Тип масла в двигателе 10W-30 10W-40
Объем масла в двигателе 700

Upvotes: 1

Views: 678

Answers (2)

Gallaecio
Gallaecio

Reputation: 3857

You should consider using html-text instead of XPath’s normalize-space to achieve your goal.

>>> from html_text import extract_text
>>> extract_text('''
... <tr itemprop="additionalProperty" itemscope="" itemtype="http://schema.org/PropertyValue">
...               <td class="label_table" itemprop="name">Мощность двигателя (л.с.)</td>
...               <td class="value_table">
...                 <span itemprop="value">4</span>
...               </td>
...             </tr>
... <tr itemprop="additionalProperty" itemscope="" itemtype="http://schema.org/PropertyValue">
...               <td class="label_table" itemprop="name">Тип масла в двигателе</td>
...               <td class="value_table">
...                 <span itemprop="value">10W-30 10W-40</span>
...               </td>
...             </tr>
... ''')
'Мощность двигателя (л.с.) 4\nТип масла в двигателе 10W-30 10W-40'

Upvotes: 1

Tom&#225;š Linhart
Tom&#225;š Linhart

Reputation: 10210

normalize-space is a XPath function, not a Python function or a method of a Python object. So you need to use it in the XPath expression like this:

for item in response.xpath('//tr[@itemprop="additionalProperty"]'):
    yield {
        'name': item.xpath('normalize-space(./*[@itemprop="name"])').extract_first(),
        'value': item.xpath('normalize-space(./*[@itemprop="value"])').extract_first()
    }

Upvotes: 2

Related Questions