CODEWITHSUNDEEP
scrapy
Jabb
Jabb

Reputation: 3502

How to properly use item_loader in Scrapy? Item_loader class not working properly

I pass scrapy items via the meta property to a callback function like so:

def parse_VendorDetails(self, response):
    loader = XPathItemLoader(response.meta['item'], response=response)

    print loader.get_collected_values('testField') <-- returns empty value
    print response.meta['item']['testField'] <-- return expected value

the first print outputs an empty list. the second print returns the value as expected. What could be the reason for this?

Upvotes: 0

Views: 785

Answers (1)

Steven Almeroth
Steven Almeroth

Reputation: 8212

There is currently a bug in ItemLoaders,get_output_value() and get_collected_values() ignore item parameters and only look at ItemLoader._values so there is some inconsistent behavior in these methods in that loaded data via load_item() is not returned:

>>> from scrapy.contrib.loader import ItemLoader
>>> il = ItemLoader(response=response, item=dict(foo=1))
>>> il.add_value('bar', 3)
>>> il._values
defaultdict(<type 'list'>, {'bar': [3]})

>>> il.item
{'foo': 1}

>>> il.get_output_value('foo')
[]

>>> il.get_output_value('bar')
[3]

>>> il.get_collected_values('foo')
[]

>>> il.get_collected_values('bar')
[3]

You can install the proposed patch or just not use get_collected_values. If you install the patch you can use the values parameter with this patch we experience a more sane result:

>>> from scrapy.contrib.loader import ItemLoader
>>> il = ItemLoader(response=response, item={}, values=dict(foo=1))
>>> il.add_value('bar', 3)
>>> il._values
defaultdict(<type 'list'>, {'foo': [1], 'bar': [3]})

>>> il.item
{}

>>> il.get_output_value('foo')
[1]

>>> il.get_output_value('bar')
[3]

>>> il.get_collected_values('foo')
[1]

>>> il.get_collected_values('bar')
[3]

Upvotes: 1

Related Questions