CODEWITHSUNDEEP
pythondictionaryscrapy
user568109
user568109

Reputation: 48003

How to get a dict from scrapy item?

I need the values in a dict. But item uses some abstraction on top of it. How to get the fields in a dict from an item ?

I know scrapy allows dict to be returned in place of item now. But I already am using item in my code, so how to convert it.

Upvotes: 5

Views: 5703

Answers (2)

Peter Wegrzyn
Peter Wegrzyn

Reputation: 71

The accepted answer works great but it won't work when you have nested items. In such case you might want to first convert the item to string and then to json like so:

class CustomEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, scrapy.Item):
            return dict(o)

Upvotes: 0

Robert Moskal
Robert Moskal

Reputation: 22553

It looks to me like :

class Product(scrapy.Item):
    name = scrapy.Field()


i = Product(name='foo)
print dict(i)

gets you a dictionary {'name': 'foo'}

vars(p)
p.__dict__

gets you: {'_values': {'name': 'foo'}}

If you don't want to create a new dictionary, just grab the _values key from the above:

vars(p)['_values']
p.__dict__['_values']

Upvotes: 8

Related Questions