CODEWITHSUNDEEP
scrapy
Jabb
Jabb

Reputation: 3502

Scrapy: Default values for items & fields. What is the best implementation?

As far as I could find out from the documentation and various discussions on the net, the ability to add default values to fields in a scrapy item has been removed.

This doesn't work

category = Field(default='null')

So my question is: what is a good way to initialize fields with a default value?

I already tried to implement it as a item pipeline as suggested here, without any success. https://groups.google.com/forum/?fromgroups=#!topic/scrapy-users/-v1p5W41VDQ

Upvotes: 6

Views: 3526

Answers (2)

Roman Bukin
Roman Bukin

Reputation: 67

Typically, a constructor is used to initialize fields.

class SomeItem(scrapy.Item):

    id = scrapy.Field()
    category = scrapy.Field()

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self['category'] = 'null'  # set default value

This may not be a clean solution, but it avoids unnecessary pipelines.

Upvotes: -1

Jabb
Jabb

Reputation: 3502

figured out what the problem was. the pipeline is working (code follows for other people's reference). my problem was, that I am appending values to a field. and I wanted the default method work on one of these listvalues... chose a different way and it works. I am now implementing it with a custom setDefault processor method.

class DefaultItemPipeline(object):

def process_item(self, item, spider):
    item.setdefault('amz_VendorsShippingDurationFrom', 'default')
    item.setdefault('amz_VendorsShippingDurationTo', 'default')
    # ...
    return item

Upvotes: 3

Related Questions