CODEWITHSUNDEEP
pythonscrapy
hadesfv
hadesfv

Reputation: 386

multiple if statements with similar pattern

I have the following code which works but I am looking for a way to write it in a more pythonic way

    if item['merchant_1']=='Google' and not item['merchant_1_price']:
        yield scrapy.Request(url=item['merchant_1_link'],callback=self.parse_google,meta={'item': item})
    elif item['merchant_2']=='Google' and not item['merchant_2_price']:
        yield scrapy.Request(url=item['merchant_2_link'],callback=self.parse_google,meta={'item': item})

    elif item['merchant_1']=='Amazon' and not item['merchant_1_price']:
        yield scrapy.Request(url=item['merchant_1_link'],callback=self.parse_amazon,meta={'item': item})        
    elif item['merchant_2']=='Amazon' and not item['merchant_2_price']:
        yield scrapy.Request(url=item['merchant_2_link'],callback=self.parse_amazon,meta={'item': item})

    elif item['merchant_1']=='Ebay' and not item['merchant_1_price']:
        yield scrapy.Request(url=item['merchant_1_link'],callback=self.parse_ebay,meta={'item': item})
    elif item['merchant_2']=='Ebay' and not item['merchant_2_price']:
        yield scrapy.Request(url=item['merchant_2_link'],callback=self.parse_ebay,meta={'item': item})
    # another 30 similar elif statements for different sites

def parse_google(self,response):
    #code
def parse_amazon(self,response):
    #code
def parse_ebay(self,response):
    #code

I get two merchants (may or may not have a price) they will sure have a link,if any of them doesn't have a price it should yield their respective parse_seller(amazon,google,ebay,...). I wrote all the parse methods in similar pattern in order to write a better looking (Pythonic) code. I am looking for a more compact way of writing those if statements

Upvotes: 1

Views: 153

Answers (2)

Thomas Pouvreau
Thomas Pouvreau

Reputation: 16

I don't know if this is more pythonic, but you could reorganize item a little bit: right now you have:

item = { merchant1 : Google,
         merchant2 : Amazon,
         merchant_1_price: XXX,
         merchant_2_price : XXX,
         merchant_1_link : url1,
         merchant_2_link : url2,
}

You could change it by something like:

item = { merchants: [Google, Amazon],
         prices : [xxx,xxx],
         links : [url1, url2]
}

Google and Amazon being Merchant objects which would be defined by a name and a specific parse method and then you just have to iterate over it:

for index,merchant in enumerate( item['merchants'] ):
    if not item['prices'][index]:
        yield scrapy.Request(url=item['links'][index],callback=merchant.parse,meta={'item': item})

Of course you have to set non-existing prize with something like 0 or None.

Here is an example on how to bind your Merchant object with a specific parse_method: 


class Merchant:

    def __init__(self, name, parse_method):
        self.name = name
        self.parse_method = parse_method

    def parse(self,data):
        return self.parse_method(data)


def parse_method_1(data):
    return data.split(":")

def parse_method_2(data):
    return data.split(",")

google = Merchant("Google", parse_method_1)
print(google.parse("hello:world"))

amazon = Merchant("Amazon", parse_method_2)
print(google.parse("hello,world"))

Upvotes: 0

AdamGold
AdamGold

Reputation: 5051

One approach would be to use a simple list.

companies = ["Google", "Amazon", "Ebay"]
for company in companies:
    for i in range(1, 3):
        if item[f"merchant_{i}"] == company and not item[f"merchant_{i}_price"]:
            yield scrapy.Request(url=item[f"merchant_{i}_link"],callback=getattr(self, f"parse_{company.lower()}"),meta={'item': item})

Upvotes: 2

Related Questions