CODEWITHSUNDEEP
pythonhtmlselenium
Analyst
Analyst

Reputation: 137

Iterate webpages through selenium python to get position of search result

I am searching a generic keyword and finding my products rank on an an e-commerce site. Here is my code for that which works successfully.

from selenium import webdriver
driver = webdriver.Chrome( "C:\All\chromedriver_win32\chromedriver.exe" )
driver.get('https://web.com/searchall?b=1&kw=printer')
items = driver.find_elements_by_class_name('productResult')
for i, item in enumerate( items ):
if 'EPSON' in item.text:
print( i )

This only fetches rank on first page but now I want to go to every page and fetch the rank of my product. I know I have to use a loop which is somewhat like this:

while True: 
try: driver.find_element_by_xpath('//div[@class="pageNavigation nextPage"]/a').click() 
except: 
break

I am finding difficulty in exact placement of the loop. Please help me with the syntax. I am a newbie.

Upvotes: 0

Views: 212

Answers (2)

Kapil
Kapil

Reputation: 407

Try below code. Hope it will help -

from selenium import webdriver
driver = webdriver.Chrome( "C:\All\chromedriver_win32\chromedriver.exe" )
driver.get('https://shop.techdata.com/searchall?b=1&kw=printer')

items_count = 1

while True: 
    items = driver.find_elements_by_class_name('productResult')
    for i, item in enumerate( items ):
        if 'EPSON' in item.text:
            print(items_count + i)
    items_count += 25
    try: 
        driver.find_element_by_xpath('//div[@class="pageNavigation nextPage"]/a').click()
    except:
        break

Upvotes: 0

Andersson
Andersson

Reputation: 52675

Try below code:

from selenium import webdriver
driver = webdriver.Chrome( "C:\All\chromedriver_win32\chromedriver.exe" )
driver.get('https://shop.techdata.com/searchall?b=1&kw=printer')

items_count = 0

while True: 
    items = driver.find_elements_by_class_name('productResult')
    for i, item in enumerate( items ):
        if 'EPSON' in item.text:
            print(items_count + i)
    items_count += len(items)
    try: 
        driver.find_element_by_xpath('//div[@class="pageNavigation nextPage"]/a').click()
    except:
        break

This should allow you to integrate your second part into first one with correct indentation

Upvotes: 1

Related Questions