CODEWITHSUNDEEP
pythonseleniumweb-scraping
Jennifer Williams
Jennifer Williams

Reputation: 9

Unable to scrape the content of the looped pages (next page)

I was trying to scrape a paginated site with selenium python. The code I write was able to extract data from the first page and also proceeded to the page 2 but it could not extract the content of the 2nd page and the rest pages.

I got the result of only page 1

from selenium import webdriver
import time
browser = webdriver.Chrome(executable_path='C:\Python27\Scripts\chromedriver.exe')



browser.get("https://www.etsy.com/ca/c/jewelry/necklaces" )


posts= browser.find_elements_by_class_name("text-gray")

for post in posts:

  print post.text

for i in range(1,3):
   u=browser.get('https://www.etsy.com/ca/c/jewelry/necklaces?ref=pagination&page=%s' % str(i))

   print".................................."+ str(i)+"......................................."
time.sleep(10)   
new= u.find_element_by_class_name("text-gray")
for we in new:
   print we.text

This is the error message I got: AttributeError: 'NoneType' object has no attribute 'find_elements_by_class_name

Upvotes: 0

Views: 49

Answers (1)

johnny1995
johnny1995

Reputation: 123

Try this:

from selenium import webdriver 

import time 

browser = webdriver.Chrome(executable_path='C:\Python27\Scripts\chromedriver.exe')
browser.get("https://www.etsy.com/ca/c/jewelry/necklaces" )
posts= browser.find_elements_by_class_name("text-gray")

for post in posts:
    print post.text

for i in range(1,3):
    gets = 'https://www.etsy.com/ca/c/jewelry/necklaces?ref=pagination&page='+str(i)
    u = browser.get(gets)
    time.sleep(10)
    new = u.find_element_by_class_name("text-gray") 
    for we in new: 
        print we.text

Upvotes: 1

Related Questions