CODEWITHSUNDEEP
pythonseleniumweb-scraping
lord stock
lord stock

Reputation: 1201

Selenium How to scrape website table when website pagination relies on Javascript?

I'm pretty new to webscraping and As of Now I only came across scraping website that had pagination link in their next button. But I came to situation where website pagination button link doesn't change when I'm clicking on the next button.

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.implicitly_wait(30)
driver.get("https://www.nepalstock.com.np/floor-sheet?fbclid=IwAR0-V615fp0ujUH8BRr3Zu4erfAtsCz0-RHUgjMIvhjpGsaya-S9v7xI_-8")

but this website doesnot change link on loading the next button. Any help or hint I can do to grab these pagination table data?

Ps: Right click is disabled on above website (I used ctrl+shift+c) to enable it

Upvotes: 0

Views: 416

Answers (2)

Yosuva Arulanthu
Yosuva Arulanthu

Reputation: 1574

Python code snippet. I tested it and it is working

from selenium import webdriver
from selenium.webdriver.common.by import By

if __name__ == '__main__':
    driver = webdriver.Chrome('/Users/username/node_modules/chromedriver/lib/chromedriver/chromedriver')  # Optional argument, if not specified will search path.
    driver.maximize_window()
    driver.implicitly_wait(15)
    
    driver.get("https://www.nepalstock.com.np/floor-sheet?fbclid=IwAR0-V615fp0ujUH8BRr3Zu4erfAtsCz0-RHUgjMIvhjpGsaya-S9v7xI_-8");
    
    #finds all the comments or profile pics
    rowSize= driver.find_elements(By.XPATH,"//tbody//tr")
    pageSize=int (driver.find_element(By.XPATH,"//li[@class='ellipsis']/following-sibling::li//span[@class='show-for-sr']/following-sibling::span").text);

    for page in range(1,pageSize):
        for i in range(1,len(rowSize)):
            for col in range(1,8):
                print(driver.find_element(By.XPATH,"//tbody//tr["+str(i)+"]/td["+str(col)+"]").text+" | ")
            
        driver.find_element(By.XPATH,"//li[@class='pagination-next']").click();
    driver.quit()

Upvotes: 1

Yosuva Arulanthu
Yosuva Arulanthu

Reputation: 1574

I just tried the below-mentioned java code for the above-mentioned website and I am able to click the next button and also able to get the data.

driver.get("https://www.nepalstock.com.np/floor-sheet?fbclid=IwAR0-V615fp0ujUH8BRr3Zu4erfAtsCz0-RHUgjMIvhjpGsaya-S9v7xI_-8");
 
int rowSize=driver.findElements(By.xpath("//tbody//tr")).size();
int pageSize=Integer.parseInt(driver.findElement(By.xpath("//li[@class='ellipsis']/following-sibling::li//span[@class='show-for-sr']/following-sibling::span")).getText());
for(int page=1;page<pageSize;page++)
{

for (int i=1;i<rowSize;i++)
{
    
    for(int col=1;col<=8;col++)
    {
    System.out.print(driver.findElement(By.xpath("//tbody//tr["+i+"]/td["+col+"]")).getText()+" | ");
    }
    System.out.println();
    }
driver.findElement(By.xpath("//li[@class='pagination-next']")).click();
}

Upvotes: 1

Related Questions