CODEWITHSUNDEEP
javascriptpythonseleniumweb-scrapingphantomjs
Swordsman
Swordsman

Reputation: 153

Scraping javascript data within a grid of a webpage

I am really new to web scraping and I am working on a project, where I need to scrape data from a grid that loads and needs to be scrolled in order to fetch all the values.
The webpage is (https://applipedia.paloaltonetworks.com/).

I need all the data within the grid - (data containing NAME , CATEGORY, SUBCATEGORY, RISK, TECHNOLOGY).

Can anyone please guide me through the way I should tackle this problem. I have researched and found out that selenium with js or phantomjs might be a good solution but not really sure about it. The programming part I will be using Python.

Upvotes: 1

Views: 1047

Answers (1)

cruisepandey
cruisepandey

Reputation: 29362

You can use this code to scrape everything from required web site: 

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC 

driver   = webdriver.Chrome(executable_path = r'C:/Users/abhishep/Downloads/chromedriver_win32/chromedriver.exe')
driver.maximize_window()

driver.get("https://applipedia.paloaltonetworks.com/") 

wait = WebDriverWait(driver,30)

table = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'tbody#bodyScrollingTable tr')))

for tab in table:
  print(tab.text)

Upvotes: 1

Related Questions