CODEWITHSUNDEEP
pythonseleniumweb-scraping
AmilaMGunawardana
AmilaMGunawardana

Reputation: 1830

Web Scraping a specific table that has no anchor tag or a button,

Hi I am using Beautiful Soup to scrape a table from following site (https://www.otcmarkets.com/market-activity/corporate-actions). There is a table called SYMBOL CHANGES. I want to grab the data inside that table. I was able to capture the data in the loading page but below there is a class called more. I cannot click that class with selenium. I have used below code to find the element.

code for the html page

<div class="_2sFaw3zGf1">
   More 
   <svg fill="currentColor" preserveAspectRatio="xMidYMid meet" height="1em" width="1em" viewBox="0 0 40 40" class="APX-ntK2Ti" style="vertical-align: middle;">
      <g>
         <path d="m37.6 18l-16.6 16.6q-0.4 0.4-1 0.4t-1-0.4l-16.6-16.6q-0.4-0.4-0.4-1t0.4-1l3.7-3.7q0.5-0.4 1-0.4t1 0.4l11.9 11.9 11.9-11.9q0.4-0.4 1-0.4t1 0.4l3.7 3.7q0.4 0.4 0.4 1t-0.4 1z"></path>
      </g>
   </svg>
</div>

Code that i have used to do the click event.

d = driver.find_element_by_xpath("/[contains(text()='More')]")
d.click()

Since that is not a button object or a anchor tag how should i proceed? Thank you in advance.

Upvotes: 0

Views: 112

Answers (1)

dimay
dimay

Reputation: 2804

Try to use requests and pandas:

import requests
import pandas as pd

r = requests.get("https://backend.otcmarkets.com/otcapi/corp-actions/symbol-changes?route=symbol-changes&pageSize=900&retainPageSize=true")
df = pd.DataFrame(r.json())
print(df.shape)
print(df)

Upvotes: 1

Related Questions