CODEWITHSUNDEEP
pythonhtmlweb-scraping
MJ O
MJ O

Reputation: 111

How to download tickers from webpage, beautifulsoup didnt get all content

I want to get the ticker values from this webpage https://www.oslobors.no/markedsaktivitet/#/list/shares/quotelist/ob/all/all/false

However when using Beautifulsoup I don't seem to get all the content, and I don't quite understand how to change my code in order to achieve my goal

import urllib3  
from bs4 import BeautifulSoup  
def oslobors():
http=urllib3.PoolManager()  
url = 'https://www.oslobors.no/markedsaktivitet/#/list/shares/quotelist/ob/all/all/false'  
response = http.request('GET', url)  
soup=BeautifulSoup(response.data, "html.parser")  
print(soup)  
return  
print(oslobors())

Upvotes: 2

Views: 97

Answers (1)

SIM
SIM

Reputation: 22440

The content you wanna parse generates dynamically. You can either use any browser simulator like selenium or you can try the below url containing json response. The following is the easy way to go.

import requests

url = 'https://www.oslobors.no/ob/servlets/components?type=table&generators%5B0%5D%5Bsource%5D=feed.ob.quotes.EQUITIES%2BPCC&generators%5B1%5D%5Bsource%5D=feed.merk.quotes.EQUITIES%2BPCC&filter=&view=DELAYED&columns=PERIOD%2C+INSTRUMENT_TYPE%2C+TRADE_TIME%2C+ITEM_SECTOR%2C+ITEM%2C+LONG_NAME%2C+BID%2C+ASK%2C+LASTNZ_DIV%2C+CLOSE_LAST_TRADED%2C+CHANGE_PCT_SLACK%2C+TURNOVER_TOTAL%2C+TRADES_COUNT_TOTAL%2C+MARKET_CAP%2C+HAS_LIQUIDITY_PROVIDER%2C+PERIOD%2C+MIC%2C+GICS_CODE_LEVEL_1%2C+TIME%2C+VOLUME_TOTAL&channel=a66b1ba745886f611af56cec74115a51'

res = requests.get(url)
for ticker in res.json()['rows']:
    ticker_name = ticker['values']['ITEM']
    print(ticker_name)

Results you may get like (partial):

APP
HEX
APCL
ODFB
SAS NOK
WWI
ASC

Upvotes: 1

Related Questions