CODEWITHSUNDEEP
pythonbeautifulsouppython-requests
Anıl Karakulak
Anıl Karakulak

Reputation: 33

web scraping - basic stock

I'm trying to get the market value of Turkish Stocks from investing.com. Here is my code:

import requests
from bs4 import BeautifulSoup

url = "https://tr.investing.com/equities/turkey"
heads = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36'}
response = requests.get(url, headers=heads)
html_icerigi = response.text
soup = BeautifulSoup(html_icerigi, "html.parser")
stock= soup.find_all("td",{"class":"bold left noWrap elp plusIconTd"})
value=soup.find_all("td",{"class":"pid-19260-last"})
stocks= []
i=1
for stock1,value1 in zip(stock,value):
stocks.append(["{}: ".format(i) + stock1.text.strip()+"-->"+value1.text.strip()])
i+=1
list(stocks)

The problem is that i want to soup all "class":"pid-***-last". How can i get this?

Upvotes: 1

Views: 171

Answers (2)

QHarr
QHarr

Reputation: 84465

You can use CSS selectors. I use the adjacent sibling combinator (+) to ensure I get the last value by its position. This will be faster than regex'ing your way through HTML.

import requests
from bs4 import BeautifulSoup

url = "https://tr.investing.com/equities/turkey"
heads = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36'}
response = requests.get(url, headers=heads)
html_icerigi = response.text
soup = BeautifulSoup(html_icerigi, "html.parser")
items = [item.text for item in soup.select('.bold.left.noWrap.elp.plusIconTd')]
values = [value.text for value in soup.select("#cross_rate_markets_stocks_1 .bold.left.noWrap.elp.plusIconTd + td")]
stocks = list(zip(items,values))

Upvotes: 3

B.Adler
B.Adler

Reputation: 1539

Use re.compile

import re
value = soup.find_all("td", class_=re.compile("pid-.+-last"))

re.compile used with class_ allows you to pass in regular expressions as the class name. This regular expression says to match the class against literally "pid-" plus any number of wild cards plus literally "-last".

https://regex101.com

Upvotes: 3

Related Questions