CODEWITHSUNDEEP
pythonweb-scrapingautomationpython-requestsstocks
Golgorie Haus
Golgorie Haus

Reputation: 55

Python webscraping - realtime data

I am trying scrape the live data at the to of this page: https://www.wallstreet-online.de/devisen/euro-us-dollar-eur-usd-kurs/realtime

My current method:

import time
import re
import bs4 from bs4 import BeautifulSoup as soup
import requests
while (1==1):
   con =  requests.request('get','https://www.wallstreet- 
   online.de/devisen/euro-us-dollar-eur-usd-kurs/realtime', stream = True)
   page = con.text
   kursSoup = soup(page, "html.parser")
   kursDiv = kursSoup.find("div", {"class":"pull-left quoteValue"})
   print(kursDiv.span)
   del con
   del page
   del kursSoup
   del kursDiv
   #time.sleep(2)
print("end")

works but is not in sync with the data on the website. I dont really get why because i delete all the variables at the end of the loop so the result should change when the data on the website changes but seems to stay the same for a fixed amount of times. Does anyone know why or has a better way of doing this (Im a bloody beginner and have no idea how the site even works thats why im parsing the html).

Upvotes: 1

Views: 1217

Answers (1)

Aidan Rosswood
Aidan Rosswood

Reputation: 1212

It looks like that web page may be using JavaScript to populate and update that number. I'm not familiar with BeautifulSoup but I don't think it will run the JavaScript on the page to update that number.

You may want to use something like Chrome Developer Tools to keep an eye on the network tab. I looked and it looks like there is a websocket connection to wss://push.wallstreet-online.de/lightstreamer going on behind the scenes. You may want to use a websocket client Python library to read from this socket and either find some API docs or reverse engineer the data that comes from the socket. Good luck!

Upvotes: 2

Related Questions