CODEWITHSUNDEEP
python-3.xweb-scrapingbeautifulsoup
innicoder
innicoder

Reputation: 2688

BeautifulSoup can't find a [ a href ] tag

enter image description here

page = "http://www.flashscore.com/"
soup =  bs(request.urlopen(page).read(), 'lxml')
typeyourcountry = input("Type in the NAME  or ID of the country you're looking for : ")

a = '/soccer/' + typeyourcountry.lower()

for link in soup.findAll('a', attrs={'href': re.compile(a)}):
    print (link)
    find_urls = re.compile(a+'(.*?)"')

OUTPUT:

<a href="/soccer/serbia/" onclick="return cjs.dic.get('Helper_Menu').lmenu(167,req_url,1);">Serbia</a>
<a href="/soccer/serbia/super-liga/">Super Liga</a>

Even tho there's 3 links that should be outputed I'm only able to find one, is it a dynamic link or has anyone got any ideas to what's happening? As you can see it on the image, there's 3 links that should be outputed and I can see each of them in the source viewer but got no idea what's going on beyond that, help is appreciated ! Just noticed the JS :

onclick="return cjs.dic.get('Helper_Menu').lmenu(167,req_url,1);

I guess this script above reveals the information (That I was looking for ), is there any way for me to reveal it with python, make the same request js request and only get that portion of the data or what's the viable solutions?

SUMMARY : One website has FrontEnd JS script that Reveals particular information

How do I implement that kind of request into Python or BeautifulSoup,is there another library that has a solution?

Upvotes: 1

Views: 540

Answers (1)

Alexis Tacnet
Alexis Tacnet

Reputation: 535

This is because the others links are generated with Javascript. In fact, if you go to your dev tools in your browser and desactivate JS, you will see that there are only the links that BeautifulSoup is finding.

If you want the others links to appear, you should use PhantomJS for example ;)

See ya !

Upvotes: 1

Related Questions