CODEWITHSUNDEEP
pythonseleniumweb-scrapingbeautifulsoup
Ariel Lavi
Ariel Lavi

Reputation: 11

How can I get all the links on a site without using the html code?

Using python 3. I've been trying to find a way to get links that are on dynamic pages but I'm only getting things in the href and the links I need aren't there.

For example, trying to get the link to a specific tournament from this page, but they don't appear in the page source. Is this something I can even do with BeautifulSoup or Selenium?

Thanks!

Upvotes: 1

Views: 55

Answers (3)

Gaj Julije
Gaj Julije

Reputation: 2183

Try to use parametrized xpath for e.g for your table //tbody/tr[1]/td[1]/a[1] and enstead tr[1] use tr["i"] where i is iterator in loop. First you will need to get num of rows like //tbody/tr. This will help you fore the tournaments links.

Upvotes: 0

user8551329
user8551329

Reputation:

I had a look at that page and the links are href. BUT, there are other ways that links can work, one can use javascript to also create links. I found this on stackoverflow just to show that it could be difficult to get all the "links":

Javascript: Setting location.href versus location

Also, see this: https://www.w3schools.com/js/js_window_location.asp

Upvotes: 1

Paul Pascher
Paul Pascher

Reputation: 327

go to the page and look in :

<section class="page-section">
<div class="page-section__inner">
<table class="table">
<tbody>
<tr class="whatson-table__tournament>"
...

There is everything you need to build the links yourself.
you won't find the complete URL since they use a JS link builder or Server side controllers.

Upvotes: 1

Related Questions