CODEWITHSUNDEEP
javaseleniumselenium-webdriverweb-scrapingselenium-chromedriver
Fjodor
Fjodor

Reputation: 539

Traversing a Table & Clicking Links Within Using Selenium

I am trying to traverse through a table using Selenium in Java (currently using the chromedriver). The content of the table consists of different people with links to their profiles, for each person in that table I will go into their profile and extract some information. This I will do for X amount of people. The table contains 5 people per page and I navigate the pages by clicking a pagination button ">". See https://www.seleniumeasy.com/test/table-pagination-demo.html for how the structure of the table looks like.

Now to the issue: So, as an example, I am currently at exampleUrl.com/page_containing_table I then enter a user's profile and extract their information, their URL being something like exampleUrl.com/user_x. I then use 

driver.navigate().back();

to come back to the table page (exampleUrl.com/page_containing_table).

The problem is that when I traverse through the table, the URL does not change. So whenever I go back from exampleUrl.com/user_x to exampleUrl.com/page_containing_table, I will always end up at the first page of the table.

This works fine when I only need to get user information from the first page in the table, but what if I need to go through 25 pages? If I am at page 11, then I would be able to retrieve the information of one user at page 11 and then I would navigate back to page 1, I would then have to paginate to page 11 again just to be able to extract one more user's information.

What I tried:

  1. I tried to just paginate all the pages to retrieve the links of all the user's to then click them but then I obviously receive the StaleElementReferenceException because that link is currently not visible on the page.

  2. I also thought that maybe there was a way to replicate the driver whenever I paginated the table and then switch to that specific driver but that also failed.

Through Google I haven't been able to find any other questions regarding this so I am a bit at a loss. Is there a way to somehow save the state so I won't have to go back to the first page every time?

Thanks in advance :)

Upvotes: 0

Views: 153

Answers (1)

JeffC
JeffC

Reputation: 25597

There are a few ways I can think of to do this:

  1. Scrape the entire table first, collecting the URL for each user's profile page and storing them in an array. Once you've finished scraping the entire table, loop through the array and navigate to each URL and scrape what you need from there. I think this is the best and fastest choice.

  2. Another possibility is that the URL contains the page of the table you are on. For example, if you are on page 5 of the table, the URL would contain something like ?page=5. I'm assuming this is not the case because the Back button is starting completely over.

  3. Another variation might be that there is a page counter on the page, e.g. something like < 1 2 3 4 5 > (where the 3 is bold or not formatted, etc.) that indicates which page you are on. Before leaving the page, you could get that info and then come back to the table and click the link associated with that particular page.

Without seeing the site/page, I can't say for sure that #2 or #3 are viable options. They are guesses based on what I've seen on other sites.

Upvotes: 1

Related Questions