CODEWITHSUNDEEP
pythonhtmlseleniumweb-scraping
jerry
jerry

Reputation: 2789

Python + Selenium - web scraping and counting occurrences of certain text data in HTML

Please help. I am trying to fetch data from a website and then count the occurrences of certain text. Unfortunately, I cannot provide the actual website, but the basics are this.

The web page is loaded and I am presented with a list of values, which are located in the table (the code below reflects this). The page would look something like this.

Header

Table 1

A00001

A00002

A00003

A00004

......

A00500

Each of the above rows (A00001- A00500) represent table a link that I need to click on. Furthermore, each of the links lead to a unique page that I need to extract information from.

I am using selenium to fetch the information and store it as variable data, as you can see in the code below. Here's my problem though- the number of links/rows that I will need to click on will depend on the timeframe that my user selects in the GUI. As you can see from my code, a time frame from 5/1/2011 to 5/30/2011 produces a list of 184 different links that I need to click on.

from selenium import selenium
import unittest, time, re

class Untitled(unittest.TestCase):
def setUp(self):
    self.verificationErrors = []
    self.selenium = selenium("localhost", 4444, "*chrome", "https://www.example.com")
    self.selenium.start()

def test_untitled(self):
    sel = self.selenium
    sel.open("https://www.example.com")
    sel.click("link=Reports")
    sel.wait_for_page_to_load("50000")
    sel.click("link=Cases")
    sel.wait_for_page_to_load("50000")
    sel.remove_selection("office", "label=")
    sel.add_selection("office", "label=San Diego")
    sel.remove_selection("chapter", "label=")
    sel.add_selection("chapter", "label=9")
    sel.add_selection("chapter", "label=11")
    sel.type("StartDate", "5/1/2011")
    sel.type("EndDate", "5/30/2011")
    sel.click("button1")
    sel.wait_for_page_to_load("30000")
    Case 1 = sel.get_table("//div[@id='cmecfMainContent']/center[2]/table.1.0")
    Case 2 = sel.get_table("//div[@id='cmecfMainContent']/center[2]/table.2.0")
    Case 3 = sel.get_table("//div[@id='cmecfMainContent']/center[2]/table.184.0")

def tearDown(self):
    self.selenium.stop()
    self.assertEqual([], self.verificationErrors)

if name == "main": unittest.main()

I am confused about 2 things.

1) What is the best way to get selenium to click on ALL of the links on the page without knowing the number of links ahead of time? The only way I know how to do this is to have the user select the number of links in a GUI, which would be assigned to a variable, which could then be included in the following method:

number_of_links = input("How many links are on the page? ") sel.get_table("//div[@id='cmecfMainContent']/center[2]/number_of_links")

2) I am also confused about how to count the occurrences of certain data that appear on the pages that the links lead to.

i.e.

A00001 leads to a page that contains the table value "Apples"

A00002 leads to a page that contains the table value "Oranges"

A00003 leads to a page that contains the table value "Apples "

I know selenium can store these as variables, but I'm unsure as to whether or not I can save these as a sequence type, with each additional occurrence being appended to the original list (or added to a dictionary), which could then be counted with the len() function.

Thanks for your help

Upvotes: 1

Views: 3044

Answers (1)

mike
mike

Reputation: 648

I'm not familiar with the python api so sorry for that, but in java I know using xpath there is a function to get the number of occurrences of your xpath. So you could write an xpath selector to find the element you want then get the number of occurrences of that path.

Then to click each one you can affix your xpath with an element selector like [1] so if your xpath was //somexpath/something do //somexpath/something[1] to get the first.

Hope that helps

Heres an example: I wrote a crappy api in java to be able to do jquery like operations on collections of xpath matches. My constructor matches the xpath gets the count then creates a list of all matches so i can do things like .clickAll()

 public SelquerySelector(String selector, Selenium selenium) {
        super("xpath=(" + selector + ")[" + 1 + "]", selenium);
        this.xpath = selector;
        this.selenium = selenium;
        //find out how many elements match
        this.length = selenium.getXpathCount(this.xpath).intValue();
        //make an array of selectedElements
        for(int i = 2; i <= this.length; i++) {
            elements.add(new SelquerySelectedElement("xpath=(" + this.xpath + ")[" + i + "]", this.selenium));
        }
    }

Heres the whole code in case you want to see it:

http://paste.zcd.me/Show.h7m1?id=8002

So I guess to answer your question (without knowing how xpath matches table) you can probably do something like

//div[@id='cmecfMainContent']/center[2]/table and get the number of matches to get the total amount of links then for loop over them. If you can't do that with xpath keep assuming their is another link till you get an acception

for i in range(1,xpathmatchcount):
    Case[i] = sel.get_table("//div[@id='cmecfMainContent']/center[2]/table." + i + ".0")

Upvotes: 3

Related Questions