CODEWITHSUNDEEP
pythonselenium
Sartorialist
Sartorialist

Reputation: 301

Clicking on Links With Variable IDs in Python / Selenium

I'm scraping a website of a university's enrollment system. Each page links to many other pages. Each link has an ID of SEC_SHORT_TITLE_x where x is an integer from 1-20. Once on each of those pages, I'd like to scrape a few pieces of data. Right now I'm just trying to scrape the section name. Will handle the logic for going back a page and clicking the next link after this.

DevTools showing xPath: enter image description here

for y in range(1):
    for j in range(1,2):
        if browser.find_elements_by_xpath("//a[@id='SEC_SHORT_TITLE_" + str(j) + "']"):
            #outputstring = ''
            browser.find_elements_by_xpath("//a[@id='SEC_SHORT_TITLE_" + str(j) + "']").click()
            time.sleep(10)
            section = browser.find_elements_by_xpath("//p[@id='VAR2']")
            print(section)

The script navigates to the proper page that contains all the links but isn't able to click on the first link as it should.

[7756:2296:0923/141749.015:ERROR:ssl_client_socket_impl.cc(941)] handshake failed; returned -1, SSL error code 1, net_error -100

Upvotes: 0

Views: 232

Answers (1)

CEH
CEH

Reputation: 5909

Based on the error message you provided (SyntaxError: Failed to execute 'evaluate' on 'Document': The string '//a[@id='SEC_SHORT_TITLE_1]' is not a valid XPath expression. (Session info: chrome=77.0.3865.90)), it looks like your XPath syntax is incorrect. You need to add a closing ' mark inside the square brackets.

Change //a[@id='SEC_SHORT_TITLE_1]

To //a[@id='SEC_SHORT_TITLE_1']

Notice how I added a single ' mark after 'SEC_SHORT_TITLE_1'.

Based on your code sample, you'll need to update this line by changing:

browser.find_elements_by_xpath("//a[@id='SEC_SHORT_TITLE_" + str(j) + "]"):

to:

browser.find_elements_by_xpath("//a[@id='SEC_SHORT_TITLE_" + str(j) + "']"):

I've added a single ' mark before your closing square bracket to correct the XPath syntax.

Upvotes: 1

Related Questions