CODEWITHSUNDEEP
javaselenium
MasterJoe
MasterJoe

Reputation: 2335

How do I get an element by its location on the page?

The problem: I want to extract phone numbers from a page. Every number is hidden under a button called "show contact info". The numbers are not found in the dom until you click the button. When you click the button, the button is replaced by a phone number.

Is there a reliable way to find an element by location on page ? If yes, then I can get the location of the button, click it and then get the phone number at the location of the button.

Here is the page: https://sfbay.craigslist.org/pen/apa/5753779484.html This page will be removed after some time. I can provide a similar page when that happens.

Thanks.

I also tried this using only xpath and it failed:

1 - Get the 1st preceding sibling element of "show contact info" buttons.

2 - Using the elements from 1, click the buttons. Buttons are replaced by phone numbers.

3 - Again, using elements from 1, get text the phone numbers.

Code:

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;

import java.util.ArrayList;
import java.util.List;

public class Temp {

    private static WebDriver browser = new ChromeDriver();

    public static List<String> get_phone_numbers() {
        String url = "https://sfbay.craigslist.org/pen/apa/5753779484.html";
        browser.get(url);

        List<String> phones = new ArrayList<String>();
        String text;
        String phone;
        WebElement contact;

        String before_contact_buttons_xpath = "//*[@id='postingbody']/a[contains(., 'show contact info')]/" +
                "preceding-sibling::*[1]";

        //Get all the preceding sibling elements of "show contact info button."
        List<WebElement> pre_contacts = browser.findElements(By.xpath(before_contact_buttons_xpath));

        for (WebElement pre_contact : pre_contacts) {
            //Click the "show contact info" button. It disappears after click & is replaced by an phone number.
            WebElement temp_contact_btn = pre_contact.findElement(By.xpath("following-sibling::*[1]"));
            System.out.println(temp_contact_btn.getText());
            temp_contact_btn.click();

            //Now get the number from the replaced "show contact info" button.
            contact = pre_contact.findElement(By.xpath("following-sibling::*[1]"));

            text = contact.getText();
            System.out.println(text);

            phone = "000-111-2222";//extract_phone_number(text);

            phones.add(phone);
        }
        return phones;
    }

    public static void main(String[] args) {
        List<String> phones = get_phone_numbers();
    }
}

Output:

show contact info
show contact info

Upvotes: 0

Views: 92

Answers (1)

qxz
qxz

Reputation: 3854

On that page, clicking the button (actually an <a>) triggers an XMLHttpRequest that reloads the entire content of the description section. It seems to just do a GET request of the <a>'s href: https://sfbay.craigslist.org/fb/sfo/apa/5753779484. Try going to that link or right-clicking the "button" and opening the link in a new tab.

Once you have the <a> element, I'd suggest requesting the page at its href and then somehow parsing the content to get the phone number. This wouldn't be too hard with regular expressions, assuming there aren't any other phone numbers.

Upvotes: 1

Related Questions