getting invaild type of elements IN SELENIUM

Question

working on Linux ubuntu 20.04 with intellij IDEA latest community version with firefox and geckodriver

I am trying to get some timetables from a webpage and copy them to a .txt file (or a list doesn't matter)

I am trying this :

WebDriver driver = new FirefoxDriver();
    driver.manage().timeouts().implicitlyWait(2000, TimeUnit.MILLISECONDS); //MAXIMUM WAIT TIME
    driver.get("http://telematics.oasa.gr/#main");
    driver.findElement(By.xpath("//option[contains(.,'021')]")).click();//selecting trip

    List<WebElement> oas = driver.findElements(By.xpath("//div/ul/li"));
    System.out.println(oas.size());
    System.out.println(oas);

page link : http://telematics.oasa.gr/#lineDetails_1151_021%20:%20%CE%A0%CE%9B%CE%91%CE%A4%CE%95%CE%99%CE%91%20%CE%9A%CE%91%CE%9D%CE%99%CE%93%CE%93%CE%9F%CE%A3%20-%20%CE%93%CE%9A%CE%A5%CE%96H%20(%CE%9A%CE%A5%CE%9A%CE%9B%CE%99%CE%9A%CE%97)_9-86

here is the html of the page :

    <li class="list-group-item scheduleEntryL"><button type="button" class="btn btn-info btn-circle" style="cursor:default;">07</button>&nbsp;&nbsp;&nbsp;07:10 &nbsp;&nbsp;&nbsp; 07:25 &nbsp;&nbsp;&nbsp; 07:40 &nbsp;&nbsp;&nbsp; 07:55 &nbsp;&nbsp;&nbsp; </li>

and after this the output is :

    19        

    [[[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li], [[FirefoxDriver: firefox on LINUX (115ffdb6-1eb7-44c4-bebd-dee885674bab)] -> xpath: //div/ul/li]]

which means my list has 19 elements but its not what i want

//SUMMARY

1. The type of the elements i get is not right

the list should contain:

[...,07:00,07:10,07:25,....]

2.It should contain 59 elemnts because there are 59 departures given in the page but some of them are in the same line

the page has 19 lines so it is propably giving every line as ONE element and this is also not what i want

PLEASE HELP

//I HAVE CHECKED SIMMILAR POSTS ON THIS PAGE AND DID NOT HELP

Answer 1

Your're using xpath which will return rows from page but not actual elements which contains required values. Also text values are between two nodes, so we need to use JavaScript for it.

        //Code to extract Departure time without *
        List<WebElement> oas = driver.findElements(By.xpath("//li[@class='list-group-item scheduleEntryL']"));

        LinkedList<String> timeValues = new LinkedList<String>();
        String myCountryProxy = null;
        
        for(WebElement element:oas)
        {
            //As text is between two nodes, we need to use javaScript as selenium getText() method didn't work for it
            //So in below javaScript, we refer to oas as parent element as then we try to find node which contain text as childNode. In this code its 2nd child node so we have passed value as 1: childNodes[1]
            myCountryProxy = ((JavascriptExecutor)driver).executeScript("return arguments[0].childNodes[1].textContent;", element).toString();
            //Code to remove extra space from String
            if(!myCountryProxy.equalsIgnoreCase("   "))
            {                   
                myCountryProxy = myCountryProxy.replaceAll("     "," ").replaceAll("   ","").replaceAll("     ","");
                myCountryProxy = myCountryProxy.trim();
                //Split string into individual value
                String[] split = myCountryProxy.split("\\s+");
                for(String str:split)
                {
                    timeValues.add(str);
                }                   
            }               
        }

        //Code to extract departure time values with *: 05:00*
        oas = driver.findElements(By.xpath("//span[@class='xtra']"));
        for(WebElement element:oas)
        {
            //here node which contains text is 1st child node only, so we have passed value as childNodes[0]
            myCountryProxy = ((JavascriptExecutor)driver).executeScript("return arguments[0].childNodes[0].textContent;", element).toString();
            timeValues.add(myCountryProxy);
        }

        //Code to extract departure values which are same line of time with * i.e.: 05:10     05:30     05:50  
        oas = driver.findElements(By.xpath("//li[@class='list-group-item scheduleEntryL']/span"));
        if(oas.size()>0)
        {
            oas = driver.findElements(By.xpath("//li[@class='list-group-item scheduleEntryL']/span/.."));
            for(WebElement element:oas)
            {
                //here node which contains text is 4th child node, so we have passed value as childNodes[4]
                myCountryProxy = ((JavascriptExecutor)driver).executeScript("return arguments[0].childNodes[3].textContent;", element).toString();
                //Code to remove extra space from String
                if(!myCountryProxy.equalsIgnoreCase("     "))
                {                   
                    myCountryProxy = myCountryProxy.replaceAll("     "," ").replaceAll("   ","").replaceAll("     ","");
                    myCountryProxy = myCountryProxy.trim();
                    //Split string into individual value
                    String[] split = myCountryProxy.split("\\s+");
                    for(String str:split)
                    {
                        timeValues.add(str);
                    }                   
                }                   
            }
        }

        System.out.println(timeValues);

I have this executed code at my end and getting below output:

[06:10, 06:25, 06:40, 06:55, 07:05, 07:15, 07:25, 07:35, 07:45, 07:55, 08:05, 08:20, 08:30, 08:40, 08:55, 09:05, 09:15, 09:30, 09:50, 10:10, 10:25, 10:45, 11:00, 11:20, 11:35, 11:55, 12:10, 12:30, 12:45, 13:20, 13:40, 13:55, 14:10, 14:25, 14:35, 14:45, 15:00, 15:10, 15:20, 15:35, 15:45, 15:55, 16:10, 16:20, 16:30, 16:45, 16:55, 17:05, 17:20, 17:35, 17:55, 18:15, 18:30, 18:45, 19:00, 19:20, 19:35, 19:55, 20:10, 20:25, 20:40, 21:00, 21:20, 21:40, 22:05, 22:30, 22:55, 05:00*, 23:20*, 05:10, 05:30, 05:50]

Please note, I have added values from row which contains time with *, at end of list. So you will find that values at end of list.

To extract text between nodes, I have referred to: this SO post

Also, if you want to select drop down value, use Select class of selenium:

Select select = new Select(driver.findElement(By.id("lineSelect")));
//to select drop down value: 021 : ΠΛΑΤΕΙΑ ΚΑΝΙΓΓΟΣ - ΓΚΥΖH (ΚΥΚΛΙΚΗ)
select.selectByValue("1151_54_9");