Use jsoup to extract text from 'form' class with variable page data

Question

First post here so i'll do my best to keep this specific. I have been using Jsoup to extract data from a host of web pages to bring into a utitity app. I have come across a page which updates the data dynamically based on the users selection from a drop down box. I can see the data when I inspect the html in Chrome, however I cannot seem to extract it. I can extract all the text elements around it, but anything dynamically generated wont come out.

The page i'm looking at has the below form class, apologies for the wrapping, I couldn't get rid of it.



  
    
      
        
          Size
        
        
          Clear selection	
        
      
    
  

  

    
      
        1kg
      
      $13.50

I am trying to extract the price "13.50" from the below div.

$13.50

My code is below:

    private class ParseFoodPriceURL extends AsyncTask {

    @Override
    protected String doInBackground(String... strings) {
        StringBuffer buffer = new StringBuffer();
        try {
            Document doc = Jsoup.connect(strings[0]).get();
            Elements foodPrice = doc.select("div.single_variation_wrap > div.single_variation");
            String priceTextSelection = foodPrice.text();
            buffer.append("Price: $" + priceTextSelection);

        }
        catch (Throwable t) {
            t.printStackTrace();
        }
        return buffer.toString();
    }

luksch · Accepted Answer

JSoup is not a browser, so it will not interpret and execute JavaScript. If the content of a website is generated dynamically you can't use JSoup directly. Two options come to my mind:

Identify the AJAX calls directly and get the information via these calls. Often the response is not HTML but JSON. So you may need other parsing libraries. This option is fast, but you need to investigate and understand how the webpage works.
Use selenium webdriver with a real browser engine (phantomjs for example). This will load the website like a real browser but you can access its contents similar to JSoup. This is relatively easy to program, but slow and uses a lot of resources. If you run within android this may be too much. Anyway for Android the right tool for this seems to be Selenoid.

Use jsoup to extract text from 'form' class with variable page data

Answers (1)

Related Questions

Use jsoup to extract text from &#39;form&#39; class with variable page data

Answers (1)

Related Questions

Use jsoup to extract text from 'form' class with variable page data