CODEWITHSUNDEEP
javaweb-scraping
Amenda.St
Amenda.St

Reputation: 21

Extracting Data from a webpage using java

I'm working on a project that consists on collecting job offers from the web. So as a first step, I want to extract data (job offer data) from a specific webpage. So I want to know if there is an API or an existing code that can help me.

Upvotes: 1

Views: 920

Answers (2)

DevOps85
DevOps85

Reputation: 6523

for example you can use for make request this:

    import org.apache.http.HttpEntity;
    import org.apache.http.HttpResponse;
    import org.apache.http.NameValuePair;
    import org.apache.http.client.ClientProtocolException;
    import org.apache.http.client.HttpClient;
    import org.apache.http.client.entity.UrlEncodedFormEntity;
    import org.apache.http.client.methods.HttpGet;
    import org.apache.http.client.methods.HttpPost;
    import org.apache.http.impl.client.HttpClientBuilder;
    import org.apache.http.message.BasicNameValuePair;
    import org.apache.http.protocol.HTTP;
    import org.apache.http.util.EntityUtils;
    import org.jsoup.Jsoup;
    import org.jsoup.nodes.Document;
    import org.jsoup.nodes.Element;
    import org.jsoup.select.Elements;

    public class ... {

        Document doc;

        HttpClient client = HttpClientBuilder.create().build();
        HttpGet requestGet = new HttpGet(url + params);
        HttpResponse response = client.execute(requestGet);
        HttpEntity entity = response.getEntity();
        String responseString = EntityUtils.toString(entity, "UTF-8");

        /*
         * Here you can retrive the information with Jsoup library
         * in thi example extract data from a table element
         */
        doc = Jsoup.parse(response);
        Element elementsByTag = doc.getElementsByTag("table").get(1);

        Elements rows = elementsByTag.getElementsByTag("tr");
        for (Element row : rows) {
         \\TODO
         }
}

Upvotes: 1

farrellmr
farrellmr

Reputation: 1905

The best project I found was jsoup(http://jsoup.org/)

Upvotes: 0

Related Questions