How do I read in the XML for from java http request?

Question

When I perform a certain action on a web app, it performs a ajax call or something along that line that returns some data in XML format. When I click On Inspect element in the browser and click on the networks tab, I can see the XML response data that was requested. see:

I tried to perform a http request in java to retrieve this XML data. Here is the java code:

private StringBuffer sendGet() {

    final String USER_AGENT = "Mozilla/5.0";
    final String CONTENT_LENGTH = "131";
    StringBuffer response = new StringBuffer();
    String url = "https://same as the request header";

    try {
        //create the http connection
        URL obj = new URL(url);
        HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();

        con.setRequestMethod("GET");
        con.setRequestProperty("User-Agent",USER_AGENT);
        con.setRequestProperty("Content-Length",CONTENT_LENGTH);

        int responseCode = con.getResponseCode();
        System.out.println("Sending 'GET' request to URL " + url);
        System.out.println("Response Code:" + responseCode);

        //read in the get reponse
        BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
        String inputLine;

        while((inputLine = in.readLine()) != null) {
            response.append(inputLine.toString());
        }

        //close input stream
        in.close();

        System.out.println("response is: " + response);

    } catch (MalformedURLException e) {
        e.printStackTrace();

    } catch (IOException e) {
        e.printStackTrace();
    }

    return response;
}

For the URL I am creating a http request, I am using the exact same request URL in the ajax calls header. see:

When I perform the GET request, I received a response code of 200 which I assume the request was successful, but on my log it displays no XML even though I try to print it out. The log displayed this:

Sending 'GET' request to URL https:"blah blah blah"
Response Code:200
response is: 

I should also note that when I try to go to the request URL directly on the browser it's just a blank white page.

When I use a request URL that contains a webpage, the http request returns all the html and js in the DOM. But the request URL I was trying to use which is just a blank page returns a blank response to me even though there are suspose to be some xml data. What am I doing wrong? Why is it that I cannot retrieve and display the XML? Do I need to parse the XML before it can be displayed?

Answer 1

I solved the issue by changing it to a POST request (same as what the browser did). I also included the form data associated with that request as shown in the pictures up there.

private StringBuffer sendPost() {

    StringBuffer response = new StringBuffer();
    String url = "https://example.com/snowbound/AjaxServlet";
    final String CONTENT_LENGTH = "131";
    final String CONTENT_TYPE = "application/x-www-form-urlencoded";
    final String ACCEPT_LANGUAGE = "en-US,en;q=0.8";

    try {
        //create http connection
        URL obj = new URL(url);
        HttpsURLConnection connection = (HttpsURLConnection) obj.openConnection();

        //add request header
        connection.setRequestMethod("POST");
        connection.setDoInput(true);
        connection.setDoOutput(true);
        connection.setUseCaches(false);
        connection.setRequestProperty("Accept-Language", ACCEPT_LANGUAGE);
        connection.setRequestProperty("Content-Type", CONTENT_TYPE);
        connection.setRequestProperty("Content-Length", CONTENT_LENGTH);

        DataOutputStream output = new DataOutputStream(connection.getOutputStream());

        //form data
        String content = "documentId=3896&action=getAnnotationModel&annotationLayer=1&pageCount=1&pageIndex=0";

        //write output stream and close output stream
        output.writeBytes(content);
        output.flush();
        output.close();

        //read in the response data
        BufferedReader input = new BufferedReader(new InputStreamReader(connection.getInputStream()));
        String inputLine;
        while((inputLine = input.readLine()) != null) {
            response.append(inputLine.toString());
        }

        //close input stream
        input.close();

        //print out content
        int responseCode = connection.getResponseCode();
        System.out.println("response code: " + responseCode);
        System.out.println("respone is: " + response);

    } catch (MalformedURLException e) {
        e.printStackTrace();

    } catch (IOException e) {
        e.printStackTrace();
    }

    return response;
}

Answer 2

Make use of DocumentBuilder:-

DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = dbf.newDocumentBuilder();
Document doc = db.parse(new URL(url).openStream());

For printing XML :-

DOMSource domSource = new DOMSource(doc);
StringWriter writer = new StringWriter();
StreamResult result = new StreamResult(writer);
TransformerFactory tf = TransformerFactory.newInstance();
Transformer transformer = tf.newTransformer();
transformer.setOutputProperty(OutputKeys.INDENT, "yes"); //pretty printing
transformer.transform(domSource, result);
System.out.println("XML IN String format is: \n" + writer.toString());