CODEWITHSUNDEEP
javaandroid
Mridang Agarwalla
Mridang Agarwalla

Reputation: 44998

Why do I get garbled text when fetching a page using DefaultHTTPClient?

I'm trying to fetch a page using Android's DefaultHTTPClient and parse it using Jsoup. I'm getting a really weird response in which all the HTML within the <body> and </body> tags are encoded into something.

<html>
  <head></head>
  <body>
       ��������������Y�#I�&amp;�\�+��*;����/U���53�*��U�=�D�I:I� ����X�����΃��=H��2�`Ѓ  ��o��nͽ�C瘹;�l2Y�I_l�����;f��W�k��o2.����?�r&gt;��œ�qYξ&lt;&lt;&lt;;;�g*��ѡl���9&gt;s@I��`R��V �c�������Ɂ��e�����,&gt; }���A�����W�?��&quot;.��ˡhޖ�Qy1�oL�_�W�h?9�E?Ofe��KO�Q��(�Av�N�h@��G�qvV�_G��W�g�'q�2�N��L�?�&quot;鳷�x�o�����$9�}/;'#ȸ Q��&amp;�2�\�a��aǔ�L�I�ԯ�=���TPFE� ���:�,�H�N�'QQԯ&lt;&gt;�i}�x��'$�'O ��qy@J�h 2��ᓃ�CH��ʤO���0�LD)��p8�챺)
  </body>
</html>

Here's my method that fetches the page.

  public String doGet(String strUrl, List<NameValuePair> lstParams) throws Exception {

          String strResponse = null;
          HttpGet htpGet = new HttpGet(strUrl);
          //htpGet.addHeader("Accept-Encoding", "gzip, deflate");
          htpGet.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1");
          DefaultHttpClient dhcClient = new DefaultHttpClient();
          PersistentCookieStore pscStore = new PersistentCookieStore(this.objContext);
          dhcClient.setCookieStore(pscStore);
          HttpResponse resResponse = dhcClient.execute(htpGet);
          strResponse = EntityUtils.toString(resResponse.getEntity());
          return strResponse;

  }

Why can this be happening?

If I fetch the page using Jsoup itself, the response is fine. I have to use Jsoup.connect("http://www.kat.ph/").get()

Upvotes: 0

Views: 214

Answers (2)

Mridang Agarwalla
Mridang Agarwalla

Reputation: 44998

It was due to the fact that the response was GZIPped. I wired up a custom response interceptor that uncompressed the response. This is it:

class Decompressor implements HttpResponseInterceptor {

    /*
     * @see org.apache.http.HttpResponseInterceptor#process(org.apache.http.
     * HttpResponse, org.apache.http.protocol.HttpContext)
     */
    public void process(HttpResponse hreResponse, HttpContext hctContext)   throws HttpException, IOException {

        HttpEntity entity = hreResponse.getEntity();

        if (entity != null) {

            Header ceheader = entity.getContentEncoding();

            if (ceheader != null) {

                HeaderElement[] codecs = ceheader.getElements();

                for (int i = 0; i < codecs.length; i++) {

                    if (codecs[i].getName().equalsIgnoreCase("gzip")) {

                        hreResponse.setEntity(new HttpEntityWrapper(entity) {

                            @Override
                            public InputStream getContent() throws IOException, IllegalStateException {

                                return new GZIPInputStream(wrappedEntity.getContent());

                            }

                            @Override
                            public long getContentLength() {

                                return -1;

                            }

                        });

                        return;

                    }

                }

            }

        }

    }

}

Upvotes: 1

Kumar Vivek Mitra
Kumar Vivek Mitra

Reputation: 33534

Try this way....is the result same.....

URL url = new URL("Your_URL");

InputStream is = url.openStream();   // or url.openConnection();

Scanner scan = new Scanner(is);

while(scan.hasNextLine()){

 System.out.println(scan.nextLine());


 }

}

Upvotes: 1

Related Questions