CODEWITHSUNDEEP
javaencodinghttpurlconnection
ton1
ton1

Reputation: 7628

HttpURLConnection Character Encoding

I'm making a simple program. There is a url with using charset "utf-8". I want to get whole source from this page, but there is character encoding problem. 

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;

class WholeTest {

    HttpURLConnection conn;


    public void openUrl() throws Exception {

        URL pageUrl = new URL("http://naver.com");
        conn = (HttpURLConnection)pageUrl.openConnection();
        conn.setRequestMethod("GET");
        conn.setUseCaches(false);

        conn.setRequestProperty("Host", "naver.com");
        conn.setRequestProperty("Connection", "keep-alive");
        conn.setRequestProperty("Cache-Control", "max-age=0");
        conn.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
        conn.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36");
        conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
        conn.setRequestProperty("Accept-Language", "ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4");


        BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream(),"utf-8"));
        String inputLine;
        StringBuffer response = new StringBuffer();

        while ((inputLine = in.readLine()) != null) {
            response.append(inputLine);
        }
        in.close();

        System.out.println("---result---");
        System.out.println(response.toString());
    }
}


public class Whole {

    public static void main(String args[]) throws Exception {
        System.out.print("Test");

        WholeTest w = new WholeTest();
        w.openUrl();
    }

}

The result is : ?????????????????????????????????????????????????? I can't view the source. I used charset "utf-8" when reading inputStream, what I did incorrectly?

I use all utf-8, UTF-8, euc-kr, EUC-KR ... same result.

Upvotes: 1

Views: 4446

Answers (1)

shazin
shazin

Reputation: 21883

Just as I suspected, comment or remove the below line. It will work like a charm.

conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");

You are expecting gzip binary when the return is actually text/html

Upvotes: 3

Related Questions