how to resolve jsoup error: unable to find valid certification path to requested target

Question

I am trying to parse the html of the following URL:

https://www.smuc.ac.kr/mbs/smuc/jsp/board/list.jsp?boardId=6993&id=smuc_040100000000

I'm getting the following error:

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
    at sun.security.validator.Validator.validate(Validator.java:260)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491)
    ... 15 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
    ... 21 more

This is my code:

public class MainActivity extends AppCompatActivity {
    private ListView listView;
    private TextView textView;
    public ArrayList<String> arrayList = new ArrayList<String>();
    private ArrayAdapter<String> arrayAdapter;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        listView = (ListView) findViewById(R.id.listView);
        new Insert().execute();
        arrayAdapter = new ArrayAdapter<String>(MainActivity.this, R.layout.list_ok, R.id.text, arrayList );
       }

    class Insert extends AsyncTask<String, Void, String> {
        @Override
        protected String doInBackground(String... params) {
            try {
               // Connection.Response res = Jsoup.connect("https://www.smuc.ac.kr/mbs/smuc/index.jsp")
               //         .method(Connection.Method.POST)
               //         .execute();
                Document document = Jsoup.connect("https://www.smuc.ac.kr/mbs/smuc/jsp/board/list.jsp?boardId=6993&id=smuc_040100000000").get();
                Elements elements = document.select(".tit");
                arrayList.clear();
                for (Element element : elements) {
                    arrayList.add(element.text());
                }
            } catch (IOException e) {
                e.printStackTrace();
            }
            return null;
        }
        @Override
        protected void onPostExecute(String result){
            listView.setAdapter(arrayAdapter);
        }

    }

}

Answer 1

The selected answer will not work with latest releases of JSoup as validateTLSCertificates is deprecated and removed. I have created the following helper class :

public class SSLHelper {

    static public Connection getConnection(String url){
        return Jsoup.connect(url).sslSocketFactory(SSLHelper.socketFactory());
    }

    static private SSLSocketFactory socketFactory() {
        TrustManager[] trustAllCerts = new TrustManager[]{new X509TrustManager() {
            public java.security.cert.X509Certificate[] getAcceptedIssuers() {
                return new X509Certificate[0];
            }

            public void checkClientTrusted(X509Certificate[] certs, String authType) {
            }

            public void checkServerTrusted(X509Certificate[] certs, String authType) {
            }
        }};

        try {
            SSLContext sslContext = SSLContext.getInstance("SSL");
            sslContext.init(null, trustAllCerts, new java.security.SecureRandom());
            SSLSocketFactory result = sslContext.getSocketFactory();

            return result;
        } catch (NoSuchAlgorithmException | KeyManagementException e) {
            throw new RuntimeException("Failed to create a SSL socket factory", e);
        }
    }
}

Then I simply call it as follows:

Document doc = SSLHelper.getConnection(url).userAgent(USER_AGENT).get();

(*) - https://dzone.com/articles/how-setup-custom - helpful in coming up with the solution

Answer 2

Note: JSoup has deprecated and removed the validateTLSCertificates method in version 1.12.1. See this answer for an alternative solution.

---

Prior to JSoup version 1.12.1, ignore TLS validation as follows:

Document doc = Jsoup.connect("URL").timeout(10000).validateTLSCertificates(false).get();

Since reading the page also takes a while, increase the timeout timeout(10000).

Answer 3

Following George's comment, I will post this as an answer.
I made a Kotlin version for checklist's solution, as follows:

package crawlers

import java.security.KeyManagementException

import java.security.NoSuchAlgorithmException

import java.security.SecureRandom
import java.security.cert.X509Certificate
import javax.net.ssl.SSLContext

import javax.net.ssl.SSLSocketFactory
import javax.net.ssl.TrustManager
import javax.net.ssl.X509TrustManager

class SSLHelperKotlin {

    companion object {

        @JvmStatic
        fun socketFactory(): SSLSocketFactory {
            val trustAllCerts = arrayOf<TrustManager>(
                    object : X509TrustManager {
                        override fun getAcceptedIssuers() = arrayOf<X509Certificate>()
                        override fun checkClientTrusted(certs: Array<X509Certificate>, authType: String) {}
                        override fun checkServerTrusted(certs: Array<X509Certificate>, authType: String) {}
                    }
            )

            return try {
                val sslContext: SSLContext = SSLContext.getInstance("SSL")
                sslContext.init(null, trustAllCerts, SecureRandom())
                sslContext.socketFactory
            } catch (e: NoSuchAlgorithmException) {
                throw RuntimeException("Failed to create a SSL socket factory", e)
            } catch (e: KeyManagementException) {
                throw RuntimeException("Failed to create a SSL socket factory", e)
            }
        }
    }
}

It's also available on this gist.