Jsoup: ignore SSL errors

Question

I'm trying to download https://www.deviantart.com using Jsoup (v1.10.3), with validateTLSCertificates(false). Java 8, UnlimitedJCEPolicyJDK8 installed.

The exception thrown is fatal, handshake_failure. After setting -Djavax.net.debug=all we receive more details: RECV TLSv1.2 ALERT: fatal, handshake_failure.

Also tried System.setProperty("https.protocols", "TLSv1,TLSv1.1,TLSv1.2"); (suggested here getting javax.net.ssl.SSLException: Received fatal alert: protocol_version while scraping data using Jsoup), and the "trust manager that does not validate certificate chains". It didn't help.

I can't believe in 2017 it's so hard for Java to download a simple URL. I just want a piece of code that downloads any given HTTPS URL while ignoring any SSL errors, without adding certificates to the trusted store or anything like that.

Thanks!

Full output for the SSL dump:

trigger seeding of SecureRandom
done seeding SecureRandom
trigger seeding of SecureRandom
done seeding SecureRandom
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
pool-1-thread-1, setSoTimeout(20000) called
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for TLSv1.1
%% No cached client session
*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1489235422 bytes = { 95, 240, 147, 176, 192, 176, 220, 188, 41, 118, 230, 166, 80, 191, 204, 5, 32, 161, 184, 60, 42, 112, 96, 165, 83, 70, 37, 105 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA256withDSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA
***
[write] MD5 and SHA1 hashes:  len = 203
0000: 01 00 00 C7 03 03 59 C4   EE DE 5F F0 93 B0 C0 B0  ......Y..._.....
0010: DC BC 29 76 E6 A6 50 BF   CC 05 20 A1 B8 3C 2A 70  ..)v..P... ..<*p
0020: 60 A5 53 46 25 69 00 00   64 C0 24 C0 28 00 3D C0  `.SF%i..d.$.(.=.
0030: 26 C0 2A 00 6B 00 6A C0   0A C0 14 00 35 C0 05 C0  &.*.k.j.....5...
0040: 0F 00 39 00 38 C0 23 C0   27 00 3C C0 25 C0 29 00  ..9.8.#.'.<.%.).
0050: 67 00 40 C0 09 C0 13 00   2F C0 04 C0 0E 00 33 00  g.@...../.....3.
0060: 32 C0 2C C0 2B C0 30 00   9D C0 2E C0 32 00 9F 00  2.,.+.0.....2...
0070: A3 C0 2F 00 9C C0 2D C0   31 00 9E 00 A2 C0 08 C0  ../...-.1.......
0080: 12 00 0A C0 03 C0 0D 00   16 00 13 00 FF 01 00 00  ................
0090: 3A 00 0A 00 16 00 14 00   17 00 18 00 19 00 09 00  :...............
00A0: 0A 00 0B 00 0C 00 0D 00   0E 00 16 00 0B 00 02 01  ................
00B0: 00 00 0D 00 16 00 14 06   03 06 01 05 03 05 01 04  ................
00C0: 03 04 01 04 02 02 03 02   01 02 02                 ...........
pool-1-thread-1, WRITE: TLSv1.2 Handshake, length = 203
[Raw write]: length = 208
0000: 16 03 03 00 CB 01 00 00   C7 03 03 59 C4 EE DE 5F  ...........Y..._
0010: F0 93 B0 C0 B0 DC BC 29   76 E6 A6 50 BF CC 05 20  .......)v..P... 
0020: A1 B8 3C 2A 70 60 A5 53   46 25 69 00 00 64 C0 24  ..<*p`.SF%i..d.$
0030: C0 28 00 3D C0 26 C0 2A   00 6B 00 6A C0 0A C0 14  .(.=.&.*.k.j....
0040: 00 35 C0 05 C0 0F 00 39   00 38 C0 23 C0 27 00 3C  .5.....9.8.#.'.<
0050: C0 25 C0 29 00 67 00 40   C0 09 C0 13 00 2F C0 04  .%.).g.@...../..
0060: C0 0E 00 33 00 32 C0 2C   C0 2B C0 30 00 9D C0 2E  ...3.2.,.+.0....
0070: C0 32 00 9F 00 A3 C0 2F   00 9C C0 2D C0 31 00 9E  .2...../...-.1..
0080: 00 A2 C0 08 C0 12 00 0A   C0 03 C0 0D 00 16 00 13  ................
0090: 00 FF 01 00 00 3A 00 0A   00 16 00 14 00 17 00 18  .....:..........
00A0: 00 19 00 09 00 0A 00 0B   00 0C 00 0D 00 0E 00 16  ................
00B0: 00 0B 00 02 01 00 00 0D   00 16 00 14 06 03 06 01  ................
00C0: 05 03 05 01 04 03 04 01   04 02 02 03 02 01 02 02  ................
[Raw read]: length = 5
0000: 15 03 03 00 02                                     .....
[Raw read]: length = 2
0000: 02 28                                              .(
pool-1-thread-1, READ: TLSv1.2 Alert, length = 2
pool-1-thread-1, RECV TLSv1.2 ALERT:  fatal, handshake_failure
pool-1-thread-1, called closeSocket()
pool-1-thread-1, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
pool-1-thread-1, called close()
pool-1-thread-1, called closeInternal(true)

Answer 1

User @dave_thompson_085 clarified everything, turns out it's a bug in Java 8 versions prior to 8u141, where it wouldn't send SNI.

Solution: make sure you have JRE/JDK >= 141.