Why does validation fail when connecting to a server by IP address instead of hostname?

Question

I am getting the bad certificate error while accessing the server using IP address instead DNS name.

Is this functionality newly introduced in tls1.1. and tls 1.2? It would be good if someone would point out OpenSSL code where it fails and return the bad certificate error.

Answer 1

Why do we get bad certificate error while accessing the server using IP address instead dns name?

It depends on the issuing/validation policies, user agents, and the version of OpenSSL you are using. So to give you a precise answer, we need to know more about your configuration.

Generally speaking, suppose www.example.com has a IP address of www.xxx.yyy.zzz. If you connect via https://www.example.com/..., then the connection should succeed. If you connect using a browser via https://www.xxx.yyy.zzz/... then it should always fail. If you connect using another user agent via https://www.xxx.yyy.zzz/... then it should succeed if the certificate includes www.xxx.yyy.zzz; and fail otherwise.

Issuing/Validation Policies

There are two bodies which dominate issuing/validation policies. They are the CA/Browser Forum, and the Internet Engineering Task Force (IETF).

Browsers, Like Chrome, Firefox and Internet Explorer, follow the CA/B Baseline Requirements (CA/B BR).

Other user agents, like cURL and Wget, follow IETF issuing and validation policies, like RFC 5280, Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile and RFC 6125, Representation and Verification of Domain-Based Application Service Identity within Internet Public Key Infrastructure Using X.509 (PKIX) Certificates in the Context of Transport Layer Security (TLS). The RFCs are more relaxed that CA/B issuing policies.

User Agents

Different user agents have different policies that apply to DNS names. Some want a traditional hostname found in DNS, while others allow IP addresses.

Browsers only allow DNS hostnames in the Subject Alternate Name (SAN). If the hostname is missing from the SAN, then the match will not occur. Putting the server name in the Common Name is a waste of time and energy because browsers require host names in the SAN.

Browsers do not match a public IP address in the SAN. They will sometimes allow a Private IP from RFC 1918, Address Allocation for Private Internets.

Other user agents allow any name in the Subject Alternate Name (SAN). They also will match a name in both the Common Name (CN) and the Subject Alternate Name (SAN). Names include a DNS name like www.example.com, a public IP address, a private IP address like 192.168.10.10 and a local name like localhost and localhost.localdomain.

OpenSSL Version

OpenSSL version 1.0.2 and below did not perform hostname validation. That is, you had to perform the matching yourself. If you did not perform hostname validation yourself, then it appeared the connection always succeeded. Also see Hostname Validation and TLS Client on the OpenSSL wiki.

OpenSSL 1.1.0 and above perform hostname matching. If you switch to 1.1.0, then you should begin experiencing failures if you were not performing hostname matching youself or you were not strictly following issuing policies.

---

It would be good if someone would point out OpenSSL code where it fails and return the bad certificate error.

The check-ins occurred in early-2015, and they have been available in Master (i.e., 1.1.0-dev) since that time. The code was also available in 1.0.2, but you had to perform special actions. The routines were not available in 1.0.1 or below. Also see Hostname Validation on the OpenSSL wiki. I don't have the Git check-ins because I'm on a Windows machine at the moment.

---

More information of the rules for names and their locations can be found at How do you sign Certificate Signing Request with your Certification Authority and How to create a self-signed certificate with openssl. There are at least four or six more documents covering them, like how things need to be presented for HTTP Strict Transport Security (HSTS) and Public Key Pinning with Overrides for HTTP.