CODEWITHSUNDEEP
javasslrabbitmq
Toryu
Toryu

Reputation: 635

RabbitMQ SSL handshake failure on any connection attempt with certificate authentication

I'm currently trying to migrate some Java components at to have RabbitMQ connection authentication be performed through SSL client certificates instead of the PLAIN method but, after several days, am still struggling with it as all java component connection attempts are met with handshake errors. Looking into RabbitMQ: handshake error when attempting to use SSL certificates or RabbitMQ SSL giving handshake failure when using SpringAMQP has unfortunately not wielded any results for me.

The environment I'm trying to make this work is an "inherited" VirtualBox VM running Ubuntu LTS 14.04 that represents pretty much the environment I want to deploy to.

The output of rabbitmqctl report is as follows:

Status of node 'rabbit@developer-VirtualBox' ...
[{pid,23352},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.4.2"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.4.2"},
      {webmachine,"webmachine","1.10.3-rmq3.4.2-gite9359c7"},
      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.4.2-git680dba8"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.4.2"},
      {rabbit,"RabbitMQ","3.4.2"},
      {ssl,"Erlang/OTP SSL application","5.3.2"},
      {public_key,"Public key infrastructure","0.21"},
      {crypto,"CRYPTO version 2","3.2"},
      {asn1,"The Erlang ASN1 compiler version 2.0.4","2.0.4"},
      {os_mon,"CPO  CXC 138 46","2.2.14"},
      {inets,"INETS  CXC 138 49","5.9.7"},
      {rabbitmq_auth_mechanism_ssl,
          "RabbitMQ SSL authentication (SASL EXTERNAL)","3.4.2"},
      {amqp_client,"RabbitMQ AMQP Client","3.4.2"},
      {xmerl,"XML parser","1.3.5"},
      {mnesia,"MNESIA  CXC 138 12","4.11"},
      {sasl,"SASL  CXC 138 11","2.3.4"},
      {stdlib,"ERTS  CXC 138 10","1.19.4"},
      {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang R16B03 (erts-5.10.4) [source] [64-bit] [async-threads:30] [kernel-poll:true]\n"},
 {memory,
     [{total,42662672},
      {connection_readers,0},
      {connection_writers,0},
      {connection_channels,0},
      {connection_other,5264},
      {queue_procs,2632},
      {queue_slave_procs,0},
      {plugins,411368},
      {other_proc,14374616},
      {mnesia,59360},
      {mgmt_db,124224},
      {msg_index,34312},
      {other_ets,1135040},
      {binary,42680},
      {code,21795549},
      {atom,793505},
      {other_system,3884122}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"},{'amqp/ssl',5671,"::"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,1658211532},
 {disk_free_limit,50000000},
 {disk_free,43703377920},
 {file_descriptors,
     [{total_limit,924},{total_used,4},{sockets_limit,829},{sockets_used,2}]},
 {processes,[{limit,1048576},{used,191}]},
 {run_queue,0},
 {uptime,12}]

Cluster status of node 'rabbit@developer-VirtualBox' ...
[{nodes,[{disc,['rabbit@developer-VirtualBox']}]},
 {running_nodes,['rabbit@developer-VirtualBox']},
 {cluster_name,<<"rabbit@developer-VirtualBox">>},
 {partitions,[]}]

Application environment of node 'rabbit@developer-VirtualBox' ...
[{amqp_client,[{prefer_ipv6,false},{ssl_options,[]}]},
 {asn1,[]},
 {crypto,[]},
 {inets,[]},
 {kernel,
     [{error_logger,tty},
      {inet_default_connect_options,[{nodelay,true}]},
      {inet_dist_listen_max,25672},
      {inet_dist_listen_min,25672}]},
 {mnesia,[{dir,"/var/lib/rabbitmq/mnesia/rabbit@developer-VirtualBox"}]},
 {mochiweb,[]},
 {os_mon,
     [{start_cpu_sup,false},
      {start_disksup,false},
      {start_memsup,false},
      {start_os_sup,false}]},
 {public_key,[]},
 {rabbit,
     [{auth_backends,[rabbit_auth_backend_internal]},
      {auth_mechanisms,['EXTERNAL']},
      {backing_queue_module,rabbit_variable_queue},
      {channel_max,0},
      {cluster_keepalive_interval,10000},
      {cluster_nodes,{[],disc}},
      {cluster_partition_handling,ignore},
      {collect_statistics,fine},
      {collect_statistics_interval,5000},
      {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
      {default_user,<<"guest">>},
      {default_user_tags,[administrator]},
      {default_vhost,<<"/">>},
      {delegate_count,16},
      {disk_free_limit,50000000},
      {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
      {error_logger,
          {file,"/var/log/rabbitmq/[email protected]"}},
      {frame_max,131072},
      {halt_on_upgrade_failure,true},
      {handshake_timeout,10000},
      {heartbeat,580},
      {hipe_compile,false},
      {hipe_modules,
          [rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
           rabbit_command_assembler,rabbit_framing_amqp_0_9_1,rabbit_basic,
           rabbit_event,lists,queue,priority_queue,rabbit_router,rabbit_trace,
           rabbit_misc,rabbit_binary_parser,rabbit_exchange_type_direct,
           rabbit_guid,rabbit_net,rabbit_amqqueue_process,
           rabbit_variable_queue,rabbit_binary_generator,rabbit_writer,
           delegate,gb_sets,lqueue,sets,orddict,rabbit_amqqueue,
           rabbit_limiter,gb_trees,rabbit_queue_index,
           rabbit_exchange_decorator,gen,dict,ordsets,file_handle_cache,
           rabbit_msg_store,array,rabbit_msg_store_ets_index,rabbit_msg_file,
           rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
           mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
           ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,ssl]},
      {log_levels,[{connection,info}]},
      {loopback_users,[<<"guest">>]},
      {mnesia_table_loading_timeout,30000},
      {msg_store_file_size_limit,16777216},
      {msg_store_index_module,rabbit_msg_store_ets_index},
      {plugins_dir,
          "/usr/lib/rabbitmq/lib/rabbitmq_server-3.4.2/sbin/../plugins"},
      {plugins_expand_dir,
          "/var/lib/rabbitmq/mnesia/rabbit@developer-VirtualBox-plugins-expand"},
      {queue_index_max_journal_entries,65536},
      {reverse_dns_lookups,false},
      {sasl_error_logger,
          {file,"/var/log/rabbitmq/[email protected]"}},
      {server_properties,[]},
      {ssl_allow_poodle_attack,false},
      {ssl_apps,[asn1,crypto,public_key,ssl]},
      {ssl_cert_login_from,distinguished_name},
      {ssl_handshake_timeout,5000},
      {ssl_listeners,[5671]},
      {ssl_options,
          [{cacertfile,"/home/developer/rabbitmqcert/devcafiles/cacert.pem"},
           {certfile,"/home/developer/rabbitmqcert/rabbitmq.public.pem"},
           {keyfile,"/home/developer/rabbitmqcert/rabbitmq.private.pem"},
           {verify,verify_peer},
           {ssl_cert_login_from,organization},
           {fail_if_no_peer_cert,true}]},
      {tcp_listen_options,
          [binary,
           {packet,raw},
           {reuseaddr,true},
           {backlog,128},
           {nodelay,true},
           {linger,{true,0}},
           {exit_on_close,false}]},
      {tcp_listeners,[5672]},
      {trace_vhosts,[]},
      {vm_memory_high_watermark,0.4},
      {vm_memory_high_watermark_paging_ratio,0.5}]},
 {rabbitmq_auth_mechanism_ssl,[{name_from,distinguished_name}]},
 {rabbitmq_management,
     [{http_log_dir,none},
      {listener,[{port,15672}]},
      {load_definitions,none},
      {rates_mode,basic},
      {sample_retention_policies,
          [{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
           {basic,[{605,5},{3600,60}]},
           {detailed,[{10,5}]}]}]},
 {rabbitmq_management_agent,[]},
 {rabbitmq_web_dispatch,[]},
 {sasl,[{errlog_type,error},{sasl_error_logger,false}]},
 {ssl,
     [{protocol_version,['tlsv1.2','tlsv1.1',tlsv1,sslv3]},
      {versions,['tlsv1.2','tlsv1.1']}]},
 {stdlib,[]},
 {webmachine,[{error_handler,rabbit_webmachine_error_handler}]},
 {xmerl,[]}]

Connections:

Channels:

Queues on /:

Exchanges on /:
name    type    durable auto_delete internal    arguments   policy
    direct  true    false   false   []  
amq.direct  direct  true    false   false   []  
amq.fanout  fanout  true    false   false   []  
amq.headers headers true    false   false   []  
amq.match   headers true    false   false   []  
amq.rabbitmq.log    topic   true    false   true    []  
amq.rabbitmq.trace  topic   true    false   true    []  
amq.topic   topic   true    false   false   []  

Bindings on /:

Consumers on /:

Permissions on /:
user    configure   write   read
O=dev,CN=rules  .*  .*  .*
guest   .*  .*  .*

Policies on /:

Parameters on /:

Going through it I can see that:

For simplicity (and sanity), I've tried connecting to the RabbitMQ instance using the following Java code:

package com.rabbitmq.sample;

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.security.KeyManagementException;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.UnrecoverableKeyException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;

import javax.net.ssl.KeyManager;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.DefaultSaslConfig;
import com.rabbitmq.client.QueueingConsumer;

public class CertificateAuthenticatedRabbitMQClientExample {

    private static final String CLIENT_CERTIFICATE_PASSWORD = "MySecretPassword";
    private static final String QUEUE_USED = "sampleQueue";
    public static void main(String[] args) throws UnrecoverableKeyException, NoSuchAlgorithmException, KeyStoreException, IOException, CertificateException, KeyManagementException, InterruptedException {
        SSLContext sslContext = SSLContext.getInstance("TLSv1.2");
        KeyManager[] clientKeyManagerList = null;
        try(FileInputStream clientCertificateInputStream = new FileInputStream(new File("/home/developer/rabbitmqcert/ruleprocessing.password.p12"))) {
            KeyStore clientKeStore = KeyStore.getInstance("PKCS12");                                        //Create a clean KeyStore
            clientKeStore.load(clientCertificateInputStream, CLIENT_CERTIFICATE_PASSWORD.toCharArray());    //Load the client's certificate into the keystore
            KeyManagerFactory clientSSLKeyManagerFactory = KeyManagerFactory.getInstance("SunX509");
            clientSSLKeyManagerFactory.init(clientKeStore, CLIENT_CERTIFICATE_PASSWORD.toCharArray());
            clientKeyManagerList = clientSSLKeyManagerFactory.getKeyManagers();                             //Get list of key managers (in essence, only the keystore with the client certificate)
        }
        TrustManager[] clientTrustManagerList = {
          new X509TrustManager() {
              //Dummy trust store that trusts any server you connect to.
              //For demo purposes only
              @Override
              public void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}
              @Override
              public void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}
              @Override
              public X509Certificate[] getAcceptedIssuers() { return null;}
          }
        };
        sslContext.init(clientKeyManagerList, clientTrustManagerList, null);        //Initialize SSL context with the key and trust managers we've created/loaded before

        ConnectionFactory rabbitMqConnectionFactory = new ConnectionFactory();      //Create factory
        rabbitMqConnectionFactory.setHost("localhost");
        rabbitMqConnectionFactory.setPort(5671);
        rabbitMqConnectionFactory.setSaslConfig(DefaultSaslConfig.EXTERNAL);        //Set authentication method as SSL auth
        rabbitMqConnectionFactory.useSslProtocol(sslContext);                       //Set the created SSL context as the one to use
        Connection rabbitMqOutboundConnection = null;
        Channel rabbitMqOutboundChannel = null;
        try {
            rabbitMqOutboundConnection = rabbitMqConnectionFactory.newConnection();
            rabbitMqOutboundChannel = rabbitMqOutboundConnection.createChannel();
            rabbitMqOutboundChannel.queueDeclare(QUEUE_USED, false, false, false, null);
            rabbitMqOutboundChannel.basicPublish("", QUEUE_USED, null, "This is a sample message".getBytes());
            System.out.println("Message successfully sent to queue");
        }finally{
            if(rabbitMqOutboundChannel != null) {
                rabbitMqOutboundChannel.close();
            }
            if(rabbitMqOutboundConnection != null) {
                rabbitMqOutboundConnection.close();
            }
        }
        Connection rabbitMqInboundConnection = null;
        Channel rabbitMqInboundChannel = null;
        try {
            rabbitMqInboundConnection = rabbitMqConnectionFactory.newConnection();
            rabbitMqInboundChannel = rabbitMqInboundConnection.createChannel();
            rabbitMqInboundChannel.queueDeclare(QUEUE_USED, false, false, false, null);
            QueueingConsumer rabbitMqQueueConsumer = new QueueingConsumer(rabbitMqInboundChannel);
            rabbitMqInboundChannel.basicConsume(QUEUE_USED, true, rabbitMqQueueConsumer);
            QueueingConsumer.Delivery deliveryResult = rabbitMqQueueConsumer.nextDelivery();
            System.out.println("Message read from the queue: " + new String(deliveryResult.getBody()));
        }finally{
            if(rabbitMqInboundChannel != null) {
                rabbitMqInboundChannel.close();
            }
            if(rabbitMqInboundConnection != null) {
                rabbitMqInboundConnection.close();
            }
        }
    }

}

I am, however, always greeted with a Exception in thread "main" java.net.SocketException: Broken pipe error on execution and, looking at the RabbitMQ server log, I see:

=INFO REPORT==== 29-Dec-2014::15:55:30 ===
accepting AMQP connection <0.644.0> (127.0.0.1:35299 -> 127.0.0.1:5671)

=ERROR REPORT==== 29-Dec-2014::15:55:30 ===
SSL: certify: ssl_handshake.erl:1343:Fatal error: handshake failure

Based on what I've seen around the internet and here, I've tried changing the verify value to verify_none but, when doing do, I get the following instead:

=INFO REPORT==== 29-Dec-2014::14:51:13 ===
accepting AMQP connection <0.311.0> (127.0.0.1:35271 -> 127.0.0.1:5671)

=ERROR REPORT==== 29-Dec-2014::14:51:17 ===
closing AMQP connection <0.311.0> (127.0.0.1:35271 -> 127.0.0.1:5671):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "EXTERNAL login refused: no peer certificate",
                             'connection.start_ok'}}

As per the advice given under the SSL troubleshooting page, I have tried performing an s_client connection to the server with, once again, different results depending on the value of verify. If verify is set to verify_peer, I get the following:

developer@developer-VirtualBox:~/rabbitmqcert$ openssl s_client -tls1_2 -connect localhost:5671 -cert ruleprocessing.public.pem -key ruleprocessing.private.pem -CAfile devcafiles/cacert.pem 
CONNECTED(00000003)
depth=1 CN = RabbitMQCA
verify return:1
depth=0 CN = rabbitmq, O = dev
verify return:1
139797055162016:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1260:SSL alert number 40
139797055162016:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:
---
Certificate chain
 0 s:/CN=rabbitmq/O=dev
   i:/CN=RabbitMQCA
 1 s:/CN=RabbitMQCA
   i:/CN=RabbitMQCA
---
Server certificate
-----BEGIN CERTIFICATE-----
<<OMMITED>>
-----END CERTIFICATE-----
subject=/CN=rabbitmq/O=dev
issuer=/CN=RabbitMQCA
---
Acceptable client certificate CA names
/CN=RabbitMQCA
---
SSL handshake has read 1646 bytes and written 2103 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA256
    Session-ID: FC972B9A5D3EC359DC0467C8F02410E3AD66DA151C4411C0D5892115A439431A
    Session-ID-ctx: 
    Master-Key: E4FE793C71692852F6F3C4E9C5CB17774D8A50511338EF2E75691DC0DC2119F56611FC959C12429BBAFD46EC760ED713
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1419871438
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

As you can see, a handshake error happens at the start but then seems to be recovered.

If verify is set to verify_none, I get the following:

CONNECTED(00000003)
depth=1 CN = RabbitMQCA
verify return:1
depth=0 CN = rabbitmq, O = dev
verify return:1
---
Certificate chain
 0 s:/CN=rabbitmq/O=dev
   i:/CN=RabbitMQCA
 1 s:/CN=RabbitMQCA
   i:/CN=RabbitMQCA
---
Server certificate
-----BEGIN CERTIFICATE-----
<<OMMITED>>
-----END CERTIFICATE-----
subject=/CN=rabbitmq/O=dev
issuer=/CN=RabbitMQCA
---
No client certificate CA names sent
---
SSL handshake has read 1666 bytes and written 663 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA256
    Session-ID: C4156551790BA116DC38A981728A71768D0B53AAEBEE969A4DA150746E5373FB
    Session-ID-ctx: 
    Master-Key: 3470DD0C0247B94EA784C3CEF94888C160205E9F06C14869B564A00AF5E4F7FAF5B4FC977E290B80DBCD140133F75AC0
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1419871570
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

This time, the handshake error does not occur occur at the start.

As a side note, since I am not very sure about "cleanliness" of the VM I got, I've actually tried creating a new VM (same Ubuntu version), installed RabbitMQ (same version), configured it in pretty much the same way (the only thing changing was certificate locations) and ran the same client code (certificate path modified). The final result was a success. Unfortunately I cannot do much with regards to getting this VM to be used as the dev VM at this time.

TL;DR; After configuring a RabbitMQ server running on an Ubuntu VM to accept SSL connections from a Java client using certificate authentication, all connection attempts fail with an handshake failure as reason if verify=verify_peer, or EXTERNAL login refused: no peer certificate if verify=verify_none

Upvotes: 4

Views: 10004

Answers (1)

Toryu
Toryu

Reputation: 635

The problem was related to the way client certificates were signed. They were signed using server_ca_extensions instead of using client_ca_extensions.

Upvotes: 5

Related Questions