OpenSSL Handshake failure with two sessions and directly connected memory BIO's C#

Question

I'm having a weird trouble with OpenSSL in C#.

Facts: I've programmed a wrapper for C# that is working completely fine, it could connect to other SSL servers, or accept connections from SSL clients. My wrapper is using memory BIO's only. Anytime I connect the MEM BIO to in and outputs of any type socket of socket handshake is successful, data transfer is working, so the problem is not in the wrapper.

I've made two programs with this wrapper, a client and a server, they estabilish the ssl session, and when I try to send data from the server to the client (or opposite way) and the other side echoes it back to the sender for some reason the data traffic is very slow (max 20kB/s even in local loopback, if I try send with higher speed like pumping more data in the SSL_read() then with TLS on TCP there are huge delays, with DTLS on UDP there is sometimes 75% of packet loss), so i've decided to benchmark the OpenSSL with the wrapper, because some methods or wrong parameters may slow down the running.

My idea was to init two SSL contexts in one program, one client and one server, connect they Memory BIO's to each other (RX-TX TX-RX), and start a handshake.

I just can't make them finish the handshaking successfully!

With DTLS1 after Clienthello, the server starts to respond with Serverhello and Cert data in 4 packets (3x256 bytes + 145), but after the first packet the client alerts:

After writing to CLIENT (packet: 2): 22016:error:14102044:SSL routines:dtls1_read_bytes:internal error:.\ssl\d1_pkt.c:1370:
After writing to CLIENT (packet: 3): 22016:error:14102044:SSL routines:dtls1_read_bytes:internal error:.\ssl\d1_pkt.c:1370:
After writing to CLIENT (packet: 4): 22016:error:14102044:SSL routines:dtls1_read_bytes:internal error:.\ssl\d1_pkt.c:1370:
After writing to CLIENT (packet: 5): 22016:error:14102044:SSL routines:dtls1_read_bytes:internal error:.\ssl\d1_pkt.c:1370:

The packet flow between the two sessions is the following:

Client -> 178 -> Server
Server -> 256 -> Client
Server -> 256 -> Client
Server -> 256 -> Client
Server -> 145 -> Client
Client -> 15 -> Server

I've analyzed these packets (after file dump) with Wireshark, and everything seems OK (this analysis is not of the packets above, but same reproduction of the problem): http://reset.tor.hu/ssl/sima/programom_dtls1.txt

After this i get more errors:

After writing to SERVER (packet: 5): 22016:error:141020E5:SSL routines:dtls1_read_bytes:ssl handshake failure:.\ssl\d1_pkt.c:819:
Before writing to SERVER (packet: 6): 22016:error:140FD10F:SSL routines:DTLS1_GET_MESSAGE_FRAGMENT:bad length:.\ssl\d1_both.c:892:
After writing to SERVER (packet: 6): 22016:error:14102417:SSL routines:dtls1_read_bytes:sslv3 alert illegal parameter:.\ssl\d1_pkt.c:1200:SSL alert number 47
22016:error:141020E5:SSL routines:dtls1_read_bytes:ssl handshake failure:.\ssl\d1_pkt.c:819:

My only usable idea was to debug this problem to dump the packets to files right after they have been read from the BIO and right before they get written in the BIO in the other session, but I've found no any difference in the content after byte-comparison, and the packet sequence was OK, too.

File dumps are here: http://reset.tor.hu/ssl/y/direct_bio_error/

I just couldn't imagine what could be the problem because they have never finished handshake without failure, so I've decided to add two UDP sockets to this program, and connect the sockets to the BIO's and to each other, and not surprisingly the handshake was successful for the first time.

File dumps are here: http://reset.tor.hu/ssl/y/socket_ok/

As it is visible there is nothing difference in the file contents or the packet sequences.

Then why are they not working with BIO's directly connected?!

You can see the program producing the error and the normal working in this video: https://www.youtube.com/watch?v=4crbSz6JMMY

My only idea was C# for some reason mixing the method calls in the time but i'm not using any threading! The unmanaged code calls a callback to read out BIO, I read out BIO to my buffer, I call another method that writes the buffer in the BIO and call SSL_read() in the unmanaged code, to use the data in the BIO. And this is exactly in the other way. If i'm not using the buffer, but the sockets instead it's working. What the sockets can do that makes it working?

I've even tried to put Thread.Sleep(100) between the BIO writings, but didn't help, actually over 140ms OpenSSL crashed (but with sockets it worked even with 1000ms, even more weird).

What could be the problem and solution for the direct BIO and the slow traffic problem?

Thank You for the answers in advance!

Answer 1

Ok, actually it seems i have just solved the Direct-Memory-BIO-Handshake problem.

It is suggested to call SSL_read() after BIO_write(), otherwise OpenSSL won't check for data in the READ BIO and will not process it, so almost everytime people use it directly after writing to BIO.

If one configured the OpenSSL context as CLIENT then one should NOT use SSL_read() after BIO_write() directly until handshaking not finished!

If OpenSSL is used with socket (even with memory bio's) it is working if we use SSL_read() directly after BIO_write(), but with direct BIO connection between two sessions NOT. At least as unmanaged code in C#!

The BIO writing method that works fine is the following:

    public bool BIO_Write(byte[] data)
    {
        if (Inited & data.Length > 0)
        {
            Wrapper.BIO_write(BIOR, data, data.Length);
            if (isServer(SSL_method))
            {
                // For SERVER MODE
                if (Wrapper.SSL_is_init_finished(SSL))
                {
                    readFromSSL();
                }
                else
                {
                    Wrapper.SSL_do_handshake(SSL);
                }               
            }
            else
            {
                // For CLIENT MODE
                if (Wrapper.SSL_is_init_finished(SSL))
                {
                    readFromSSL();
                }
                else
                {
                    // It is extremely important to call the SSL_do_handshake()
                    // only if SSL_want() returns 3, otherwise OpenSSL will
                    // throw a FATAL ERROR and crash with direct BIO connection
                    // return value 3 is SSL_ERROR_WANT_WRITE
                    if (Wrapper.SSL_want(SSL) == 3)
                    {
                        Wrapper.SSL_do_handshake(SSL);
                    }
                }
            }
            return true;
        }
        else
        {
            return false;
        }
    }

I hope this will help someone!