How can I implement server side SMTP STARTTLS?

Question

I am trying to implement a simple SMTP server using Vala and GLib + GIO.

Plain text communication is no problem so far, but when it comes to TLS using STARTTLS things get harder.

This is the code I have so far:

const string appname = "vsmtpd";
const string hostname = "myserver";
const uint16 listenport = 10025;
const string keyfile = "vsmtpd.key";
const string certfile = "vsmtpd.crt";
// TODO: Parse EHLO instead of constant string
const string username = "myclient";

void process_request_plain (InputStream input, OutputStream output) throws Error {
    output.write (@"220 $hostname ESMTP $appname\n".data);
    var data_in = new DataInputStream (input);
    string line;
    while ((line = data_in.read_line (null)) != null) {
        stdout.printf ("%s\n", line);
        line = line.chomp ();
        if (line.substring (0, 5) == "EHLO ") {
            output.write (@"250-$hostname Hello $username\n".data);
            output.write ("250 STARTTLS\n".data);
        }
        else if (line == "STARTTLS") {
            output.write ("220 Go ahead\n".data);
            break;
        }
        else {
            output.write ("502 Command not implemented\n".data);
        }
    }
}

int main () {
    try {
        TlsCertificate cert = new TlsCertificate.from_files
            (certfile, keyfile);
        var service = new SocketService ();
        service.add_inet_port (listenport, null);
        service.start ();
        while (true) {
            SocketConnection conn = service.accept (null);
            process_request_plain (conn.input_stream, conn.output_stream);
            TlsServerConnection tlsconn = TlsServerConnection.@new (conn, cert);
            assert_nonnull (tlsconn);
            // TODO: Is this neccessary?
            tlsconn.accept_certificate.connect ((peer_cert, errors) => {
                stdout.printf ("TLS accepting peer cert\n");
                return true;
            });
            try { 
                tlsconn.handshake ();
                stdout.printf ("TLS handshake ok\n");
            } catch (Error e) {
                stdout.printf ("TLS handshake failed\n");
                stderr.printf ("%s\n", e.message);
            }
        }
    } catch (Error e) {
        stderr.printf ("%s\n", e.message);
    }
    return 0;
}

Given a valid SSL certificate in vsmtpd.key and vsmtpd.crt (which I generated with openssl req -x509 -newkey rsa:2048 -keyout vsmtpd.key -out vsmtpd.pem -days 365 -nodes) I start the program and I also run this OpenSSL command to test STARTTLS:

openssl s_client -connect localhost:10025 -starttls smtp -debug

The output from my program is:

EHLO openssl.client.net
STARTTLS
TLS handshake failed
Stream is already closed

The output from OpenSSL is:

CONNECTED(00000003)
read from 0x6ae470 [0x6af050] (4096 bytes => 26 (0x1A))
0000 - 32 32 30 20 6d 79 73 65-72 76 65 72 20 45 53 4d   220 myserver ESM
0010 - 54 50 20 76 73 6d 74 70-64 0a                     TP vsmtpd.
write to 0x6ae470 [0x6b0060] (25 bytes => 25 (0x19))
0000 - 45 48 4c 4f 20 6f 70 65-6e 73 73 6c 2e 63 6c 69   EHLO openssl.cli
0010 - 65 6e 74 2e 6e 65 74 0d-0a                        ent.net..
read from 0x6ae470 [0x6af050] (4096 bytes => 28 (0x1C))
0000 - 32 35 30 2d 6d 79 73 65-72 76 65 72 20 48 65 6c   250-myserver Hel
0010 - 6c 6f 20 6d 79 63 6c 69-65 6e 74 0a               lo myclient.
read from 0x6ae470 [0x6af050] (4096 bytes => 13 (0xD))
0000 - 32 35 30 20 53 54 41 52-54 54 4c 53 0a            250 STARTTLS.
write to 0x6ae470 [0x7ffdb4aea9e0] (10 bytes => 10 (0xA))
0000 - 53 54 41 52 54 54 4c 53-0d 0a                     STARTTLS..
read from 0x6ae470 [0x6a13a0] (8192 bytes => 13 (0xD))
0000 - 32 32 30 20 47 6f 20 61-68 65 61 64 0a            220 Go ahead.
write to 0x6ae470 [0x6aefa0] (204 bytes => 204 (0xCC))
0000 - 16 03 01 00 c7 01 00 00-c3 03 03 0e ac 05 35 45   ..............5E
0010 - db 95 f6 a7 37 55 d8 ca-14 d7 5f 8e 6a 62 08 50   ....7U...._.jb.P
0020 - c9 81 b7 55 75 a8 4c 17-c0 a1 53 00 00 76 00 a5   ...Uu.L...S..v..
0030 - 00 a3 00 a1 00 9f 00 6b-00 6a 00 69 00 68 00 39   .......k.j.i.h.9
0040 - 00 38 00 37 00 36 00 88-00 87 00 86 00 85 00 9d   .8.7.6..........
0050 - 00 3d 00 35 00 84 00 a4-00 a2 00 a0 00 9e 00 67   .=.5...........g
0060 - 00 40 00 3f 00 3e 00 33-00 32 00 31 00 30 00 9a   .@.?.>.3.2.1.0..
0070 - 00 99 00 98 00 97 00 45-00 44 00 43 00 42 00 9c   .......E.D.C.B..
0080 - 00 3c 00 2f 00 96 00 41-00 07 00 05 00 04 00 16   .<./...A........
0090 - 00 13 00 10 00 0d 00 0a-00 15 00 12 00 0f 00 0c   ................
00a0 - 00 09 00 ff 02 01 00 00-23 00 23 00 00 00 0d 00   ........#.#.....
00b0 - 16 00 14 06 01 06 02 05-01 05 02 04 01 04 02 03   ................
00c0 - 01 03 02 02 01 02 02 00-0f 00 01 01               ............
read from 0x6ae470 [0x6b4500] (7 bytes => -1 (0xFFFFFFFFFFFFFFFF))
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 80 bytes and written 239 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
---

What I understand from the output my program closes the connection before the TLS handshake can complete. (I also tried using Thunderbird and Claws Mail)

What am I doing wrong here?

PS: I couldn't find any example on how to use GTLsServerConnection in a STARTTLS situation.

Update:

I tried -ssl2, -ssl3, -tls1, -tls1_1, -tls1_2 options of OpenSSL which also don't work.

openssl s_client -connect localhost:10025 -starttls smtp -state

yields:

CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:error in SSLv2/v3 read server hello A
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 100 bytes and written 239 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
---

So the client sends "client hello A", but the server doesn't send a correct "server hello A".

As an alternative you can also try gnutls-cli --crlf --starttls-proto=smtp --port 10025 localhost.

The output from GNUTLS_DEBUG_LEVEL=11 ./vsmtpd is:

gnutls[2]: Enabled GnuTLS logging...
gnutls[2]: Intel SSSE3 was detected
gnutls[2]: Intel AES accelerator was detected
gnutls[2]: Intel GCM accelerator was detected
gnutls[2]: Enabled GnuTLS logging...
gnutls[2]: Intel SSSE3 was detected
gnutls[2]: Intel AES accelerator was detected
gnutls[2]: Intel GCM accelerator was detected
gnutls[3]: ASSERT: x509_b64.c:299
gnutls[9]: Could not find '-----BEGIN RSA PRIVATE KEY'
gnutls[3]: ASSERT: x509_b64.c:299
gnutls[9]: Could not find '-----BEGIN DSA PRIVATE KEY'
gnutls[3]: ASSERT: x509_b64.c:299
gnutls[9]: Could not find '-----BEGIN EC PRIVATE KEY'
gnutls[3]: ASSERT: privkey.c:503
gnutls[2]: Falling back to PKCS #8 key decoding
EHLO openssl.client.net
STARTTLS
gnutls[5]: REC[0xfa67e0]: Allocating epoch #0
gnutls[3]: ASSERT: gnutls_constate.c:586
gnutls[5]: REC[0xfa67e0]: Allocating epoch #1
gnutls[3]: ASSERT: gnutls_buffers.c:1138
gnutls[10]: READ: -1 returned from 0xfa4120, errno=0 gerrno=5
gnutls[3]: ASSERT: gnutls_buffers.c:364
gnutls[3]: ASSERT: gnutls_buffers.c:572
gnutls[3]: ASSERT: gnutls_record.c:1058
gnutls[3]: ASSERT: gnutls_record.c:1179
gnutls[3]: ASSERT: gnutls_buffers.c:1392
gnutls[3]: ASSERT: gnutls_handshake.c:1428
gnutls[3]: ASSERT: gnutls_handshake.c:3098
gnutls[3]: ASSERT: gnutls_db.c:334
TLS handshake failed
Stream is already closed
gnutls[5]: REC[0xfa67e0]: Start of epoch cleanup
gnutls[5]: REC[0xfa67e0]: End of epoch cleanup
gnutls[5]: REC[0xfa67e0]: Epoch #0 freed
gnutls[5]: REC[0xfa67e0]: Epoch #1 freed

Answer 1

The problem is hidden somewhere in the implementation of DataInputStream.

Once I removed it and used the following replacement for read_line () instead, it works just fine.

string? read_line (InputStream input) throws Error {
    var buffer = new uint8[1];
    var sb = new StringBuilder ();
    buffer[0] = '\0';
    while (buffer[0] != '\n') {
        input.read (buffer);
        sb.append_c ((char) buffer[0]);
    }
    return (string) sb.data;
}

void process_request_plain (InputStream input, OutputStream output) throws Error {
    output.write (@"220 $hostname ESMTP $appname\n".data);
    string line;
    while ((line = read_line (input)) != null) {
        stdout.printf ("%s\n", line);
        line = line.chomp ();
        if (line.substring (0, 5) == "EHLO ") {
            output.write (@"250-$hostname Hello $username\n".data);
            output.write ("250 STARTTLS\n".data);
        }
        else if (line == "STARTTLS") {
            output.write ("220 Go ahead\n".data);
            break;
        }
        else {
            output.write ("502 Command not implemented\n".data);
        }
    }
}