SSL_Accept fails, SSL_get_error returns 1

Question

I am new to SSL socket programming and my first task is to get the SSL server client to be working and understanding.

I have obtained the server and client sources floating on the web and compiled them against my openssl library (compiled from source).

When I start the server I am able to create a normal clientfd using accept system call, however SSL_accept fails.

clientsocketfd = accept(serversocketfd, NULL, 0);
serverssl = SSL_new(ssl_server_ctx);
if(!serverssl)
{
    printf("Error SSL_new\n");
    return -1;
}
SSL_set_fd(serverssl, clientsocketfd);

if((ret = SSL_accept(serverssl))!= 1)
{
    printf("Handshake Error %d\n", SSL_get_error(serverssl, ret));
    return -1;
}

SSL_accept fails printing Handshake error 1.

Why is SSL_accept failing with error 1?

---

Server source:

/*---------------------------------------------------------------------*/
/*--- main - create SSL socket server.                              ---*/
/*---------------------------------------------------------------------*/
int main(int count, char *strings[])
{   SSL_CTX *ctx;
    int server;
    char *portnum;

    if ( count != 2 )
    {
        printf("Usage: %s <portnum>\n", strings[0]);
        exit(0);
    }
    portnum = strings[1];
    SSL_library_init();
    ctx = InitServerCTX();                                                              /* initialize SSL */
    LoadCertificates(ctx, "newreq.pem", "newreq.pem");  /* load certs */
    server = OpenListener(atoi(portnum));                               /* create server socket */
    while (1)
    {   struct sockaddr_in addr;
        int len = sizeof(addr);
        SSL *ssl;

        int client = accept(server, (struct sockaddr*)&addr, &len);             /* accept connection as usual */
        printf("Connection: %s:%d\n",
                inet_ntoa(addr.sin_addr), ntohs(addr.sin_port));
        ssl = SSL_new(ctx);                                             /* get new SSL state with context */
        SSL_set_fd(ssl, client);                                                /* set connection socket to SSL state */
        Servlet(ssl);                                                                   /* service connection */
    }
    close(server);                                                                              /* close server socket */
    SSL_CTX_free(ctx);                                                                  /* release context */
}

/*---------------------------------------------------------------------*/
/*--- InitServerCTX - initialize SSL server  and create context     ---*/
/*---------------------------------------------------------------------*/
SSL_CTX* InitServerCTX(void)
{   SSL_METHOD *method;
    SSL_CTX *ctx;

    OpenSSL_add_all_algorithms();               /* load & register all cryptos, etc. */
    SSL_load_error_strings();                   /* load all error messages */
    method = SSLv3_server_method();             /* create new server-method instance */
    ctx = SSL_CTX_new(method);                  /* create new context from method */
    if ( ctx == NULL )
    {
        ERR_print_errors_fp(stderr);
        abort();
    }
    return ctx;
}

The OpenSSL version and compilation options are as folows:

/usr/bin/openssl version -a
OpenSSL 1.0.2g  1 Mar 2016
built on: reproducible build, date unspecified
platform: debian-amd64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx) 
compiler: cc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT 
-DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -g -O2 -fstack-protector-strong 
-Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,
-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DMD32_REG_T=int 
-DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT 
-DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM 
-DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM 
-DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
OPENSSLDIR: "/usr/lib/ssl"

Answer 1

The man page for SSL_accept states:

If the underlying BIO is non-blocking, SSL_accept() will also return when the underlying BIO could not satisfy the needs of SSL_accept() to continue the handshake, indicating the problem by the return value -1. In this case a call to SSL_get_error() with the return value of SSL_accept() will yield SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE. The calling process then must repeat the call after taking appropriate action to satisfy the needs of SSL_accept()...

I believe that means your code should look something like:

while (1)
{
    int acc = SSL_accept(ssl);
    if (acc == 0)
    {
        /* Hard error */
        exit(-1);
    }
    else if (acc == -1)
    {
        int err = SSL_get_error(ssl, ret);
        if (err == SSL_ERROR_WANT_READ)
        {
            /* Wait for data to be read */
        }
        else if (err == SSL_ERROR_WANT_WRITE)
        {
            /* Write data to continue */
        }
        else if (err == SSL_ERROR_SYSCALL || err == SSL_ERROR_SSL)
        {
            /* Hard error */
            exit(-1);
        }
        else if (err == SSL_ERROR_ZERO_RETURN)
        {
            /* Same as error */
            exit(-1);
        }
    }
    else
    {
        /* Continue */
        break;
    }
}

OpenSSL has sample code in <openssl src>/apps/s_client.c and <openssl src>/apps/s_server.c

Answer 2

Apparently, SSLv3 has been disabled in packages distributed with some Linux distros (e.g. Ubuntu 16.04):

openssl (1.0.2g-1ubuntu1) xenial; urgency=medium

• Merge with Debian, remaining changes.
  • Disable SSLv3 without changing ABI:
    • debian/patches/no-sslv3.patch: Disable SSLv3 without using the no-ssl3-method option
    • debian/rules: don't use no-ssl3-method, don't bump soname
    • debian/patches/engines-path.patch: don't bump soname
    • debian/patches/version-script.patch: don't bump soname
    • debian/patches/soname.patch: removed
    • debian/lib*: don't bump soname

Therefore you need to use another method:

method = SSLv3_server_method();