PKCS11 Windows invalid engine YubiHSM2

Question

it's me again. I am working with a YubiHSM2 HSM Module and I am trying to set it up for the use of pkcs11 engine which will allow me to use OpenSSL with the HSM.

I am implementing this on Windows, which brings me a lot of trouble. I HAVE installed OpenSSL 32,64, OpenSC,YubiHSM2 drivers as well as libp11 (built with MSYS2).

The interesting part of my OpenSSL.cnf looks like this:

openssl_conf = openssl_init

[openssl_init]
engines = engine_section

[engine_section]
pkcs11 = pkcs11_section

[pkcs11_section]
engine_id = pkcs11
dynamic_path = "C:\Windows\System32\opensc-pkcs11.dll"
MODULE_path = "C:\Users\myUser\Desktop\SecureTemial\yubihsm2-sdk\bin\yubihsm_pkcs11.dll"
PIN = "0001password"
init = 0

When I try:

 C:\OpenSSL-Win64\bin\openssl.exe req -new -x509 -days 365 -sha256 -config C:\Users\myUser\Desktop\SecureTemial\openssl.cnf -engine pkcs11 -keyform engine -key slot_0-label_my_key -out cert.pem

I receive the following:

C:\OpenSSL-Win64\bin\openssl.exe : invalid engine "pkcs11"
In Zeile:1 Zeichen:2
+  C:\OpenSSL-Win64\bin\openssl.exe req -new -x509 -days 365 -sha256 -c ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (invalid engine "pkcs11":String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

16056:error:25078067:DSO support routines:win32_load:could not load the shared 
library:crypto\dso\dso_win32.c:106:filename(C:\Program Files\OpenSSL\lib\engines-1_1\pkcs11.dll)
16056:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto\dso\dso_lib.c:161:
16056:error:260B6084:engine routines:dynamic_load:dso not found:crypto\engine\eng_dyn.c:414:
16056:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto\engine\eng_list.c:339:id=pkcs11
16056:error:25078067:DSO support routines:win32_load:could not load the shared 
library:crypto\dso\dso_win32.c:106:filename(pkcs11.dll)
16056:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto\dso\dso_lib.c:161:
16056:error:260B6084:engine routines:dynamic_load:dso not found:crypto\engine\eng_dyn.c:414:
Error configuring OpenSSL modules
16056:error:25078067:DSO support routines:win32_load:could not load the shared 
library:crypto\dso\dso_win32.c:106:filename(C:WindowsSystem32opensc-pkcs11.dll)
16056:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto\dso\dso_lib.c:161:
16056:error:260B6084:engine routines:dynamic_load:dso not found:crypto\engine\eng_dyn.c:414:
16056:error:260BC066:engine routines:int_engine_configure:engine configuration 
error:crypto\engine\eng_cnf.c:141:section=pkcs11_section, name=dynamic_path, value=C:WindowsSystem32opensc-pkcs11.dll
16056:error:0E07606D:configuration file routines:module_run:module initialization 
error:crypto\conf\conf_mod.c:173:module=engines, value=engine_section, retcode=-1   

I have already checked if the dll's are locked and ran as admin etc. If you have any clue what's responsable for the trouble here please let me know!

Thank you very much!

Answer 1

This question was one of the first that appeared in my search results when doing some research on a similar topic. As it doesn't have an answer yet, I'll outline the results of my solution:

For using libp11's PKCS#11 engine with OpenSSL, it must be compiled as dynamic engine that is statically linked against the OpenSSL version you are using. As you are using the binaries from Shining Light Productions (a good guess based on the install directory you mentioned in your question), using a MSYS2 version obtained from a third-party resource might not work, neither does using the PKCS#11 library that ships with the OpenSC projects' Windows installers.

Fortunately, the Shining Light Productions' OpenSSL version comes with all required libraries, so you can easily compile libp11 yourself, e.g. by using NMAKE (follow the link to see how to obtain it and how to properly setup your command line for its use):

1. Download the OpenSSL binaries that meet your requirements (x86 or x64) and install them to the proposed standard destinations (e.g., C:\OpenSSL-Win32 or C:\OpenSSL-Win64). - The makefile of libp11 expects these folders for its bindings.
2. Download and extract, or clone the libp11 projects' source code.
3. Open your Windows command-line with NMAKE environment variables set and change to the location of the previously downloaded libp11 files.

4. When building for the 64 bit version of OpenSSL, you have to set the BUILD_FOR environment variable accordingly. Run

   set BUILD_FOR=WIN64
   

   on your command-line.

5. Now compile the libraries by running

   NMAKE /F Makefile.mak
   

6. If everything went well, you then have two new libraries within libp11's src folder: libp11.dll and pkcs11.dll. The latter is the PKCS#11 engine to use with your OpenSSL. Copy it to e.g., the Windows libraries folder (System32 for the 32 bit version, SysWOW64 for the x64 version).

7. Adapt your openssl.cnf file accordingly. Copy

   openssl_conf = openssl_init
   

   to the beginning of the file, and the rest to its end:

   [openssl_init]
   engines = engine_section
   
   [engine_section]
   pkcs11 = pkcs11_section
   
   [pkcs11_section]
   dynamic_path = "C:\\Windows\\SysWOW64\\pkcs11.dll"
   module_path = "C:\\Users\\myUser\\Desktop\\SecureTemial\\yubihsm2-sdk\\bin\\yubihsm_pkcs11.dll"
   PIN = "0001password"
   

Some final notes:

1. Make sure, that the adapted openssl.cnf file really is picked up by OpenSSL. The OpenSSL installation comes with several example files. By default, the location of the config files for above binaries is C:\Program Files\Common Files\SSL\openssl.cnf for the x64 version and C:\Program Files (x86)\Common Files\SSL\openssl.cnf for the x86 version. But other OpenSSL installations on your system (e.g., from OpenVPN, MingW, MSYS2, and alike that ship with bundled OpenSSL) might interfere with the settings file location. You can ensure the right settings file is used by setting the OPENSSL_CONF environment variable accordingly.
2. When using double quotes for the Windows paths, make sure to escape the backslashes correctly by using \\ instead of \.
3. You can safely omit the engine_id and init part of your openssl.cnf's [pkcs11_section].
4. While libp11's dynamic PKCS#11 engine needs to be compiled against the same architecture (x86 or x64) and libraries as OpenSSL, the module library might be required as 32 bit version (even when running the 64 bit build of OpenSSL). - At least that is what happened within the scenario of our systems (we use Gemalto Safenet e-Tokens, so the Aladdin module libraries that ship with the Safenet Authentication Client).