Using the Nitrokey HSM with GnuTLS applications

The Nitrokey HSM is an open hardware security module, in the form of a smart card token, which is used to isolate a server's private key from the application. That is, if you have an HTTPS server, such a hardware security module will prevent an attacker which temporarily obtained privileged access on the server (e.g., via an exploit like heartbleed), from copying the server's private key, allowing for impersonating it. See my previous post for a more elaborate discussion on that defense mechanism.

The rest of this post will explain how you can initialize this token and utilize it from GnuTLS applications, and in the process explain more about smart card and HSM usage in applications. For the official (and more advanced) Nitrokey setup instructions and tips you can see this OpenSC page, another interesting guide is here.

HSMs and smart cards

 

Nitrokey HSM is something between a smart card and an HSM. However, there is no real distinction between smart cards and Hardware Security Module from a software perspective. Hardware-wise one expects better (in terms of cost to defeat) tamper-resistance on HSMs, and at the same time sufficient performance for server loads. An HSM module is typically installed on PCI slots, USB, while smart cards are mainly USB or via a card reader.

On the software-side both smart cards and HSMs are accessed the same way, over the PKCS#11 API. That is an API which abstracts keys from operations, i.e., the API doesn't require direct access to the private key data to complete the operation. Most crypto libraries today support this API directly as GnuTLS and NSS do, or via an external module like OpenSSL (i.e., via engine_pkcs11).

Each HSM or smart card, comes with a "driver", i.e., a PKCS#11 module, which one had to specify on legacy applications. On modern systems, which have p11-kit, the available drivers are registered with p11-kit and applications can obtain and utilize them on run-time (see below for more information). For Nitrokey the OpenSC driver is being used, a driver for almost every other smart card that is supported on Linux.

If you are familiar with old applications, you would have noticed that objects were referred to as "slot1_1", which meant the first object on the first slot of the driver, or "1:1", and several other obscure methods depending on the application. The "slots" notion is an internal to PKCS#11, which is inherently unstable (re-inserting may change the slot number assignment), thus these methods to refer to objects cannot accommodate easily for multiple cards, or for referring to an object within a specific card if multiple are present, nor to easily utilize cards which are under the different drivers. More recent applications support PKCS#11 URIs, a method to identify tokens, and objects within the token which is unique system-wide; the URI looks like:

pkcs11:token=SmartCard-HSM;object=my-ecc-key;type=private

For GnuTLS applications, only PKCS#11 URIs can be used to refer to objects.

Driver setup and token discovery

 

On a typical Linux system which runs the pcscd server, and has opensc and p11-kit properly installed the following command should list the nitrokey token once inserted.

    $ p11tool --list-tokens

One of the entries printed should be something like the following.

Token 5:
    URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=DENK0100424;token=SmartCard-HSM20%28UserPIN%29
    Type: Hardware token
    Manufacturer: www.CardContact.de
    Model: PKCS#15 emulated
    Serial: DENK0100424
    Module: /usr/lib64/pkcs11/pkcs11/opensc-pkcs11.so

The above information contains the identifying PKCS#11 URI of the token as well as information about the manufacturer and the driver library used. The PKCS#11 URI is a standardized unique identifier of tokens and objects stored within a token. If you do not see that information, verify that you have all of pcsc-lite, pcsc-lite-ccid, opensc, gnutls and p11-kit installed. If that's the case, you will need to register the opensc token to make it known to p11-kit manually (modern distributions take care of this step). This can be done with the following commands as administrator.

    # mkdir -p /etc/pkcs11/modules
    # echo "module: /usr/lib64/pkcs11/opensc-pkcs11.so" >/etc/pkcs11/modules/opensc.conf

It is implied that the your system's libdir for PKCS#11 drivers  should be used instead of the "/usr/lib64/pkcs11" path used above. Alternatively, one could append the --provider parameter on the p11tool command, to explicitly specify the driver, as in the following example. For the rest of this text we assume a properly configured p11-kit and omit the --provider parameter.

    $ p11tool --provider /usr/lib64/pkcs11/opensc-pkcs11.so --list-tokens

Token initialization

 

An HSM token prior to usage needs to be initialized, and be provided two PINs. One PIN is for operations requiring administrative (security officer in PKCS#11 jargon) access, and the second (the user PIN ) is for normal token usage. To initialize use the following command, with the PKCS#11 URL listed by the 'p11tool --list-tokens' command; in the following text we will use $URL to refer to that.

    $ p11tool --initialize "$URL"

Alternatively, when the driver supplied supports a single card, the URL can be specified as "pkcs11:" as shown below.

    $ p11tool --provider  /usr/lib64/pkcs11/opensc-pkcs11.so --initialize "pkcs11:"

The initialization commands above will ask to setup the security officer's PIN, which for nitrokey HSM is by default "3537363231383830". At the initialization process, the user PIN will also be asked. The user PIN is PIN which must be provided by applications and users, in order to use the card. Note that the command above (prior to GnuTLS 3.5.6) will ask for the administrator's PIN twice, once for initialization and once for setting the user PIN.

Key and certificate generation

It is possible to either copy an existing key on the card, or generate a key in it, a key which cannot be extracted. To generate an elliptic curve (ECDSA) key use the following command.

    $ p11tool --label "my-key" --login --generate-ecc "pkcs11:token=SmartCard-HSM20%28UserPIN%29"

The above command will generate an ECDSA key which will be identified by the name set by the label. That key can be then by fully identified by the PKCS#11 URL "pkcs11:token=SmartCard-HSM20%28UserPIN%29;object=my-key;type=private". If the command was successful, the command above will list two objects, the private key and the public key.

    $ p11tool --login --list-all "pkcs11:token=SmartCard-HSM20%28UserPIN%29"

Note that both objects share the same ID but have different type. As this key cannot be extracted from the token, we need to utilize the following commands to generate a Certificate Signing Request (CSR).

    $ certtool --generate-request --load-privkey "pkcs11:token=SmartCard-HSM20%28UserPIN%29;object=my-key;type=private" --outfile cert.csr

After providing the required information to certtool, it will generate a certificate request on cert.csr file. Alternatively, to generate a self-signed certificate, one can replace the '--generate-request' parameter with the '--generate-self-signed'.

The above generated certificate signining request, will allow to get a real certificate to use for the key stored in the token. That can be generated either with letsencrypt or a local PKI. As the details vary, I'm skipping this step, and I'm assuming a certificate is generated somehow.

After the certificate is made available, one can write it in the token. That step is not strictly required, but in several scenarios it simplifies key/cert management by storing them at the same token. One can store the certificate, using the following command.

    $ p11tool --login --write --load-certificate cert.pem --label my-cert --id "PUBKEY-ID" "pkcs11:token=SmartCard-HSM20%28UserPIN%29"

Note that specifying the PUBKEY-ID is not required, but it is generally recommended for certificate objects to match the ID of the public key object listed previously with the --list-all command. If the IDs do not match some (non-GnuTLS) applications may fail to utilize the key. The certificate stored in the token will have the PKCS#11 URL "pkcs11:token=SmartCard-HSM20%28UserPIN%29;object=my-cert;type=cert".

Testing the generated keys

Now that both the key and the certificate are present in the token, one can utilize their PKCS#11 URL in any GnuTLS application in place of filenames. That is if the application is asking for a certificate file, enter "pkcs11:token=SmartCard-HSM20%28UserPIN%29;object=my-cert;type=cert", and for private key "pkcs11:token=SmartCard-HSM20%28UserPIN%29;object=my-key;type=private".

The following example will run a test HTTPS server using the keys above.

    $ gnutls-serv --port 4443 --http --x509certfile "pkcs11:token=SmartCard-HSM20%28UserPIN%29;object=my-cert;type=cert" --x509keyfile "pkcs11:token=SmartCard-HSM20%28UserPIN%29;object=my-key;type=private;pin-value=1234"

That will setup a server which answers on port 4443 and will utilize the certificate and key on the token to perform TLS authentication. Note that the command above, demonstrates the use of the "pin-value" URI element. That element, specifies the object PIN on command line allowing for non-interactive token access.

Applicability and performance

While the performance of this HSM will most likely not allow you to utilize it in busy servers, it may be a sufficient solution for a private server, VPN, a testing environment or demo. On client side, it can certainly provide a sufficient solution to protect the client assigned private keys. The advantage a smart card provides to OTP, is the fact that it is simpler to provision remotely, with the certificate request method shown above. That can be automated, at least in theory, when a protocol implementation of SCEP is around. In practice, SCEP is well established in the proprietary world, but it is hard to find free software applications taking advantage of it.

Converting your application to use PKCS#11

A typical application written to use GnuTLS as TLS back-end library should be able to use smart cards and HSM tokens out of the box. The only requirement is for the applications to use the high-level file loading functions, which can load files or PKCS#11 URIs when provided. The only new requirement is for the application to obtain the PIN required for accessing the token, that can be done interactively using the PIN callbacks, or via the PKCS#11 URI "pin-value" element. For source examples, I'll refer you to GnuTLS documentation.

• Client example utilizing PIN callbacks
• Server example which loads PKCS#11 URIs

Some indicative applications which I'm aware they can use tokens via PKCS#11 URIs transparently, and can be used for testing, are mod_gnutls, lighttpd2, and openconnect.

A brief look at the Linux-kernel random generator interfaces

Most modern operating systems provide a cryptographic pseudo-random number generator (CPRNG), as part of their OS kernel, intended to be used by applications involving cryptographic operations. Linux is no exception in that, and in fact it was the first operating system that actually introduced a CPRNG into the kernel. However, there is much mystery around these interfaces. The manual page is quite unclear on its suggestions, while there is a web-site dedicated to debunking myths about these interfaces, which on a first read contradicts the manual page.

In this post, triggered by my recent attempt to understand the situation and update the Linux manual page, I'll make a brief overview of these interfaces. Note that, this post will not get into the insights of a cryptographic pseudo-random generator (CPRNG); for that, consider reading this article. I will go through these interfaces, intentionally staying on the high-level, without considering internal details, and discuss their usefulness for an application or library that requires access to such a CPRNG.

• /dev/random: a file which if read from, will output data from the kernel CPRNG. Reading from this file blocks once the kernel (using some a little arbitrary metric) believes not enough random events have been accumulated since the last use (I know that this is not entirely accurate, but the description is sufficient for this post).
• /dev/urandom: a file which if read from, will provide data from the kernel CPRNG. Reading from /dev/urandom will never block.
• getrandom(): A system call which provides random data from the kernel CPRNG. It will block only when the CPRNG is not yet initialized.

A software engineer who would like to seed a PRNG or generate random encryption keys, and reads the manual page random(4) carefully, he will most likely be tempted to use /dev/random, as it is described as "suitable for uses that need very high quality randomness such as ... key generation". In practice /dev/random cannot be relied on, because it requires large amounts of random events to be accumulated in order to provide few bytes of random data to running processes. Using it for key generation (e.g, for ssh keys during first boot) is most likely going to convert the first boot process to a coin flip; heads and system is up, tails and the system is left hanging waiting for random events. This (old) issue with a mail service process hanging for more than 20 minutes prior to doing any action, illustrates the impact of this device to real-world applications which need to generate fresh keys on startup.

On the other hand, the device /dev/urandom provides access to the same random generator, but will never block, nor apply any restrictions to the amount of new random events that must be read in order to provide any output. That is quite natural given that modern random generators when initially seeded can provide enormous amounts of output prior to being considered broken (in an informational-theory sense). So should we use only /dev/urandom today?

There is a catch. Unfortunately /dev/urandom has a quite serious flaw. If used early on the boot process when the random number generator of the kernel is not fully initialized, it will still output data. How random are the output data is system-specific, and in modern platforms, which provide specialized CPU instructions to provide random data, that is less of an issue. However, the situation where ssh keys are generated prior to the kernel pool being initialized, can be observed in virtual machines which have not been given access to the host's random generator.

Another, though not as significant, issue is the fact that both of these interfaces require a file descriptor to operate. That, on a first view, may not seem like a flaw. In that case consider the following scenarios:

• The application calls chroot() prior to initializing the crypto library; the chroot environment doesn't contain any of /dev/*random.
• To avoid the issue above, the crypto library opens /dev/urandom on an library constructor and stores the descriptor for later use. The application closes all open file descriptors on startup.

Both are real-world scenarios observed over the years of developing the GnuTLS library. The latter scenario is of particular concern since, if the application opens few files, the crypto library may never realize that the /dev/urandom file descriptor has been closed and replaced by another file. That may result to reading from an arbitrary file to obtain randomness. Even though one can introduce checks to detect such case, that is a particularly hard issue to spot, and requires inefficient and complex code to address.

That's where the system call getrandom() fits. Its operation is very similar to /dev/urandom, that is, it provides non-blocking access to kernel CPRNG. In addition, it requires no file descriptor, and will also block prior to the kernel random generator being initialized. Given that it addresses, the issues of /dev/urandom identified above, that seems indeed like the interface that should be used by modern libraries and applications. In fact, if you use new versions of libgcrypt and GnuTLS today, they take advantage of this API (though that change wasn't exactly a walk in the park).

On the other hand, getrandom() is still a low-level interface, and may not be suitable to be used directly by applications expecting a safe high-level interface. If one carefully reads its manual page, he will notice that the API may return less data than the requested (if interrupted by signal), and today this system call is not even wrapped by glibc. That means that can be used only via the syscall() interface. An illustration of (safe) usage of this system call, is given below.

#include <sys/syscall.h>
#include <errno.h>
#define getrandom(dst,s,flags) syscall(SYS_getrandom, (void*)dst, (size_t)s, (unsigned int)flags)

static int safe_getrandom(void *buf, size_t buflen, unsigned int flags)
{
  ssize_t left = buflen;
  ssize_t ret;
  uint8_t *p = buf;
  while (left > 0) {
   ret = getrandom(p, left, flags);
   if (ret == -1) {
    if (errno != EINTR)
     return ret;
   }
   if (ret > 0) {
    left -= ret;
    p += ret;
   }
  }
  return buflen;
}

The previous example code assumes that the Linux kernel supports this system call. For portable code which may run on kernels without it, a fallback to /dev/urandom should also be included.

From the above, it is apparent that using the Linux-kernel provided interfaces to access the kernel CPRNG, is not easy. The old (/dev/*random) interfaces APIs are difficult to use correctly, and while the getrandom() call eliminates several of their issues, it is not straightforward to use, and is not available in Linux kernels prior to 3.17. Hence, if applications require access to a CPRNG, my recommendation would be to avoid using the kernel interfaces directly, and use any APIs provided by their crypto library of choice. That way the complexity of system-discovery and any other peculiarities of these interfaces will be hidden. Some hints and tips are shown in the Fedora defensive coding guide (which may be a bit out-of-date but still a good source of information).

Restricting the scope of CA certificates

The granting of an intermediate CA certificate to a surveillance firm generated quite some fuss. Setting theories aside, the main reason behind that outcry, is the fact that any intermediate CA certificate trusted by the browsers has unlimited powers to certify any web site on the Internet. Servers can protect themselves against an arbitrary CA generating a valid certificate for their web site, using certificate pinning, but there is very little end-users can do. In practice, end-users either trust the whole bundled CA list in their browser/system or not.

An option for end-users is to utilize trust on first use, but that is not a widespread practice, and few software, besides for SSH, support it. A way for me as a user to defend against a believed to be rogue CA, is by disabling or removing that CA from my trusted bundle. But what if I trust that CA for a particular web site or domain, but not for the whole Internet?

On this post I'll try to provide more information on some lesser documented aspects of p11-kit, which provide additional control over the CA certificate bundle in a system. That is, I'll explain how we can do better than disabling CAs, and how we can restrict CAs to particular domains. The following instructions are limited to Fedora 22+ which has deployed a shared trust database for certificates based on p11-kit. This database, is not only an archive of trusted certificates, but also provides the option to attach additional attributes to CA certificates in the form of PKIX extensions. These extensions are called stapled extensions in p11-kit jargon and they override any extensions available in the trust certificates. That, allows to enforce additional restrictions to the purpose and scope of a certificate.

I'll attempt to demonstrate this feature using an example. Let's consider the case where your employer's IT department provided you with a CA certificate to trust for communications within the company. Let's also assume that the company's internal domain is called "example.com". In that scenario as a user I'd like to restrict the provided CA certificate to example.com domain to prevent anyone with access to the corporate private key from being able to hijack any connection outside the company scope. This is not only out of paranoia against a potential corporate big-brother but also to keep a good security practice and avoid having master keys. A stolen corporate CA key which is trusted for everything under the sun provides a potential attacker not only with access to company's internal communication, but also with access to Internet communication of any corporate user.

How would we install such certificate in a way that it is restricted only to example.com? Assuming that the CA certificate is provided at the example.com-root.pem file, the following command will add the company's certificate to the trusted list.

$ sudo trust anchor example.com-root.pem

That will create a file in /etc/pki/ca-trust/source containing the CA certificate (for more information on adding and removing CA certificates in Fedora see the update-ca-trust manpage).

If we edit this file we will see something like the following.

[p11-kit-object-v1]
trusted: true
x-distrusted: false
private: false
certificate-category: authority
-----BEGIN CERTIFICATE-----
MIIDsDCCAxmgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBnTELMAkGA1UEBhMCVVMx
...
-----END CERTIFICATE-----

This contains the certificate of the CA as well as various basic flags set to it.
How can we now attach a stapled extension to it?

We need to add another object in that database containing the extension. But let's see the process step by step. First we need to extract the certificate's public key because that's how p11-kit identifies existing objects. A command to achieve that is the following:

$ certool --pubkey-info --infile example.com-root.pem --outfile example.com-pubkey.pem

 The output file will contain a public key in PEM format (identifiable by the "-----BEGIN PUBLIC KEY-----" header). We now edit the p11-kit file in  /etc/pki/ca-trust/source containing our certificate and append the following.

[p11-kit-object-v1]
class: x-certificate-extension
label: "Example.com CA restriction"
object-id: 2.5.29.30
value: "%30%1a%06%03%55%1d%1e%04%13%30%11%a0%0f%30%0d%82%0b%65%78%61%6d%70%6c%65%2e%63%6f%6d"
-----BEGIN PUBLIC KEY-----
...
-----END PUBLIC KEY-----

Where the public key part is copied from the example.com-pubkey.pem file.

This added object, is a stapled extension containing a PKIX name constraints extension which allows this CA to be used for certificates under the "example.com" domain. If you attempt to connect to a host with a certificate of this CA you will get the following error:

$ gnutls-cli www.no-example.com
...
Status: The certificate is NOT trusted. The certificate chain violates the signer's constraints.
*** PKI verification of server certificate failed...

Note that, although NSS and openssl applications check some extensions (such as key purpose) from this trust database, they do not consider the name constraints extension. This may change in the future, but currently only GnuTLS applications under Fedora will honor this extension. The reason it works under Fedora distribution is because GnuTLS is compiled using the --with-default-trust-store-pkcs11="pkcs11:" configuration option which makes it use the p11-kit trust DB directly.

A question at this point, after seeing the p11-kit object format, is how can we generate the "value" listed above containing the desired constraints? The value contains a DER encoded certificate extension which corresponds to the object identifier "object-id" field. In this case the object-id field contains the object identifier for NameConstraints extension (2.5.29.30).

Unfortunately there are no available tools to generate this value, that I'm aware of. I created a sample application which will generate a valid name constraints value to be set above. The tool can be found at this github repository.

After you compile, run:

$ ./nconstraints mydomain.com myotherdomain.com
%30%30%06%03%55%1d%1e%04%29%30%27%a0%25%30%0e%82%0c%6d%79%64%6f%6d%61%69%6e%2e%63%6f%6d%30%13%82%11%6d%79%6f%74%68%65%72%64%6f%6d%61%69%6e%2e%63%6f%6d

and as you see, this command will provide the required string.

Happy hacking!