Applied PKCS #11

PKCS #11 is the name given to a standard defining an API for cryptographic hardware. While it was developed by RSA, as part of a suite of standards, the standard is not exclusive to RSA ciphers and is meant to cover a wide range of cryptographic possibilities. PKCS #11 is most closely related to Java’s JCE and Microsoft’s CAPI.

Concepts in PKCS #11

Slots and Tokens

A slot originally referred to a single card slot on a smartcard device that could accept a token. A token was a smartcard that contained secure, encrypted keys and certificates. You would insert your smartcard (token) into the slot, and use its contents to do cryptographic operations.

Nowadays the distinction is more blurry. Many USB-key HSMs appear as a single slot containing a hardwired single token (their internal storage). Server devices often make use of software tokens (softcards), which appear as slots within PKCS #11, but no physical device exists. These devices can also feature physical slots and accelerator slots.

See also

Slots have pkcs11.Slot.flags which can tell you something about what kind of slot this is.

Tokens are secured with a passphrase (PIN). Not all implementations use pins in their underlying implementation, but these are required for PKCS#11. Some implementations let you control the behaviour of their PKCS #11 module in ways not specified by the specification through environment variables (e.g. default token pins).

Note

The PKCS #11 library is running within your process, using your memory, etc. It may talk to a daemon to access the underlying hardware, or it may be talking directly.

Environment variables set on your process can be used to configure the behaviour of the library, check the documentation for your device.

Finding Tokens

Tokens are identified by a label or serial number.

You can retrieve all tokens matching search parameters:

for slot in lib.get_slots():
    token = slot.get_token()
    # Check the parameters
    if token.label == '...':
        break

for token in lib.get_tokens(token_label='smartcard'):
    print(token)

Retrieving a single token has a shortcut function:

try:
    lib.get_token(token_label='smartcard')
except NoSuchToken:
    pass
except MultipleTokensReturned:
    pass

Mechanisms and Capabilities

Different devices support different cryptographic operations. In PKCS #11 mechanisms refer to the combination of cipher (e.g. AES), hash function (e.g. SHA512) and block mode (e.g. CBC). Mechanisms also exist for generating keys, and deriving keys and parameters.

The capabilities of a mechanism indicate what types of operations can be carried out with the mechanism, e.g. encryption, signing, key generation.

Not all devices support all mechanisms. Some may support non-standard mechanisms. Not all devices support the same capabilities for mechanisms or same key lengths. This information can be retrieved via pkcs11.Slot.get_mechanisms() and pkcs11.Slot.get_mechanism_info() or from your device documentation.

Some mechanisms require mechanism parameters. These are used to provide additional context to the mechanism that does not form part of the key. Examples of mechanism parameters are initialisation vectors for block modes, salts, key derivation functions, and other party’s shared secrets (for Diffie-Hellman).

See also

The pkcs11.mechanisms.Mechanism type includes information on the required parameters for common mechanisms. A complete list of current mechanisms and historical mechanisms includes the mechanism parameters and input requirements for each mechanism.

Objects and Attributes

An object is a piece of cryptographic information stored on a token. Objects have a class (e.g. private key) which is exposed in python-pkcs11 as a Python class. They also have a number of other attributes depending on their class.

There are three main classes of object:

• keys (symmetric secret keys and asymmetric public and private keys);

• domain parameters (storing the parameters used to generate keys); and

• certificates (e.g. X.509 certificates).

Note

Irregardless of the PKCS #11 specification, not all devices reliably handle all object attributes. They can also have different defaults. python-pkcs11 tries to abstract that as much as possible to enable writing portable code.

See also

pkcs11.constants.Attribute describes the available attributes and their Python types.

biginteger

One type is handled specially: biginteger, an arbitrarily long integer in network byte order. Although Python can handle arbitrarily long integers, many other systems cannot and pass these types around as byte arrays, and more often than not, that is an easier form to handle them in.

biginteger attributes can be specified as bytes, bytearray or an iterable of byte-sized integers.

If you do have integers, you can convert them to bytes using pkcs11.util.biginteger().

Finding Objects

Objects can be found on a token using their attributes. Usually an ID or LABEL.

for obj in session.get_objects({
    Attribute.CLASS: ObjectClass.SECRET_KEY,
    Attribute.LABEL: 'aes256',
}):
    print(obj)

Finding a specific key is so common there’s a shortcut function:

try:
    key = session.get_key(label='aes256')
except NoSuchKey:
    pass
except MultipleObjectsReturned:
    pass

Keys

There are three classes of key objects:

• symmetric secret keys;

• asymmetric public keys; and

• asymmetric private keys.

The following attributes can be set for keys:

PRIVATE

Private objects can only be accessed by logged in sessions.

LOCAL

This key was generated on the device.

EXTRACTABLE

The key can be extracted from the HSM.

SENSITIVE

The key is sensitive and cannot be removed from the device in clear text.

ALWAYS_SENSITIVE

The key has never not been SENSITIVE.

NEVER_EXTRACTABLE

The key has never been EXTRACTABLE.

ALWAYS_AUTHENTICATE

The key requires authentication every time it’s used.

Note

Keys should be generated on the HSM rather than imported. Generally only public keys should not be PRIVATE and SENSITIVE. Allowing private keys to be accessed defeats the purpose of securing your keys in a HSM. python-pkcs11 sets meaningful defaults.

Domain Parameters

Domain parameters are the parameters used to generate cryptographic keys (e.g. the name of the elliptic curve being used). They are public information. Obscuring the domain parameters does not increase the security of a cryptosystem. Typically the domain parameters form part of a protocol specification, and RFCs exist giving pre-agreed, named domain parameters for cryptosystems.

In python-pkcs11 domain parameters can either be stored as an object in your HSM, or loaded via some other mechanism (e.g. in your code) and used directly without creating a HSM object.

See also

OpenSSL can be used to generate unique or named domain parameters for Diffie-Hellman, DSA and EC.

pkcs11.util includes modules for creating and decoding domain parameters.

Sessions

Accessing a token is done by opening a session. Sessions can be public or logged in. Only a logged in session can access objects marked as private. Depending on your device, some functions may also be unavailable.

Warning

It is important to close sessions when you are finished with them. Some devices will leak resources if sessions aren’t closed.

Where possible you should use sessions via a context manager.

Concepts related to PKCS #11

Binary Formats and Padding

PKCS #11 is protocol agnostic and does not define or implement any codecs for the storing of enciphered data, keys, initialisation vectors, etc. outside the HSM. 1 For example, CBC mechanisms will not include the initialization vector. You must choose a storage/transmission format that suits your requirements.

Some mechanisms require input data to be padded to a certain block size. Standardized PAD variants of many mechanisms exist based on upstream specifications. For other mechanisms PKCS #11 does not define any specific algorithms, and you must choose one that suits your requirements.

See also

Lots of standards exist for the storing and transmission of cryptographic data. If you’re not implementing a specific protocol, there may still be an RFC standard with a Python implementation to ensure people can understand your binary data in the future.

See also:

• RFC 5652 (Cryptographic Message Standard) (supercedes PKCS #7)

1

It does define types for data inside the HSM, e.g. attribute data types and binary formats (e.g. EC parameters, X.509 certificates).

PKCS #15

PKCS #15 defines a standard for storing cryptographic objects within the HSM device to enable interoperability between devices and tokens. PKCS #15 is often referenced in conjunction with PKCS #11 as the storage format used on the tokens.

ASN.1, DER, BER

ASN.1 is a data model for storing structured information. DER and BER are binary representations of that data model which are used extensively in cryptography, e.g. for storing RSA key objects, X.509 certificates and elliptic curve information.

Accessing ASN.1 encoded objects is mostly left to packages other than python-pkcs11, however pkcs11.util does include some utilities to encode and decode objects where required for working with PKCS #11 itself (e.g. converting PKCS #1 encoded RSA keys into PKCS #11 objects and generating parameters for elliptic curves).

PEM

PEM is a standard for handling cryptographic objects. It is a base64 encoded version of the binary DER object. The label indicates the type of object, and thus what ASN.1 model to use. python-pkcs11 does not include PEM parsing, you should include another package if required. asn1crypto.pem is a dependency of python-pkcs11.

Getting a Session

Given a PKCS #11 library (.so) that is stored in the environment as PKCS11_MODULE.

To open a read-only session on a token named smartcard:

import pkcs11

lib = pkcs11.lib(os.environ['PKCS11_MODULE'])
token = lib.get_token(token_label='smartcard')

with token.open() as session:
    print(session)

To open a user session with the passphrase/pin secret:

with token.open(user_pin='secret') as session:
    print(session)

To open a read/write session:

with token.open(rw=True, user_pin='secret') as session:
    print(session)

See also

pkcs11.Token.open() has more options for opening the session.

Generating Keys

Keys can either live for the lifetime of the session or be stored on the token. Storing keys requires a read only session.

To store keys pass store=True. When storing keys it is recommended to set a label or id, so you can find the key again.

Symmetric Keys

AES

AES keys can be generated by specifying the key length:

from pkcs11 import KeyType

key = session.generate_key(KeyType.AES, 256)

Generally AES keys are considered secret. However if you’re using your HSM to generate keys for use with local AES (e.g. in hybrid encryption systems). You can do the following:

from pkcs11 import KeyType, Attribute

key = session.generate_key(KeyType.AES, 256, template={
    Attribute.SENSITIVE: False,
    Attribute.EXTRACTABLE: True,
})
# This is the secret key
print(key[Attribute.VALUE])

VALUE

Secret key (as biginteger).

DES2/3

Warning

DES2 and DES3 are considered insecure because their short key lengths are brute forcable with modern hardware.

DES2/3 keys are fixed length.

from pkcs11 import KeyType

des2 = session.generate_key(KeyType.DES2)
des3 = session.generate_key(KeyType.DES3)

These secret key objects have the same parameters as for AES.

Asymmetric Keypairs

RSA

RSA keypairs can be generated by specifying the length of the modulus:

from pkcs11 import KeyType

public, private = session.generate_keypair(KeyType.RSA, 2048)

The default public exponent is 65537. You can specify an alternative:

from pkcs11 import KeyType, Attribute

public, private = session.generate_keypair(KeyType.RSA, 2048,
                                           public_template={Attribute.PUBLIC_EXPONENT: ...})
# This is the public key
print(public[Attribute.MODULUS])
print(public[Attribute.PUBLIC_EXPONENT])

The public key has two parameters:

MODULUS

Key modulus (as biginteger).

PUBLIC_EXPONENT

Public exponent (as biginteger).

These can be exported as RFC 2437 (PKCS #1) DER-encoded binary using pkcs11.util.rsa.encode_rsa_public_key().

DSA

DSA keypairs can be generated by specifying the length of the prime in bits.

from pkcs11 import KeyType

public, private = session.generate_keypair(KeyType.RSA, 2048)

This will generate unique domain parameters for a key. If you want to create a key for given domain parameters, see DSA from Domain Parameters.

The public key has a single important attribute:

VALUE

Public key (as biginteger).

This can be encoded in RFC 3279 format with pkcs11.util.dsa.encode_dsa_public_key().

From Domain Parameters

Note

Choosing domain parameters is not covered in this document. Domain parameters are often either specified by the requirements you are implementing for, or have a standard implementation to derive quality parameters. Some domain parameters (e.g. choice of elliptic curve) can drastically weaken the cryptosystem.

DSA

Diffie-Hellman key pairs require three domain parameters, specified as bigintegers.

BASE

The prime base (g) (as biginteger).

PRIME

The prime modulus (p) (as biginteger).

SUBPRIME

The subprime (q) (as biginteger).

from pkcs11 import Attribute

parameters = session.create_domain_parameters(KeyType.DSA, {
    Attribute.PRIME: b'prime...',
    Attribute.BASE: b'base...',
    Attribute.SUBPRIME: b'subprime...',
}, local=True)

public, private = parameters.generate_keypair()

RFC 3279 defines a standard ASN.1 encoding for DSA parameters, which can be loaded with pkcs11.util.dsa.decode_dsa_domain_parameters():

params = session.create_domain_parameters(
    KeyType.DSA,
    decode_dsa_domain_parameters(b'DER-encoded parameters'),
    local=True)

If supported, unique domain parameters can also be generated for a given PRIME length (e.g. 1024 bits) with pkcs11.Session.generate_domain_parameters():

params = session.generate_domain_parameters(KeyType.DSA, 1024)

These can be encoded into the standard ASN.1 DER encoding using pkcs11.util.dsa.encode_dsa_domain_parameters().

Note

You can create a DSA key directly from freshly generated domain parameters with Session.generate_keypair().

Diffie-Hellman

Diffie-Hellman key pairs require several domain parameters, specified as bigintegers. There are two forms of Diffie-Hellman domain parameters: PKCS #3 and X9.42.

BASE

The prime base (g) (as biginteger).

PRIME

The prime modulus (p) (as biginteger).

SUBPRIME

(X9.42 only) The subprime (q) (as biginteger).

from pkcs11 import Attribute

parameters = session.create_domain_parameters(KeyType.DH, {
    Attribute.PRIME: b'prime...',
    Attribute.BASE: b'base...',
}, local=True)

public, private = parameters.generate_keypair()

RFC 3279 defines a standard ASN.1 encoding for DH parameters, which can be loaded with pkcs11.util.dh.decode_x9_42_dh_domain_parameters():

params = session.create_domain_parameters(
    KeyType.X9_42_DH,
    decode_x9_42_dh_domain_parameters(b'DER-encoded parameters'),
    local=True)

If supported, unique domain parameters can also be generated for a given PRIME length (e.g. 512 bits) with pkcs11.Session.generate_domain_parameters():

params = session.generate_domain_parameters(KeyType.DH, 512)

X9.42 format domain parameters can be encoded back to their RFC 3279 format with pkcs11.util.dh.encode_x9_42_dh_domain_parameters().

Key pairs can be generated from the domain parameters:

public, private = parameters.generate_keypair()
# This is the public key
print(public[Attribute.VALUE])

The public key has a single important attribute:

VALUE

Public key (as biginteger).

This can be encoded in RFC 3279 format with pkcs11.util.dh.encode_dh_public_key().

Elliptic Curve

Elliptic curves require a domain parameter describing the curve. Curves can be described in two ways:

• As named curves; or

• As a complete set of parameters.

Not all devices support both specifications. You can determine what curve parameters your device supports by checking pkcs11.Slot.get_mechanism_info() pkcs11.constants.MechanismFlag.

Both specifications are specified using the same attribute:

EC_PARAMS

Curve parameters (as DER-encoded X9.62 bytes).

from pkcs11 import Attribute


parameters = session.create_domain_parameters(KeyType.EC,
    Attribute.EC_PARAMS: b'DER-encoded X9.62 parameters ...',
}, local=True)

public, private = parameters.generate_keypair()

Named curves (e.g. secp256r1) can be specified like this:

from pkcs11 import Attribute
from pkcs11.util.ec import encode_named_curve_parameters


parameters = session.create_domain_parameters(KeyType.EC, {
    Attribute.EC_PARAMS: encode_named_curve_parameters('secp256r1')
}, local=True)

Key pairs can be generated from the domain parameters:

public, private = parameters.generate_keypair()
# This is the public key
print(public[Attribute.EC_POINT])

The public key as a single important attribute:

EC_POINT

Public key (as X9.62 DER-encoded bytes).

Importing/Exporting Keys

Warning

It is best to only import/export public keys. You should, whenever possible, generate and store secret and private keys within the boundary of your HSM.

The following utility methods will convert keys encoded in their canonical DER-encoded into attributes that can be used with pkcs11.Session.create_object().

Note

PEM certificates are base64-encoded versions of the canonical DER-encoded forms used in python-pkcs11. Conversion between PEM and DER can be achieved using asn1crypto.pem.

AES/DES

Warning

Whenever possible, generate and store secret keys within the boundary of your HSM.

AES and DES keys are stored as binary bytes in pkcs11.constants.Attribute.VALUE.

Keys must be marked as EXTRACTABLE and not SENSITIVE to export.

RSA

To import a PKCS #1 DER-encoded RSA key, the following utility methods are provided:

• pkcs11.util.rsa.decode_rsa_public_key(), and

• pkcs11.util.rsa.decode_rsa_private_key().

To export an RSA public key in PKCS #1 DER-encoded format, use pkcs11.util.rsa.encode_rsa_public_key().

DSA

To import an RFC 3279 DER-encoded DSA key, the following utility methods are provided:

• pkcs11.util.dsa.decode_dsa_domain_parameters(), and

• pkcs11.util.dsa.decode_dsa_public_key().

To export a DSA public key, use:

• pkcs11.util.dsa.encode_dsa_domain_parameters(), and

• pkcs11.util.dsa.encode_dsa_public_key().

Elliptic Curve

The pkcs11.constants.Attribute.EC_PARAMS and pkcs11.constants.Attribute.EC_POINT attributes for elliptic curves are already in DER-encoded X9.62 format.

You can import keys from OpenSSL using:

• pkcs11.util.ec.decode_ec_public_key(), and

• pkcs11.util.ec.decode_ec_private_key().

To export an EC public key in OpenSSL format, use pkcs11.util.ec.encode_ec_public_key().

X.509

The function pkcs11.util.x509.decode_x509_public_key() is provided to extract public keys from X.509 DER-encoded certificates, which is capable of handling RSA, DSA and ECDSA keys.

Encryption/Decryption

Ciphers can generally be considered in two categories:

• Symmetric ciphers (e.g. AES), which use a single key to encrypt and decrypt, and are good at encrypting large amounts of data; and

• Asymmetric ciphers (e.g. RSA), which use separate public and private keys, and are good for securing small amounts of data.

Symmetric ciphers operate on blocks of data, and thus are used along with a block mode. python-pkcs11 can consume block mode ciphers via a generator.

Asymmetric ciphers are used for public-key cryptography. They cannot encrypt large amounts of data. Typically these ciphers are used to encrypt a symmetric session key, which does the bulk of the work, in a so-called hybrid cryptosystem.

Cipher	Block modes	Block Size (IV len)	Mechanism Param
AES	Yes	128 bits	IV (except EBC)
DES2/3	Yes	64 bits	IV (except EBC)
RSA	No	N/A	Optional

AES

The AES cipher requires you to specify a block mode as part of the mechanism.

The default block mode is CBC with PKCS padding, which can handle data not padded to the block size and requires you to supply an initialisation vector of 128-bits of good random.

A number of other mechanisms are available:

Mechanism	IV	Input Size	Notes
AES_ECB	No	128-bit blocks	Only suitable for key-wrapping. Identical blocks encrypt identically!
AES_CBC	Yes	128-bit blocks
AES_CBC_PAD	Yes	Any	Default mechanism
AES_OFB	Yes	Any
AES_CFB_*	Yes	Any	3 modes: AES_CFB8, AES_CFB64, and AES_CFB128.
AES_CTS	Yes	>= 128-bit
AES_CTR	Not currently supported 2
AES_GCM
AES_CGM

2

AES encryption with multiple mechanism parameters not currently implemented due to lack of hardware supporting these mechanisms.

Warning

Initialisation vectors

An initialization vector (IV) or starting variable (SV) is data that is used by several modes to randomize the encryption and hence to produce distinct ciphertexts even if the same plaintext is encrypted multiple times.

An initialization vector has different security requirements than a key, so the IV usually does not need to be secret. However, in most cases, it is important that an initialization vector is never reused under the same key. For CBC and CFB, reusing an IV leaks some information about the first block of plaintext, and about any common prefix shared by the two messages. For OFB and CTR, reusing an IV completely destroys security.

In CBC mode, the IV must, in addition, be unpredictable at encryption time; in particular, the (previously) common practice of re-using the last ciphertext block of a message as the IV for the next message is insecure.

We recommend using pkcs11.Session.generate_random() to create a quality IV.

A simple example:

# Given an AES key `key`
iv = session.generate_random(128)
ciphertext = key.encrypt(plaintext, mechanism_param=iv)

plaintext = key.decrypt(ciphertext, mechanism_param=iv)

Or using an alternative mechanism:

from pkcs11 import Mechanism

iv = session.generate_random(128)
ciphertext = key.encrypt(plaintext,
                         mechanism=Mechanism.AES_OFB,
                         mechanism_param=iv)

Large amounts of data can be passed as a generator:

buffer_size = 8192
with \\
        open(file_in, 'rb') as input, \\
        open(file_out, 'wb') as output:

    # A generator yielding chunks of the file
    chunks = iter(lambda: input.read(buffer_size), '')

    for chunk in key.encrypt(chunks,
                             mechanism_param=iv,
                             buffer_size=buffer_size):
        output.write(chunk)

Note

These mechanisms do not store the IV. You must store the IV yourself, e.g. on the front of the ciphertext. It is safe to store an IV in the clear.

DES2/3

Warning

DES2 and DES3 are considered insecure because their short key lengths are brute forcable with modern hardware.

DES2/3 have the same block mode options as AES. The block size is 64 bits, which is the size of the initialization vector.

# Given an DES3 key `key`
iv = session.generate_random(64)
ciphertext = key.encrypt(plaintext, mechanism_param=iv)

plaintext = key.decrypt(ciphertext, mechanism_param=iv)

RSA

The default RSA cipher is PKCS #1 OAEP

A number of other mechanisms are available:

Mechanism	Parameters	Input Length	Notes
RSA_PKCS	None	<= key length - 11	RSA v1.5. Don’t use for new applications.
RSA_PKCS_OAEP	See below	<= k - 2 - 2hLen	Default mechanism.
RSA_X_509	None	key length	Raw mode. No padding.
RSA_PKCS_TPM_1_1	None	<= key length - 11 - 5	See TCPA TPM Specification Version 1.1b
RSA_PKCS_OAEP_TPM_1_1	See below	<= k - 2 - 2hLen

A simple example using the default parameters:

# Given an RSA key pair `public, private`
ciphertext = public.encrypt(plaintext)

plaintext = private.decrypt(ciphertext)

RSA OAEP can optionally take a tuple of (hash algorithm, mask generating function and source data) as the mechanism parameter:

ciphertext = public.encrypt(plaintext,
                            mechanism=Mechanism.RSA_PKCS_OAEP,
                            mechanism_param=(Mechanism.SHA_1,
                                             MGF.SHA1,
                                             None))

Signing/Verifying

Signing and verification mechanisms require two components:

• the cipher; and

• the hashing function.

Raw versions for some mechanisms also exist. These require you to do your own hashing outside of PKCS #11.

Signing functions typically work on a finite length of data, so the signing of large amounts of data requires hashing with a secure one-way hash function.

AES

A MAC is required for signing with AES. The default mechanism is AES_MAC.

# Given a secret key, `key`
signature = key.sign(data)

assert key.verify(data, signature)

DES2/3

A MAC is required for signing with DES. The default mechanism is SHA512_HMAC (aka HMAC-SHA512).

Operation is the same as for AES.

RSA

The default signing and verification mechanism for RSA is RSA_SHA512_PKCS.

Other mechanisms are available:

Mechanism	Notes
RSA_PKCS	No hashing. Supply your own.
SHA*_RSA_PKCS	SHAx message digesting.
RSA_PKCS_PSS	Optionally takes a tuple of parameters.
SHA*_RSA_PKCS_PSS
RSA_9796	ISO/IES 9796 RSA signing. Use PSS instead.
RSA_X_509	X.509 (raw) RSA signing. You must supply your own padding.
RSA_X9_31	X9.31 RSA signing.

Simple example using the default mechanism:

# Given a private key `private`
signature = private.sign(data)

# Given a public key `public`
assert public.verify(data, signature)

RSA PSS optionally takes a tuple of (hash algorithm, mask generating function and salt length) as the mechanism parameter:

signature = private.sign(data,
                           mechanism=Mechanism.RSA_PKCS_PSS,
                           mechanism_param=(Mechanism.SHA_1,
                                           MGF.SHA1,
                                           20))

DSA

The default signing and verification mechanism for RSA is DSA_SHA512.

Other mechanisms are available:

Mechanism	Notes
DSA	No hashing. 20, 28, 32, 48 or 64 bits.
DSA_SHA*	DSA with SHAx message digesting.

# Given a private key `private`
signature = private.sign(data)

# Given a public key `public`
assert public.verify(data, signature)

The parameters r and s are concatenated together as a single byte string (each value is 20 bytes long for a total of 40 bytes). To convert to the ASN.1 encoding (e.g. as used by X.509) use pkcs11.util.dsa.encode_dsa_signature(). To convert from the ASN.1 encoding into PKCS #11 encoding use pkcs11.util.ec.decode_dsa_signature().

ECDSA

The default signing and verification mechanism for ECDSA is ECDSA_SHA512.

Other mechanisms are available:

Mechanism	Notes
ECDSA	No hashing. Input truncated to 1024 bits.
ECDSA_SHA*	ECDSA with SHAx message digesting.

# Given a private key `private`
signature = private.sign(data)

# Given a public key `public`
assert public.verify(data, signature)

The parameters r and s are concatenated together as a single byte string (both values are the same length). To convert to the ASN.1 encoding (e.g. as used by X.509) use pkcs11.util.ec.encode_ecdsa_signature(). To convert from the ASN.1 encoding into PKCS #11 encoding use pkcs11.util.ec.decode_ecdsa_signature().

Wrapping/Unwrapping

The expectation when using HSMs is that secret and private keys never leave the secure boundary of the HSM. However, there is a use case for transmitting secret and private keys over insecure mediums. We can do this using key wrapping.

Key wrapping is similar to encryption and decryption except instead of turning plaintext into crypttext it turns key objects into crypttext and vice versa.

Keys must be marked as EXTRACTABLE to remove them from the HSM, even wrapped.

Key wrapping mechanisms usually mirror encryption mechanisms.

AES

Default key wrapping mode is AES_ECB. ECB is considered safe for key wrapping due to the lack of repeating blocks. Other mechanisms, such as the new AES_KEY_WRAP (if available), are also possible..

The key we’re wrapping can be any sensitive key, either a secret key or a private key. In this example we’re extracting an AES secret key:

# Given two secret keys, `key1` and `key2`, we can extract an encrypted
# version of `key2`
crypttext = key1.wrap_key(key2)

Wrapping doesn’t store any parameters about the keys. We must supply those to import the key.

key = key1.unwrap_key(ObjectClass.SECRET_KEY, KeyType.AES, crypttext)

DES2/3

Default key wrapping mode is DES3_ECB. ECB is considered safe for key wrapping due to the lack of repeating blocks. Other mechanisms are available.

Operation is the same as for AES.

RSA

The key we’re wrapping can be any sensitive key, either a secret key or a private key. In this example we’re extracting an AES secret key:

# Given a public key, `public`, and a secret key `key`, we can extract an
encrypted version of `key`
crypttext = public.wrap_key(key)

Wrapping doesn’t store any parameters about the keys. We must supply those to import the key.

# Given a private key, `private`, matching `public` above we can decrypt
# and import `key`.
key = private.unwrap_key(ObjectClass.SECRET_KEY, KeyType.AES, crypttext)

Deriving Shared Keys

Warning

Key derivation mechanisms do not verify the authenticity of the other party. Your application should include a mechanism to verify the other user’s public key is really from that user to avoid man-in-the-middle attacks.

Where possible use an existing protocol.

Diffie-Hellman

DH lets us derive a shared key using shared domain parameters, our private key and the other party’s public key, which is passed as a mechanism parameter.

The default DH derivation mechanism is DH_PKCS_DERIVE, which uses the algorithm described in PKCS #3.

Note

Other DH derivation mechanisms including X9.42 derivation are not currently supported.

# Given our DH private key `private` and the other party's public key
# `other_public`
key = private.derive_key(
    KeyType.AES, 128,
    mechanism_param=other_public)

If the other user’s public key was encoded using RFC 3279, we can decode this with pkcs11.util.dh.decode_dh_public_key():

from pkcs11.util.dh import decode_dh_public_key

key = private.derive_key(
    KeyType.AES, 128,
    mechanism_param=decode_dh_public_key(encoded_public_key))

And we can encode our public key for them using pkcs11.util.dh.encode_dh_public_key():

from pkcs11.util.dh import encode_dh_public_key

# Given our DH public key `public`
encoded_public_key = encode_dh_public_key(public)

The shared derived key can now be used for any appropriate mechanism.

If you want to extract the shared key from the HSM, you can mark the key as EXTRACTABLE:

key = private.derive_key(
    KeyType.AES, 128,
    mechanism_param=other_public,
    template={
        Attribute.SENSITIVE: False,
        Attribute.EXTRACTABLE: True,
    })
# This is our shared secret key
print(key[Attribute.VALUE])

EC Diffie-Hellman

ECDH is supported using the ECDH1_DERIVE mechanism, similar to plain DH, except that the mechanism parameter is a tuple consisting of 3 parameters:

• a key derivation function (KDF);

• a shared value; and

• the other user’s public key.

The supported KDFs vary from device to device, check your HSM documentation. For pkcs11.mechanisms.KDF.NULL (the most widely supported KDF), the shared value must be None.

Note

Other ECDH derivation mechanisms including co-factor derivation and MQV derivation are not currently supported.

from pkcs11 import KeyType, KDF

# Given our DH private key `private` and the other party's public key
# `other_public`
key = private.derive_key(
    KeyType.AES, 128,
    mechanism_param=(KDF.NULL, None, other_public))

The value of the other user’s public key should usually be a raw byte string however some implementations require a DER-encoded byte string (i.e. the same format as EC_POINT) 3. Use the encode_ec_point parameter to pkcs11.util.ec.decode_ec_public_key().

Implementation	Other user’s EC_POINT encoding
SoftHSM v2	DER-encoded
Nitrokey HSM	Raw
Thales nCipher	?

If you want to extract the shared key from the HSM, you can mark the key as EXTRACTABLE:

key = private.derive_key(
    KeyType.AES, 128,
    mechanism_param=(KDF.NULL, None, other_public),
    template={
        Attribute.SENSITIVE: False,
        Attribute.EXTRACTABLE: True,
    })
# This is our shared secret key
print(key[Attribute.VALUE])

3

The incompatibility comes from this being unspecified in earlier versions of PKCS #11, although why they made it a different format to EC_POINT is unclear.

Digesting and Hashing

PKCS #11 exposes the ability to hash or digest data via a number of mechanisms. For performance reasons, this is rarely done in the HSM, and is usually done in your process. The only advantage of using this function over hashlib is the ability to digest pkcs11.Key objects.

To digest a message (e.g. with SHA-256):

from pkcs11 import Mechanism

digest = session.digest(data, mechanism=Mechanism.SHA_256)

You can also pass an iterable of data:

with open(file_in, 'rb') as input:
    # A generator yielding chunks of the file
    chunks = iter(lambda: input.read(buffer_size), '')
    digest = session.digest(chunks, mechanism=Mechanism.SHA_512)

Or a key (if supported):

digest = session.digest(public_key, mechanism=Mechanism.SHA_1)

Or even a combination of keys and data:

digest = session.digest((b'HEADER', key), mechanism=Mechanism.SHA_1)

Certificates

Certificates can be stored in the HSM as objects. PKCS#11 is limited in its handling of certificates, and does not provide features like parsing of X.509 etc. These should be handled in an external library (e.g. asn1crypto. PKCS#11 will not set attributes on the certificate based on the VALUE and these must be specified when creating the object.

X.509

The following attributes are defined:

VALUE

The complete X.509 certificate (BER-encoded)

SUBJECT

The certificate subject (DER-encoded X.509 distinguished name)

ISSUER

The certificate issuer (DER-encoded X.509 distinguished name)

SERIAL

The certificate serial (DER-encoded integer)

Additionally an extended set of attributes can be stored if your HSM supports it:

START_DATE

The certificate start date (notBefore)

END_DATE

The certificate end date (notAfter)

HASH_OF_SUBJECT_PUBLIC_KEY

The identifier of the subject’s public key (bytes)

HASH_OF_ISSUER_PUBLIC_KEY

The identifier of the issuer’s public key (bytes)

Importing Certificates

pkcs11.util.x509.decode_x509_certificate() can be used to decode X.509 certificates for storage in the HSM:

from pkcs11.util.x509 import decode_x509_certificate

cert = self.session.create_object(decode_x509_certificate(b'DER encoded X.509 cert...'))

Exporting Certificates

The full certificate is stored as VALUE. Any X.509 capable library can use this data, e.g. asn1crypto or PyOpenSSL.

OpenSSL:

import OpenSSL
from pkcs11 import Attribute, ObjectClass

for cert in session.get_objects({
    Attribute.CLASS: ObjectClass.CERTIFICATE,
}):
    # Convert from DER-encoded value to OpenSSL object
    cert = OpenSSL.crypto.load_certificate(
        OpenSSL.crypto.FILETYPE_ASN1,
        cert[Attribute.VALUE],
    )

    # Retrieve values from the certificate
    subject = cert.get_subject()

    # Convert to PEM format
    cert = OpenSSL.crypto.dump_certificate(
        OpenSSL.crypto.FILETYPE_PEM,
        cert
    )

asn1crypto:

from asn1crypto import pem, x509

der_bytes = cert[Attribute.VALUE]

# Load a certificate object from the DER-encoded value
cert = x509.Certificate.load(der_bytes)

# Write out a PEM encoded value
pem_bytes = pem.armor('CERTIFICATE', der_bytes)