5.

The Protocol

So, given what we now know about asymmetric encryption, certificates and RSA, let’s put it
together in a single protocol:

1. Alice needs a certificate:

a. She chooses a public exponent ​e​ and generates a private key
b. She generates a public key from that private key
c. She generate a certificate signing request ➝ sends that to the CA
d. The CA generates a certificate for Alice and signs it ➝ sends it to Alice
2. Alice gets Bob’s certificate from him:
a. She verifies it using the CA’s certificate (already pre-installed on her computer)
b. She extracts Bob’s public key from it
c. She attempts to encrypt her large message using Bob’s public key ➝ error!
3. Alice picks a symmetric key:
a. She picks a strong symmetric key using a pseudo-random number generator
b. She encrypts it with Bob’s public key ➝ [Link]
c. She hashes it and then encrypts it with her private key ➝ [Link]
d. She sends both [Link] and [Link] to Bob
4. Bob deciphers and verifies the symmetric key:
a. He decrypts [Link] using his private key
b. He gets and verifies Alice’s certificate and extracts her public key
c. He decrypts [Link] using Alice’s public key
d. He compares a hashed symkey with the decrypted signature, they must match
5. Alice encrypts her large message with that symmetric key:
a. She needs to use choose a symmetric key encryption algorithm
b. Bob decrypts using the symmetric key and that same algorithm
c. She can also use something like ​HMAC​ now for authentication (this won’t be
covered in this document but it’s similar to how she creates her signature, and
you can read more ​here​)

Note​: In real life, the protocols used are a little more complicated than this. You’ll notice that
both parties need to be using the same hashing and encryption algorithms, requiring more
initial communication (this is done in the TLS handshake for example).

9
6. OpenSSL Demo

Here we’ll implement all the steps of that protocol, using openssl terminal commands. In
practice you’re more likely to use openssl in the form of an API in another language- but
learning the terminal commands is still valuable as a transferable skill. Each command is
displayed with some explanations of its flags below.

If you want to follow along, you can make 3 folders, 1 for Alice, Bob and the CA respectively.
You need to repeat steps 1.a and 1.b for Bob and CA so they can have their own pair of keys.
And you need to generate a self-signed certificate for the CA (shown below).

Step 1.a - Alice generates a private key

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -pkeyopt

rsa_keygen_pubexp:3 -out [Link]

● genpkey ➝ ​generate a private key

● -algorithm RSA ➝​ use the RSA algorithm (can also take “EC” for elliptic-curve)
● -pkeyopt opt:value ➝​ set opt to value (see items below)
● rsa_keygen_bits:2048 ➝ ​sets the size of the key to 2048 bits (the default is 1024)
● rsa_keygen_pubexp:3 ➝ ​sets the public exponent ​e​ to 3 (default is 65, 537)
● -out [Link] ➝​ outputs to the file [Link]

Step 1.b - Alice generates a public key

openssl pkey -​in​ [Link] -pubout -out [Link]

● pkey ➝ ​processes public or private keys

● -in [Link] ➝ ​read the key from filename [Link]
● -pubout ➝ ​output a public key (by default, a private key is output)

Aside: viewing the keys in plain text

The keys are saved in base64, and aren’t human readable if you open them in a text editor or
the terminal. Luckily, openssl provides us with a handy set of commands to convert them to
text. The (​-noout​) flag suppresses the command from printing out the base64 encoding as well.

openssl pkey -​in​ [Link] -text -noout

openssl pkey -pubin -​in​ [Link] -text -noout

10
Note the size difference in the private key and the public key (one is a subset of the other,
afterall). There are some additional values stored in the private key that you won’t recognize
(exponent1, exponent2 and coefficient). These are stored by openssl to speed up decryption.

Step 1.c - Alice generates a certificate signing request

openssl req -new -key [Link] -out [Link]

● req ➝​ creates and processes signing requests

● -new ➝​ generates a new certificate request, will prompt Alice for some information
● -key [Link] ➝ ​signs the request with Alice’s private key

The command will prompt Alice with these questions:

● C​ountry code [​C​]: {Alice fills in her country code}
● Province/​ST​ate name [​ST​]: {Alice fills in her province name fully}
● City/​L​ocation [​L​]: {The city Alice’s business is registered in, for example}
● O​rganization Name [​O​]: {Alice’s business name, for example}
● O​rganizational ​U​nit Name [​OU​]: (Optional) {What part of the company is she?}
● C​ommon ​N​ame [​CN​]: the hostname+domain, i.e. “[Link]”
● A challenge password []: {this can be used as a secret nonce between Alice and CA}

Aside: generating a self-signed certificate for the CA

openssl req -x509 -new -nodes -key [Link] -sha256 -days 1024 -out
[Link]

Step 1.d - CA generates and signs a certificate for Alice

openssl x509 -req -​in​ [Link] -CA [Link] -CAkey [Link]

-CAcreateserial -out [Link] -days 500 -sha256

● x509 ➝ ​an x509 certificate utility (displays, converts, edits and signs x509 certificates)
● -req ➝ ​a certificate request is taken as input (default is a certificate)
● -CA [Link] ➝ ​specifies the CA certificate to be used as the issuer of Alice’s certificate
● -CAkey [Link] ➝ ​specifies the private key used in signing ([Link])
● -CAcreateserial ➝ ​creates a serial number file which contains a counter for how many
certificates were signed by this CA
● -days 500 ➝ ​sets Alice’s certificate to expire in 500 days
● -sha256 ➝ ​specifies the hashing algorithm to be used for the certificate’s signature

11
Aside: viewing the certificate as text

openssl x509 -​in​ [Link] -text -noout

Step 2.a - Alice verifies Bob’s public certificate

openssl verify -CAfile [Link] [Link]

● verify ➝ ​a utility that verifies certificate chains

● -CAfile [Link] ➝ ​specified the trusted certificate ([Link])
● [Link] ➝​ the certificate to verify
● If you get an OK, you know the certificate can be trusted

Step 2.b - Alice extracts Bob’s public key

openssl x509 -pubkey -​in​ [Link] -noout > [Link]

● -pubkey ➝ ​outputs the certificate’s public key (in ​PEM format​)

Step 2.c - Alice tries to encrypt her [Link] with Bob’s public key

openssl pkeyutl -encrypt -​in​ [Link] -pubin -inkey [Link] -out

[Link]

● pkeyutl ➝​ utility to perform public key operations

● -encrypt ➝​ encrypt the input data
● error! ​(recall: RSA is not meant for encrypting arbitrary large files- Alice needs to use
symmetric key encryption for that)

Step 3.a - Alice generates a symmetric key

openssl rand -base64 32 -out [Link]

● rand ➝​ generates pseudo-random bytes (seeded by default by $HOME/.rnd)

● -base64 32 ➝​ outputs 32 random bytes and encodes it in base64

12
 Step 3.b - Alice encrypts [Link] using Bob’s public key

openssl pkeyutl -encrypt -​in​ [Link] -pubin -inkey [Link] -out

[Link]

Step 3.c - Alice hashes [Link] and encrypts it using her private key

openssl dgst -sha1 -sign [Link] -out [Link] [Link]

● dgst -sha1 ➝​ hash the input file using the sha1 algorithm
● -sign [Link] ➝​ sign the hash with the specified private key
● [Link] ➝​ the input file to be hashed

Step 4.a - Bob decrypts [Link] using his private key

openssl pkeyutl -decrypt -​in​ [Link] -inkey [Link] -out

[Link]

● -decrypt ➝​ decrypt the input file

Step 4.b - Bob gets and verifies Alice’s certificate and extracts her public key

(This is simply a retread of what Alice did in step 2)

Step 4.c - Bob verifies the message is from Alice

Steps 4.c and 4.d in the protocol are combined in this step. Bob hashes [Link], decrypts
[Link], and compares the two results in one command:

openssl dgst -sha1 -verify [Link] -signature [Link] [Link]

● -verify [Link] ➝​ verify the signature using the specified filename

● -signature [Link] ➝ ​specifies the signature to be verified
● [Link] ➝ ​the file to be hashed

Step 5.a - Alice encrypts her [Link] with the symmetric key

13
 openssl enc -aes-256-cbc -pass file:[Link] -p -md sha256 -​in
[Link] -out [Link]

● enc -aes-256-cbc ➝ ​encrypt a file using the aes-256-cbc symmetric key algorithm
● -pass file:[Link] ➝​ specified the file to get the symmetric key from
● -p ➝​ prints the key, salt, initialization vector to the screen
● -md sha256 ➝​ uses sha256 as part of the ​key derivation function​ (a f​ unction that
derives one or more secondary secret keys from a primary secret key)

Step 5.b - Bob decrypts [Link] with the same symmetric key

openssl enc -aes-256-cbc -d -pass file:[Link] -p -md sha256 -​in

[Link] -out [Link]

● -d ➝​ decryption flag

7. References

[Link]

[Link]

[Link]

[Link]

[Link]

14