Tag: ssh key management

Before we get to the meat of the discussion, we need to set up some definitions.  Last week we mentioned that the Certificate Authority can produce certificates for both hosts and users.  We’re going to cover both today.  If it looks like we’re being repetitive, we’re really not.  Pay attention to which section you are in when following along, since flags will vary.

Definitions:
• Certificate Authority (CA) – The trusted third party that signs keys to produce certificates.
• User Key – The user public key that will be signed by the CA to produce a user certificate.
• Host Key – The host public key that will be signed by the CA to produce a host certificate.
• User Certificate – The certificate generated by the CA from the user key provided.  This reduces the need for AuthorizedKeysFile or AuthorizedKeysCommand.
• Host Certificate – The certificate generated by the CA from the host key provided.  This simplifies the known_hosts management, and makes this process more secure.
• Principal – A means of restricting validity of the certificate to a specific set of user/host names. By default, generated certificates are valid for all users or hosts.
• Trust – In order for a CA issued certificate to work, the server needs to be told to trust the CA before it will accept user certificates, and the client needs to be told to trust the CA before it will accept host certificates.
• Key Revocation List – A means of revoking keys and certificates when they are no longer valid.
• Validity Lifetime – A means of restricting the lifetime of a certificate’s validity.  If a certificate becomes invalid after a limited time frame, it will need to be re-issued with a new validity lifetime.  This allows for automatic revocation of certificates in case managing the Key Revocation List overlooks an intended removal.
• Additional Limitations – Further restrictions can be applied to the certificates along the same lines as the public key prefix options discussed in a previous blog post.

The first thing we need to do after standing up and hardening the machine where the CA will live is add the unprivileged user that will be used for signing keys to issue certificates.

sudo groupadd -g 3000 sshca
useradd -m -u 3000 -g sshca -G sshca -c "SSH Certificate Authority Signing User" -s /bin/bash -d /home/sshca sshca

Now we need to build out the directory structure.

sudo -i -u sshca
mkdir -p {hostca,userca}


Next, we need to create the key what will be used for issuing HOST certificates.

cd hostca
ssh-keygen -t rsa -b 4096 -f host_ca -C "Host Certificate Signing Key"

We also need to create the key that will be used for issuing USER certificates.

cd ../userca
ssh-keygen -t rsa -b 4096 -f user_ca -C "User Certificate Signing Key"

At this point, there are two files in each directory. The private key file will have no extension, and the public key file will have the “.pub” extension. All certificates will be signed using the private key file, but we also need that public key file, so don’t discard it.

In order to create the TRUST needed for a server to recognize USER CERTIFICATES signed by our CA, we need to push that USER CA public key to each host, and set a configuration option.  You can place it anywhere, but I recommend making a subdirectory under the /etc/ssh directory to store these keys.

sudo mkdir -p /etc/ssh/sshca

Then copy the pub file over from the CA and stick it in this directory. Edit the /etc/ssh/sshd_config file to include this directive:

TrustedUserCAKeys /etc/ssh/sshca/user_ca.pub

Restart sshd (or force it to reload its configuration file) and this trust should now be created.

In order to take advantage of this trust, the user’s logging into the server need their public key to be signed by the USER CA.  This issues a certificate that will need to be given back to the user.

The syntax for signing a key looks like this:

ssh-keygen -s <ca_key> -I <certificate_identity> [-h] -n <principals> -O <options> -V <validity_interval> -z <serial_number> <public_key_to_be_signed>

The “ca_key” is the private key for the USER CA when signing user public keys, or the private key for the HOST CA when signing host public keys.

The “certificate_identity” is a “key identifier” that is logged by the server when the certificate is used for authentication. It is a good idea to use a unique identifier for this that is recognizable by your organization, since you can set up trust for multiple CAs.  For our example, the certificate_identity will be “unixseclab.”

If this is a HOST KEY being signed, ensure that you include the “-h” flag.

The “principals” are a list of users that can be authenticated with this USER CERTIFICATE.  Alternatively, it is a list of hosts that can be authenticated with this HOST CERTIFICATE.  Multiple principals may be allowed, separated by commas.  It is highly recommended that you actually set the principal to the username of the user or hostname of the server it is being issued for.  Blanket authentication can create forensic issues.

The “options” are a list of restrictions that can be applied.  These are like the prefixes we mentioned before.  Be aware that the newest versions of OpenSSH have changed one behavior regarding forced commands. Also note, that “options” are only valid for USER CERTIFICATES.  You would leave off the “-O <options>” when issuing HOST CERTIFICATES.

From Undeadly:

As of OpenSSH 7.4, when a forced-command appears in both a certificate and an authorized keys / principals “command=” restriction, sshd will now refuse to accept the certificate unless they are identical.  The previous (documented) behavior of having the certificate forced-command override the other could be a bit confusing and error-prone.

The “validity_interval” is used to set not only the expiration date of the issued certificate, but to also set a beginning date in case it should only become valid in the future.

Finally, the “serial_number” is an arbitrary number that can be assigned to make KEY REVOCATION easier.

The HOST CERTIFICATE that gets issued should go in the same directory as the HOST KEYS.  The sshd_config file needs to be modified to include a new “HostCertificate” for each new HOST CERTIFICATE issued.  The HOST KEY must also still exist, and must have its own “HostKey” entry in the sshd_config file.  Don’t remove them in exchange for the certificate entries.

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
HostCertificate /etc/ssh//etc/ssh/ssh_host_ecdsa_key-cert.pub

When the server has been configured to offer a HOST CERTIFICATE, the client side needs to also be configured to TRUST the CA that signed it.  To do this, we need to add the following entry to the user’s “known_hosts” file:

@cert-authority *.example.com <public key of the HOST CA that signed the host keys>

It may be necessary to remove existing host key entries in the known_hosts file for this host if it was recently migrated to use certificates.  A clean way to handle this is to make a back up copy of your known_hosts, zero the file out, and add only the certificate lines (by hand.)  Then any time you run into a non-certificate host, you can compare the offered key to your known good key in your backup copy, and accept if it’s good for the hosts that don’t use certificates, yet.

This is a good stopping spot for today’s post.  It ran longer than I expected, so next week we’ll cover Key Revocation Lists, Certificate Inspection, and run the actual example of generating our initial CA, signing a host key, and signing a user key, then using them to allow a client to connect to a server.  I wanted to include that recording this week, but I didn’t realize how long this post was going to get before I planned that out.

Thanks for reading, and a quick “thank you/shout out” to the folks at the CRON.WEEKLY newsletter for the pingback on last week’s article in this series for their issue #63!

The way the OpenSSH Certificate Authority works depends on a few components.  First, there needs to be one trusted signing authority.  This can be any system, and it does NOT have to actively be connected to the network for the client/server handshake to take place using the CA signed keys.  There should also be a Key Revocation List, as well as a means for keeping the KRL updated.  A proper Identity and Access Management (IAM) platform could possibly handle this.  A close second would be a proper Configuration Management / Server Automation tool such as Puppet, Chef, Salt, or Ansible.  We will not cover using either of these tools in this series, but we will (most likely) cover an alternative solution when neither of the prior recommendations is available.  That’s for another day, though.  Today, we’re only going to introduce the basic concepts and fundamentals of how the OpenSSH Certificate Authority works.

Let’s set up the players.  There is a person (User_A) that needs to log into a target machine (Server_A) as himself.  He is coming from his laptop (Workstation_A.)  Normally, User_A would generate his key pair, log into Server_A as himself, and place the public key into the authorized_keys file in his home directory.  Instead, we’re going to pull in a new player that acts as a trusted third party.  This will be the Certificate Authority (CA.)  The CA should be run by a non privileged user on a server that is either not directly connected to the network, or is heavily protected.  The actual privilege of signing should also be restricted to a small team of people with a job role title that allows for this type of provisioning.  For our example, we will assume it is network isolated.

We are assuming the CA is already set up, but here are the steps that should have been taken to do so.  Create the non privileged user (and group) for signing.  Switch to that user and create the CA signing directory structure(s.)  Use ssh-keygen to create the certificate signing key(s.)

There are two types of certificates that can be signed.  The user certificate authenticates users to servers.  The host certificate authenticates hosts to users.  Why do we need both?

A host certificate gives us the ability to stand up a new server in our environment, sign its host keys with the certificate authority, and then the client knows that the new key is okay without prompting the user to trust the key first.  This reduces some of the issues with managing the known_hosts file.

A user certificate gives us the ability to tell the server that our key is okay without having to place the key on the server first.  This removes some of the issues with managing key distribution.

A signed user certificate can place restrictions on the signed public key, including all of the restrictions we discussed with the pre-amble section to the authorized_keys entries.

Let’s look at the broad overview work flow for today.  Next week, we will cover the commands needed to stand up that certificate authority structure listed above, plus the commands to sign both host and user certificates.

Work flow scenario: a new machine is built.  The host keys are regenerated (if for example this is a cloned virtual machine) and signed by the Certificate Authority.  This signed certificate is placed back onto the new machine, and that’s all that is needed, as long as the clients are configured correctly.  For the client to take advantage of this, the client needs a special known_hosts entry that begins with @cert-authority and is followed by the public key for the signed host certificates.  When the user goes to log into the new machine, the connection flow will include the server presenting a host certificate to the client, who then checks that the known_hosts “@cert-authority” entry can decipher the host certificate, and the connection is then accepted on success.  This helps prevent confusion on intentionally newly built systems when IP or hostnames change regularly.

Work flow scenario: a new user needs access to a system.  The user generates their key, sends the public key to be signed, and when the certificate is received, places it in their .ssh directory with the rest of the key related files.  The host machines have already been configured to trust the certificate authority in the sshd_config file.  When the user goes to connect with ssh, the client presents the signed certificate to the target machine.  The target machine’s sshd opens the TrustedUserCAKeys entry to open the appropriate public key to decode the certificate.  If this is successfully decoded, the connection is trusted as if the key were in authorized_keys for that user.  This helps reduce the heavy work load of managing multiple authorized_keys files per user.

Of course, there is more to it than this, but we’ll go into the finer details over the next few weeks.  Next week will be an explanation of the commands needed to set up the CA, (including revocation lists, and why they are important.)

Thanks for reading!