SSH is widely used and considered to be the standard for remotely managing Linux systems; however, it was designed in 1995, when computers were accessed by a handful of people in a local network. To help SSH scale to current cloud scale, using legacy SSH Keys that do not expire becomes impossible; this is why organizations such as Facebook, Uber, Netflix, and Keytos have transitioned to SSH Certificates.
The user experience with SSH is far from ideal. Setting up new users is a slow, manual process. Security alerts when connecting to new hosts are often perplexing, leaving users with unfamiliar credentials and little advice on handling them.
Managing SSH on a large scale is problematic. The process of key approval and distribution is inefficient and time-consuming. Reusing host names is simply not possible. Homemade tools distribute key information throughout your network, creating a mess that needs cleaning up when removing users and let’s be honest, when was the last time you rotated your shared key when a user left?
SSH tends to promote poor security habits, making SSH one of the weakest points in your infrastructure. Since rekeying is complex, it’s frequently avoided. Users often have direct access to key material and are led to reuse keys across multiple devices. Keys are permanently trusted, meaning errors lead to vulnerabilities.
We know that you’re nauseous right about now, but put down the ginger ale! These issues can be resolved! While none of these problems are inherent to SSH itself, they do stem from how SSH public key authentication is implemented. The answer to these problems lies in transitioning to certificate authentication. Using SSH certificate authentication simplifies usage, improves operation, and enhances security.
At Keytos, we have been pushing for SSH certificates for our entire lifespan (we even talk about them in one of our very first blogs!). That being said, are other people and organizations taking note of SSH certificates just as we have?
The short answer is yes. Practically every organization that is operating at scale and has a good grasp of cybersecurity is using SSH certificates (here’s looking at you, Facebook, Uber and Netflix).
Of course, those are the biggest companies with the largest information networks – why wouldn’t they be on top of the latest goings-on in the world of SSH? For those smaller or mid-size organizations, though, it is more challenging to keep up with SSH trends. After all, certificates and PKI are pretty hard to grasp, and people don’t always see the upside right away. Additionally, SSH certificates haven’t received anywhere near the attention that they deserve. Most people outside of these constantly-updating large organizations have never even heard of them. And that’s a shame.
That’s why we’re here. To educate. To inform. To spread the good word of SSH certificates! Before we can dive into why SSH certificates are the best, it’s important to first understand the difference between public key authentication and certificate authentication.
The majority of SSH setups utilize public key authentication. This method employs asymmetric (public key) cryptography, creating a unique public/private key pair for each user and host for authentication purposes. Asymmetric cryptography’s key feature lies in the unique relationship between a public and private key. Data signed with a private key can be verified using the corresponding public key. Like hashing, it’s virtually impossible to counterfeit a signature; therefore, if a signature can be verified and the private key’s owner is known, the origin of the signature is confirmed.
A basic form of authentication can be achieved by asking someone to sign a randomly generated large number. For instance, if I were to establish a connection to you, send you a random number, and you return a valid signature of that number, it would confirm that I am communicating with you. While this is a rather simplified explanation, it broadly describes how SSH public key authentication operates. Certificate authentication functions similarly, with a key difference that will be discussed shortly.
To use SSH with public key authentication, the host must recognize your public key. Typically, your public key should be placed in ~/.ssh/authorized_keys. Managing this file for numerous users across a network is complex and prone to mistakes.
Initiating SSH with public key authentication often begins with a complicated sequence of ssh-keygen commands, usually sourced from a guide or, more commonly, from online forums. The next step involves submitting your public key for approval and distribution, which is often a manual and non-transparent process. You might need to email an administrator or create a helpdesk ticket, then wait. Fun, we know. Meanwhile, an operator is tasked with adding your key to a directory in a repository and initiating a deployment. Once this is completed, you gain SSH access. Since these key associations are permanent, your SSH access remains active indefinitely (unless someone actively removes it).
Certificate authentication does away with the need for key approval and distribution. Rather than dispersing public keys through static files, it associates a public key with a specific identity using a certificate! A certificate is essentially a data format that encompasses a public key, an identity, and additional information such as an expiry date and permissions. This data format is then signed by a CA.
b$ ssh-keygen -L -f testkey-cert.pub
testkey-cert.pub:
Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
Public key: ECDSA-CERT SHA256:O6M6oIjDm5gPm1/aTY619BgC3KSpS4c3aHVWxYh/uGQ
Signing CA: ECDSA SHA256:BE2EXJGoPv2LA6yEbjH+sf9JjG9Rd45FH1Wt/6H1k7Y
Key ID: "brett@example.com"
Serial: 4309395459650363134
Valid: from 2023-11-11T11:11:11 to 2023-11-11T12:11:11
Principals:
root
Critical Options: (none)
Extensions:
permit-X11-forwarding
permit-agent-forwarding
permit-port-forwarding
permit-pty
permit-user-rc
To activate certificate authentication, set up both clients and hosts to authenticate certificates with the public key of your CA, meaning they should trust certificates issued by your CA. On every host, you need to modify the /etc/ssh/sshd_config file. This involves specifying the public key of the CA to verify user certificates, as well as the private key and certificate of the host, as seen in the code sample below:
# Path to the CA public key for verifying user certificates
TrustedUserCAKeys /etc/ssh/ssh_user_key.pub
To quote the Looney Tunes end screen, “That’s All Folks!” That is all that you have to do to start employing SSH certificate authentication, well, here is a more detailed version on how to create your own SSH Certificate Authority. If you want, you can even use it in conjunction with public key authentication to ease the transitionary process.
You’re probably wondering, “that’s great and all, but why should I use certificate authentication instead?” We’re glad you asked.
Removing the need for SSH key approval and distribution brings immediate operational advantages. It frees up operational resources from repetitive key management tasks and does away with the continuous expenses involved in monitoring and maintaining custom solutions for adding, removing, synchronizing, and auditing static public key files throughout your network.
Additionally, the option to issue SSH user certificates through various authentication methods enhances operational automation. This means that automated processes like cron jobs or scripts requiring SSH access can secure a temporary SSH certificate as needed, rather than relying on a preset, long-lived static private key.
The SSH protocol is fundamentally secure, but public key authentication fosters various poor security practices, making it difficult to maintain optimal security hygiene. In public key authentication, keys are trusted indefinitely. This means a compromised or improperly authorized key might remain undetected or unaddressed for an extended period. Lack of proper key management, such as failing to remove keys of former employees, often leads to SSH being vulnerable to unauthorized access.
Conversely, certificates have expiration dates. In any incident, whether it’s an error, theft, misuse, or key exfiltration, compromised SSH credentials will automatically become invalid after a certain period, even if the incident isn’t noticed or reported. SSH certificates are designed to be fail-secure; access naturally terminates unless actively renewed. Plus, regular check-ins with your CA for credential renewal create an automatic audit trail.
Rekeying for users is also cumbersome with public key authentication. The tediousness of key approval and distribution discourages users from rekeying, despite available tools. This leads to the risky practice of copying and reusing private keys across devices, sometimes for extended periods. Such reuse of keys is a significant security risk. Private keys should never be transferred across networks, but public key authentication exposes users to these sensitive keys and fails to provide effective key management tools, inviting misuse and abuse.
An SSH CA, along with a user-friendly client, can streamline key generation and shield users from many complexities. While certificate authentication doesn’t eliminate all security risks, it supports SSH workflows that are more straightforward, user-friendly, and less prone to errors.
At Keytos, we are staunch believers that SSH certificates are the backbone of the optimal SSH flow.
To do SSH, users start by executing a login command in their terminal. This triggers the opening of a browser and the start of an SSO process within your organization’s identity provider.
Using a web-based SSO process simplifies the integration of robust MFA (Multi-Factor Authentication) methods like FIDO U2F, along with other advanced authentication features offered by the identity provider. Users log in using a familiar procedure, and removing a user from the main identity system promptly revokes their SSH access.
After completing the SSO, the user receives a bearer token (like an OIDC identity token), which the login utility uses. This utility then creates a new key pair and requests a signed certificate from the CA, authenticating and authorizing the request with the bearer token.
The CA issues a certificate valid for about a workday (16-20 hours). The login utility automatically loads the signed certificate and its corresponding private key into the user’s ssh-agent.
Users don’t need to know these details. They just need to understand that to access SSH, they must run EZSSH first. After this, they can use SSH as usual:
$ ezssh ssh -e brett@10.7.0.1
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 4.15.0-1036-gcp x86_64)
Last login: Wed Dec 20 04:05:55 2023 from 10.7.0.9
brett@pi:~$
Similar to browser cookies, the short-lived certificates provided are ephemeral, lasting just enough for a workday. Logging into SSH is akin to logging into a website; it’s a straightforward process that needs to be done at most once daily. This infrequency allows for the use of strong MFA without it becoming a nuisance or desensitizing users.
Each login generates new private keys and certificates that are never stored on disk. Directly placing them into ssh-agent keeps users away from sensitive credentials (you can even set it up so the key is not saved anywhere making your certificates truly a one-time use). If a user wants to connect from another device, it’s simpler to run EZSSH there than to transfer keys from ssh-agent.
While some critics argue that SSH certificate authentication is new, lacks broad support, and doesn’t have practical tooling, the reality is different. Certificate authentication was introduced in OpenSSH 5.4 nearly 14 years ago. It’s well-tested and employed in large-scale operations. The necessary tools to implement this ideal SSH workflow are available now.
EZSSH simplifies the whole process by eliminating the need for handling, updating, and deleting SSH keys for all users across various hosts. This is achieved by utilizing SSH certificates discreetly, while users experience a seamless Single Sign-On (SSO) using their AAD Identity. This approach ensures that users no longer need to store keys on their personal computers, reducing the risk of these keys being compromised by malicious entities.
