Let’s talk about SSH, SSH keys, and SSH certificates. SSH has been the de facto way of gaining access to and managing Linux systems for the last 3 decades. While robust, capable and useful, it unfortunately wasn’t designed to meet the needs of organizations today that are focused on efficiency, security and scale. Although SSH does make it easy to manage machines through an interactive command line, identity management in SSH is a pain. One of the core problems is that we can’t federate our precious corporate identities that we spend lots of time and money; adding MFA, conditional access and actively audit / lifecycle. Instead, we must rely hard to manage decentralized identities and credentials such as SSH keys or even worse passwords. These “old school” identity techniques do not scale with the needs of modern enterprises operating at scale.
One of the core tenets of identity lifecycle management is having a strong onboarding and offboarding process. With standalone, traditional SSH onboarding a new user is a pain, you must go to each of the endpoints and add the user key to the authorized_keys file. While this is (hopefully) done by a central security team, that (hopefully) maintains an inventory of who owns each key and (hopefully) does audits on each key to make sure they are still needed.
What we have seen is that in practice that (hopefully) is the exception to the norm and if done correctly, is a very costly operation. Furthermore, even if the technical best practices are followed, operational breakdowns still occur, for example, a new requestor simply asks another engineer that already has access to the machine and just adds the new key. This creates a huge security hole, since there is no central administration of keys and fear. We’ve also had clients express the fear of breaking something when removing a key has created an environment where keys are added but never deleted. Tatu Ylonen the inventor of SSH mentioned that in his experience he has seen between 50-200 keys per server with 90% of them not being used. This opens the door for attackers to find one of those keys and be able to access the endpoints, such as, ex-employee who gains access to your the endpoints and disrupts your services as in a similar attack to the one Cisco suffered in 2018.
Simply put, SSH Keys are hard to manage and most engineers are not given proper security training on best practices on how to keep their private keys safe. Having a public and private key, adding them to the endpoint, having trouble, trying to debug this, and all in getting on the way on doing their “real job” makes engineers cut corners, such as positing a key for all the team to use in the team wiki, or leaving their keys without a passphrase in their desktops, and many other horrifying practices our clients have shared with us in confidence.
The good news is that there is a solution to this: SSH Certificates. They can be issued with an expiration date, and Linux endpoints support them out of the box. From an operational standpoint, you only have to add the CA (Certificate Authority) to the trusted CA file and after that; all certificates issued by that CA that match the machine requirements will grant you access to that machine. Removing the need of adding and removing each user to each of your endpoints.
This is a proven approach taken by industry leaders that take security seriously. Facebook released a whitepaper explaining how to implement SSH Certificates, but didn’t address the challenge that the process is very manual and an admin has to sign each user certificate. While an improvement, it again is just an iterative improvement. Facebook mentioned that they had a tool that automated that process but unfortunately it is only for their own systems. All of this is to say what happens is most organizations without a world class security engineering team use SSH keys or SSH certificates but really struggle to manage both operationally and leave themselves vulnerable.
For more details on SSH Certificates, check out our “how SSH Certificates work” blog post
At Keytos we don’t believe in status quo, we were founded out of the need to do better than “good enough”. We think SSH deserves the “Keytos treatment” and as of today we are launching EZSSH, built on SSH Certificates and reliable and secure automation. We built an SSH client solution that:
Usually, Identity management systems need to have an agent with admin privileges running on your endpoints to add and remove users. In recent attacks such as the Solar Winds attack the security community was reminded of the risks and challenges associated with using 3rd party tooling and infrastructure. Since SSH Certificates are natively supported by OpenSSH and only the CA key has to be added to the endpoint, EZSSH does not need to run an agent. The only step to get your endpoint to trust the certificate is to run a Script that adds the CA key to the trusted CAs and then the certificates issued with that CA will be trusted by the endpoint.
Here is a sample script:
#!/bin/bash
authkeys=$(awk '$1 ~ /^TrustedUserCAKeys/' /etc/ssh/sshd_config)
if [ ! -z "$authkeys" ]
then
echo "TrustedUserCAKeys found removing from file"
awk '$1 !~ /^TrustedUserCAKeys/' /etc/ssh/sshd_config > tmp.txt
cat tmp.txt > /etc/ssh/sshd_config
rm tmp.txt
fi
authPrincipals=$(awk '$1 ~ /^AuthorizedPrincipalsFile/' /etc/ssh/sshd_config)
if [ ! -z "$authPrincipals" ]
then
echo "AuthorizedPrincipalsFile found removing from file"
awk '$1 !~ /^AuthorizedPrincipalsFile/' /etc/ssh/sshd_config > tmp.txt
cat tmp.txt > /etc/ssh/sshd_config
rm tmp.txt
fi
echo "Adding trusted CA file to /etc/ssh/sshd_config"
echo "TrustedUserCAKeys /etc/ssh/trusted_ca_keys.pub" >> /etc/ssh/sshd_config
echo "Adding Authorized Principals file to /etc/ssh/sshd_config"
echo "AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u" >> /etc/ssh/sshd_config
echo "Adding CA Public Key"
echo "YOUR_CA_PUBLIC_KEY" >> /etc/ssh/trusted_ca_keys.pub
EZSSH fixes what is broken with SSH keys and SSH certificates, and it has to do with scale, management and human error (operational challenges), all of these improvements lead to better security which is directly correlated to reduced operational costs and improved risk management posture. We’ve improved SSH by implementing automation and helping IT leaders carry out security operations best practices.
Unfortunately, typical users do not follow best practices, and since SSH keys do not have expiration dates, it makes it very easy for your security team to overlook a compromised key / a key from a former employee or to not remove any keys at all because they might be afraid of breaking something in production.
SSH Certificates fix all those problems by having timebound certificates that are only used for a few hours, with fewer keys to manage in the endpoint (only the CA key), and EZSSH managing the ephemeral private keys for the user, it leaves little room for the bad practices to expose your organization to a compromise.
EZSSH does not only make it harder for attackers to get in, it also helps your security team catch unauthorized access by providing easy to consume audit logs. The logging capabilities built into EZSSH can be correlated with your Azure AD authentication logs and your SSH access logs, bringing the two technologies into a “single pane of glass” for them to analyze.
The best part of all of this? The User experience improvements! As an engineer I always hated having to wait to get access to a new server. First I had to create a new key, send the public key to the admin, wait for them to add it, and then once I got tired of waiting and started working on something else. I got an email saying that the key was added.
With EZSSH, this grueling process is gone! Now to be fair, depending on your access policies, you might have to join an AAD group or request for approval in EZSSH. However, once you are added, all you have to do is run:
ezssh ssh -e username@endpoint
We will take care of the rest. No more managing SSH keys! Your manager gave you a list of endpoints you have to connect to and change something? Use our batch ssh command
ezssh batchSSH -f endpoints.csv
Or even better pass the command that you want to use:
ezssh batchSSH -f endpoints.csv -c “echo ezssh rocks!”
In summary, EZSSH enhances the foundations of SSH by utilizing SSH certificates that are centrally managed, automate SSH key management, with no agent - in an easy to use, secure, and scalable fashion.
So what are you waiting for? Make your engineers work less, and your organization more secure by talking to an identity expert today!