Category: SSH Start to Finish Architecture

The SSH Start to Finish Architecture series.

SSH Start to Finish Architecture – X11 Forwarding

I was reviewing previous posts, and realized I haven’t really covered this aspect of forwarding, yet. I would be remiss to leave it out.

X11 is a client/server protocol, which means you can run software on one machine, and display it’s graphical output on another. It also has some inherent security risks, so a common way to mitigate some of those risks is to allow SSH to forward the X11 client connection to your local X11 server when you start a client on a remote system.

It seems backwards for some people, but you run the server on your workstation, and you run the remote graphical command as a client that calls back to your server. The server generates the graphics on behalf of the client. If you’re running a workstation with Linux, a BSD derivative, or something like one of the OpenSolaris forks, you are likely already running X11 for your desktop needs. We will make the assumption that you are for this process.

In order to do X11 forwarding, the remote server needs to be configured to allow such forwarding. The settings that matter are: “X11Forwarding yes” to turn on the forwarding, “X11DisplayOffset 10” (default) to determine the offset for the display that will be used, “X11UseLocalhost loopback” (default) to tell sshd to bind the forwarding server to the loopback device, and “XAuthLocation /usr/X11R6/bin/xauth” (default) if you need to provide a path to the xauth program, because it’s not in the default location on your system.

It may be that the only setting you need to adjust is “X11Forwarding” from “no” to “yes” on your target system.

Once you’ve done this, you can make your connection to the target system by passing the -X or -Y flags to the ssh client. The -X flag enforces the ForwardX11Trusted settings for the ssh client, which set higher restrictions on how the X11 forwarding can be used, as well as setting a 20 minute expiration timer on the xauth token. The -Y flag does not set these restrictions. It’s up to you to decide which flag you want to use.

After you connect, you can check that your environment is set up appropriately. You should have a “.Xauthority” file in your home directory, and you should have an environment variable already set called ${DISPLAY} that should probably show “localhost:10.0” when you echo it out.
ls -ld .Xauthority
echo ${DISPLAY}

After you’ve confirmed these, you can test your forwarding with something simple, such as xeyes, or xclock, if either of those are installed on the target machine. If they are not, go ahead and try whatever X11 program you intended to run. You should see the program show up on your desktop once you do.

Finally, if you have need of running an X11 program as a different user, you can use the xauth command to merge your .Xauthority tokens with the other user’s environment and then switch to that user to run your command. You will need to extract your xauth token for your DISPLAY, and merge it for the other user’s environment. The standard way to do this is with “xauth extract” piped to “xauth merge” as shown in the full session example below.

ssh -Y User_A@Server_B
ls -ld .Xauthority
echo ${DISPLAY}
xclock
xauth extract – ${DISPLAY} | sudo -i -u User_B xauth merge –
#OR xauth extract – ${DISPLAY} | sudo su – User_B xauth merge –
sudo -i -u User_B #(or sudo su – User_B)
echo ${DISPLAY} #(May need to manually set this to what you were given upon login)
xclock

The client configuration has several settings to always or never set this for you. These should probably be set in Match blocks for just the servers you need to run X programs on regularly, and not set at all otherwise.

ForwardX11 yes/no
ForwardX11Trusted yes/no
ForwardX11Timeout

The time format will be a number followed by a modifier unit. “S” or “s” for seconds. “M” or “m” for minutes, and so on all the way up to weeks. No unit indicates seconds by default.

You can use xauth commands to delete your tokens manually, when you are done by doing “xauth remove ${DISPLAY}” if you so desire.

Hopefully this helped shed some light on how to get X11 Forwarding working from a basic to complex scenario. This is one of the most commonly asked questions I’ve had in the past, and I’m sorry it wasn’t covered sooner.

If you have any questions on this, leave a comment. Thanks for reading!

SSH Start to Finish Architecture – Forced Commands

Last week, we covered SSH Key options that can restrict how a private key is used to connect to a server. One of those options was the “command=” option, which allows restricting the key to calling only one command, regardless of the command issued as part of the ssh connection attempt. There are actually several ways to enforce this.

You can do the public key “command=” option we already covered. You can also use the sshd_config settings to apply a ForceCommand option. This is most useful for applying the same kind of forced command scenario we described last week via a “Match User” directive. It’s also useful for applying an sftp-only situation to a given user, so that the only thing the user can do is transfer files. The option would look like this, if that is your goal:

ForceCommand internal-sftp

The user’s shell needs to be a valid one for this to work, since the forced command is invoked via “ -c.” This means a shell of /bin/false or /bin/nologin is a no go. Since you are forcing a command, this should be less of a problem, though.

Finally, there is also a way to force this command with ssh-keygen options when signing a key via the certificate authority system for OpenSSH, but we will go into more detail on that when we get to the CA stuff.

The force command options don’t allow running the user’s ~/.ssh/rc, so that would not be a work around that the user could use to hack this system. ControlMaster overrides the public key force command if the option is set after a master session has already been established, so you may need to terminate all ssh connections for that user after making changes that enforce restrictions, moving forward.

A lot of people grumble about the force command options, because they believe that a single key is needed for each command that gets passed, but there is actually a means for handling that. There is an environment variable that gets set when an ssh session that wants to call a remote command is used to connect while force command is in use for a user. That environment variable is SSH_ORIGINAL_COMMAND, and it retains the command that was requested. This means you could have a wrapper script that is your forced command, have it check this environment variable for sanity (are all of the commands in the list provided in our whitelist, or not? If not, log a rejection and terminate. If so, log the call and execute.) The variable is unset if the user just tries to ssh in without calling a remote command, so be sure to check for that if you go this route. Assume if the variable is unset/empty that they tried to log in for an interactive session, and handle that however you feel is best. I would assume “log a rejection and terminate” is better, though, since an interactive session can’t be restricted properly without a restricted shell, that may still be jail broken if misconfigured. Your own needs may vary, though. Just be very thorough in your design, and be sure to sanitize all input before executing, and you should be fine.

Thanks for reading!

SSH Start to Finish Architecture – Public Key Options

Since we’ve introduced a few ways of doing tunneling, I thought now would be a good time to bring out some ways of restricting what a user can do in the event that they have access to a private key they shouldn’t.

We already talked about setting a pass phrase on the private key side, but there is still the chance that the user was able to brute force that, so let’s assume they have usable access to that private key. What options do we have for keeping things secure?

For starters, we can restrict which source IP/DNS names a user can come from using this key. If we modify the public key entry in the authorized_keys file to include a “from” directive before the key starts, we can provide a list of IP addresses/hostnames we expect to be coming from. This prevents an attack from the user’s workstation, for example. One caveat of this is, you can’t be coming through a source NAT, since that would reduce the effectiveness of this restriction. Everyone coming into the box from a source NAT VLAN or subnet would look like they are coming from the same place.

In order to set this directive, we will need to modify the public key by hand. It’s better to modify the public key before adding it to the authorized_keys file, but you can modify the authorized_keys file entry for this key, if you prefer.

To use this directive, we need to put the a “from” option with all of its values, comma separated, BEFORE the key starts. An example of how this might look follows:
from=”172.16.84.1,lanturtle,lanturtle.mydomain” ssh-rsa AAAA User_A@lanturtle

Now, let’s assume you also want to prevent proxying with this key. In order to restrict that, we need to set several options:
no-X11-forwarding – Prevents forwarding X11 sessions back to an X server via the SSH tunnel.
no-port-forwarding – Prevents port forwarding via TCP, TUN devices, and direct stdin proxying.
no-agent-forwarding – Prevents “forwarding” of the ssh-agent for handling the private key.

To add these to the example above, we would use a comma separated list of options like this:
from=”172.16.84.1,lanturtle,lanturtle.mydomain”,no-X11-forwarding,no-port-forwarding,no-agent-forwarding ssh-rsa AAAA User_A@lanturtle

We can further restrict this to only allowing this particular key to be used for running a specific command. Doing this ignores any commands passed by the user upon connection, and prevents the user from obtaining a full login shell (unless that’s the forced command.)
command=”/usr/local/bin/ssh-restricted-command.sh” (where “/usr/local/bin/ssh-restricted-command.sh” is the command you want it to run. Put the actual command you want to use, here.)
from=”172.16.84.1,lanturtle,lanturtle.mydomain”,no-X11-forwarding,no-port-forwarding,no-agent-forwarding,command=”/usr/local/bin/ssh-restricted-command.sh” ssh-rsa AAAA User_A@lanturtle

Finally, if we want to be sure that there isn’t a land mine waiting on the user’s .ssh/rc file, we can tell it to not read this by using the following option:
no-user-rc
from=”172.16.84.1,lanturtle,lanturtle.mydomain”,no-X11-forwarding,no-port-forwarding,no-agent-forwarding,no-user-rc,command=”/usr/local/bin/ssh-restricted-command.sh” ssh-rsa AAAA User_A@lanturtle

When we get into managing a certificate authority, you’ll find that these options can be set by ssh-keygen for certificate related keys, but the option names are slightly different. Unfortunately, using ssh-keygen to generate the initial public/private key with these options doesn’t work. It only works on certificates. Modifying the public key entry by hand does restrict this, however.

We will go into more detail about the forced command option (and setting one in sshd_config) next week.

Thanks for reading!

SSH Start to Finish Architecture – Tunneling IP with TUN device

We’ve covered the various tunneling / proxy capabilities of OpenSSH for tunneling TCP and setting up port proxies, but the “-w” flag can let you forward IP traffic. This happens by way of creation of a temporary “tun” device upon SSH connection, which can be configured like any network interface by standard tools such as “ifconfig.” There is some post connection set up that has to happen, including modifying route table information, so treat this like a right and proper VPN solution. That’s what it is meant to be. A VPN solution built into the SSH client and server software. I will do a proper write up with lab and examples later. For now, know that the server has to have the “PermitTunnel” option set, and it must be set to a value other than “no,” which is the default value.

The options are:
point-to-point – This allows a Layer 3 tunnel device to be configured.
ethernet – This allows a Layer 2 tunnel device to be configured.
yes – This allows either device type to be configured.

When connecting to the server, you need to pass the “-w” flag and provide a tun device id. You can also provide the remote tun device id. If you provide no remote id, it defaults to “any” which provides the next available. You may also specify “any” for the local tun device id.

The following example sets up a specific “tun0” on both the local and remote side.
ssh -w 0:0 User_B@Server_B

The following will set up a “tun” device using the next available ID on both the local and remote sides.
ssh -w any:any User_B@Server_B

The following will do the same as the example above, but saves 4 characters worth of typing.
ssh -w any User_B@Server_B

Just setting these up will not be sufficient. You will also need to configure the routes/interface information, of course. You will also need to configure any bastion host (local firewall) traffic rules as well, if you’re running one. The best explanation for this I’ve found is here:
Daemon Forums

When I do this as a lab exercise, I will show all of the detailed configuration for both the server set up, and the client set up. I feel like I half dropped the ball on this post for not having the lab set up already, but I think I got enough of the general overview across that this should help anyone interested in trying this out at least get started. We’re all still a little under the weather in the house, so a proper lab will have to wait, but I promise it will be done for this series before I am through.

Thanks for reading!

SSH Start to Finish Architecture – TCP Forwarding of STDIN/STDOUT

Last week we covered the standard TCP forwarding flags: -L, -R, and -D.

I explicitly held off on showing -W and -w, because I felt they deserved their own coverage.

We’ll cover the -w (dealing with tunnel devices) next week. This week is all about -W. Before we continue, you should note that one of my Wednesday posts was a brief overview of the netcat family of tools. If you think of “-W :” as a built in netcat for ssh, you’ll grasp the power of this flag pretty quickly. The most common example given for using this flag is the use of the ProxyCommand option to call SSH against a jumphost, and passing this flag to that host. The old way of utilizing ProxyCommand in this fashion was to actually call netcat as the remote proxy command, like so:

ssh -o ProxyCommand=’ssh jumphost netcat target_server 22′ target_server

Or if “Server_B” is our jumphost and we want to actually end up on “Server_A” by utilizing “Server_B” as a proxy, we would do it like this:
ssh -o ProxyCommand=’ssh Server_B netcat Server_A 22′ Server_A

That was all well and good, if netcat was actually installed, but sometimes it wasn’t. Luckily, SSH has its own built in netcat of sorts, via the -W flag. To do the same thing as above, we would do this, instead:
ssh -o ProxyCommand=”ssh -W %h:%p Server_B” Server_A

This is shorter, cleaner, and doesn’t rely on netcat at the jump server proxy point. Now let’s take this a step further. Let’s assume you want to pull a page down using command line, and you want to proxy through Server_B to get that page from Server_A.

echo “GET /some/page.html” | ssh -W Server_A:80 Server_B >page.html

Why might you want to do this? If you don’t have a direct route to Server_A on that port, but you can get to Server_B via SSH, and IT has access to Server_A on that port, you have a quick and dirty makeshift proxy.

Let’s sideline this thought process for a moment, and think about how many people try to lock down local accounts (stored in /etc/passwd.) Many people will give an account the “/bin/false” process as the “shell” to prevent logins to a shell. However, if you never actually attempt to get a shell with that account, you can use that account to proxy around all day long with the default sshd_config settings. If you want to disable this functionality, (and you should consider it, for the reasons demonstrated above,) you should set AllowTCPForwarding to “no” in the sshd_config file, and restart sshd.

There is a conversation floating around on social media right now about the Internet of Things botnet that took down Krebs’ blog, recently. One of the security vendors has suggested that an attack they are calling “SSHowDowN,” and this kind of functionality is exactly what they are referring to.

This kind of thing is really cool, when used correctly, but it’s also dangerous for poking holes in router and firewall access control lists.

If you have any experiences with this you’d like to share, leave a comment!

SSH – Start to Finish Architecture – TCP Tunneling

One of the great benefits of SSH is the ability to create port forwarding redirects for temporary access to a system that otherwise might not be able to access a machine. This is really easy to set up, and works quite well for many scenarios, but the downside to this is… it’s really easy to set up and works quite well for many scenarios you might not WANT it to work for. This can be used to punch holes in firewall rules, and that can get you in hot water with the wrong people, so make sure you understand your corporate policy before attempting to use anything like this at work.

SSH provides several kinds of TCP port forwarding, so we’ll cover them one at a time.
We will NOT be covering the tunneling of “TUN” devices via the -w flag this time. That will be covered at a later date.
We will also NOT be covering standard input/output forwarding via the -W flag this time. This is only going to cover TCP forwarding.

The first type of TCP forwarding we will look at is the Dynamic Port Forwarding (using SOCKS4/SOCKS5 protocol.) The flag for this is -D and takes an optional “bind_address” and a required “port” be passed to it.
-D [bind_address:]port

This creates a listening socket locally on the provided port and optionally the given bind_address that will act as a SOCKS4 / SOCKS5 application-level proxy. It’s best to set this up to bind to localhost, and then use one of the other tunneling options to point to this in order to protect the proxy from users you don’t want using it. Of course, anyone that can access the machine via ssh, will be able to use this, so be selective about where you set this up. Any application that is SOCKS aware may take advantage of this, as long as it can hit the listening port.

The next two options are really the same thing, but one sets up a listening port on the local (client) machine, and the other sets up a listening port on the remote (server) machine. The flags are -L and -R, respectively.

-L [bind_address:]port:host:hostport
-R [bind_address:]port:host:hostport

When you want to forward a local (client) listening port to the remote machine, use “L” for “local.”
When you want to forward a remote (server) listening port to the client machine, use “R” for “remote.”

In other words, that “listening port” is bound either locally, or remotely, and is one end of the tunnel. The “host” and “hostport” are not necessarily the same as the destination server you are connecting to with SSH. They just have to be a host and port that the remote system can actually get to.

“-L 8080:example.com:80” would bind a local port 8080 to the client when it establishes the connection to the remote ssh server. Then the remote ssh server would tie a connection to port 80 at example.com on its end. From there, you should be able to connect to port 8080 locally on your client machine, and talk to that remote example.com server on its port 80, all tunneled via ssh to the ssh server.

“-R 2222:localhost:22” would create a listening “localhost” bind on port 2222 on the remote server, that points to port 22 on the local client. If you were to connect outside of your network with this, it would allow people outside of the network to ssh in from that remote machine, assuming they have the correct credentials.

All of these options as given create a shell on login. You can establish JUST the tunnel by using the “-N” “-T” and “-f” options.

“-N” says to not call a remote command, just establish the tunnels.
“-T” says not to establish a pseudo TTY (PTY) terminal, which is appropriate since you aren’t passing any commands.
“-f” says to background ssh, since you probably want to do more work after the session is established.

ssh -fNT -L localhost:8080:google.com:80 User_B@jumphost

This says to background ssh after establishing a tunnel, and don’t allocate a PTY. Establish the tunnel with a locally bound port 8080, and have the remote ssh server “jumphost” establish a tunnel to “google.com” on port 80. Connect as User_B to the ssh server.

You can check with “netstat -an | grep LISTEN” to see if you have a listening port 8080 after establishing this. You can test the tunnel with netcat or telnet to localhost on port 8080 and enter “GET /” then hit enter. You might need to hit enter twice, depending. You’ll have to replace “User_B” and “jumphost” with an actual system you have access to, that also has access to the internet, of course.

I hope this didn’t muddy the waters more than it cleared them up a bit. If so, let me know in the comments, and I’ll try to clarify later.

SSH – Start to Finish Architecture – The Connection Flow

Before we get into any more advanced stuff with configuring SSH, I thought we should take a look at what actually happens when a client connects to an OpenSSH server, and what the decision tree is for granting or not granting access.

From the sshd man pages:
When a user successfully logs in, sshd does the following:
1. If the login is on a tty, and no command has been specified, prints last login time and /etc/motd (unless prevented in the configuration file or by ~/.hushlogin; see the FILES section).
~/.hushlogin
This file is used to suppress printing the last login time and /etc/motd, if PrintLastLog and PrintMotd, respectively, are enabled. It does not suppress printing of the banner specified by Banner.
2. If the login is on a tty, records login time.
3. Checks /etc/nologin; if it exists, prints contents and quits (unless root).
4. Changes to run with normal user privileges.
5. Sets up basic environment.
6. Reads the file ~/.ssh/environment, if it exists, and users are allowed to change their environment. See the PermitUserEnvironment option in sshd_config(5).
~/.ssh/environment
This file is read into the environment at login (if it exists). It can only contain empty lines, comment lines (that start with ‘#’), and assignment lines of the form name=value. The file should be writable only bythe user; it need not be readable by anyone else. Environment processing is disabled by default and is controlled via the PermitUserEnvironment option.
7. Changes to user’s home directory.
8. If ~/.ssh/rc exists, runs it; else if /etc/ssh/sshrc exists, runs it; otherwise runs xauth. The “rc” files are given the X11 authentication protocol and cookie in standard input. See SSHRC, below.
~/.ssh/rc
Contains initialization routines to be run before the user’s home directory becomes accessible. This file should be writable only by the user, and need not be readable by anyone else.
9. Runs user’s shell or command.

So from the above, we can see a few more ways to control our client connection and what it outputs when we connect. We looked at “LogLevel QUIET” in the ~/.ssh/config file last week, but we can also take advantage of the “.hushlogin” file to suppress some information.

We also see that the login gets logged only if there is a TTY associated. This is important to remember for forensics purposes.

We can temporarily disable logging into a system with SSH (other than root) by creating an /etc/nologin file, and the contents of that file will be displayed when the connection attempt gets rejected. It’s dangerous to use this if you don’t have console access, so be careful with it.

The service drops privileges and sets up a basic environment, then adjusts it from the ~/.ssh/environment file if it exists, and users are allowed to change their environment. The default behavior is to disallow this, but it’s something to check when locking down your systems. Finally, it changes to the user’s home directory to finish the environment preparations.

Next it reads and automatically runs the ~/.ssh/rc file if it exists. This is also important to know for forensics and for locking down your system. This is an excellent spot to drop a persistent misbehaving script, so it’s worth looking for and reviewing.

Finally, it runs the shell or whatever command was requested. Seems pretty simple, right? Well, the man pages stop short at that point.

So from a defense perspective, we want to review more than just the ~/.ssh/{config,authorized_keys,authorized_keys2,known_hosts} files. We also want to look at any rc and environment files in that directory. This is especially true of the root user.

Next week we’ll look at the sshd_config file and cover how to check the running configuration against the written config file.

SSH – Start to Finish Architecture – Client Config Pt. 1

This week, we’re going to focus on the client side configuration file(s) and how to use them to make ssh a more enjoyable experience. There are a lot of options in the configuration files, so we’re going to break this up into different parts. We’ll cover some common settings people might want to play with for making session handling more efficient, and then after we’ve covered some more advanced topics over the next few weeks, we’ll do a part two to go over some of the more advanced configuration options that mesh with those previously covered topics. This means part two won’t be next week.

There are two default files that handle client side ssh configuration in OpenSSH. The global file is usually found at /etc/ssh/ssh_config, and contains all of the default settings for all users on the system. The user can also write configuration options that override the global config file by including the file “config” inside that user’s .ssh directory. When the ssh client is called, the order of priority when parsing options is covered thusly:

1. command-line options
2. user’s configuration file (~/.ssh/config)
3. system-wide configuration file (/etc/ssh/ssh_config)

This whatever is given on the command line trumps everything else. The user’s config trumps the system-wide config. The system-wide config covers everything not explicitly overridden by the other two. This makes sense, but it’s important to know what the priority of parsing is when troubleshooting issues.

The focus for today will cover the ~/.ssh/config file that should contain customized settings.

One of the first options I set in my own config is the “LogLevel” setting. I prefer to suppress extraneous information (such as banners,) because I like to batch process reports from multiple servers, and I only want the information from the commands I run, not anything else. To suppress these messages, I use:

LogLevel QUIET

You can tune this to whatever level of noise you prefer, all the way up to DEBUG3, which is equivalent to “ssh -vvv” for “verbose debugging.” I don’t recommend using the user config for this, though. Just use the flags to override, and use the config to suppress as above.

The next set of options people often use are the “Match,” “User,” “Host,” and “HostName” settings.
The “User” setting is used to set what the target username to login as should be. This is like going from User_A@Workstation_A to User_B@Server_B scenario we set up before. In this case, setting “User User_B” would mean you could type “ssh Server_B” and it would know to use “User_B” as the target, instead of the default of “User_A.”

The “Host” option sets a block for options that apply to a given hostname until the next “Host” or “Match” block. The name can be a pattern match with wild cards, and can include negation via a preceding “!” if needed. “Host !*.web.com” would say “the following settings up to and not including the next Match or Host block apply to all systems that are NOT like *.web.com.” The previous example is a bad one, but it gets the point across. To set global stuff at the bottom, use a “Host *” option to match everything, and the earlier Match/Host blocks will override as needed, since they have already set values, if something matches them.

The “Match” option is used to fine tune matching against various objects. It can match a username with “Match user” or an address with “Match address” and so on. You can do a “Match host,” but just using “Host” as a block option is probably better for almost all cases.

The “HostName” option lets you set aliases within your config. This would generally be set as an option after a “Host” match. You can use “%h” to indicate the hostname that was passed to ssh on the command line before altering it with the “HostName” so that “HostName” prepends or appends to it as needed. You can also set to this an address to override whatever DNS would have returned, for example.

Using what we’ve learned so far, let’s assume that User_A wants to log in as User_B in almost all cases. However, User_A is an AIX systems administrator, and thus might need to log into a VIO server for virtual IO setup, which means the target user is most likely “padmin” rather than User_A or User_B. Also, AIX is managed by a Hardware Management Console (HMC) which often uses the “hscroot” user, rather than local accounts. Of course, every shop is different, but this is a standard practice, so we’ll go with it. The organization uses a standard naming scheme that all VIO servers have a name that starts with “vio” followed by some unique identifier. The HMCs are similarly named as “hmc” followed by some unique identifier. The VIO servers are also in a different domain than the standard, called ‘internal.net.’ We also know that if the user logs in as root, a key will never be used, so we want to skip the key exchange and force a password for root, instead. Armed with this knowledge, we can make User_A’s life easier if we build out a config that looks like this:

LogLevel QUIET
Host vio*
   HostName “%h.internal.net”
   User padmin
Host hmc*
   User hscroot
Match user root
   PubkeyAuthentication no
   PasswordAuthentication yes
   PreferredAuthentications password
Match user User_B
   PreferredAuthentications publickey,password
Host *
   User User_B

The options we set for the “Match user root” block should be fairly self explanatory at this point. There are many more options than this, and we will cover them eventually, but this is decent general overview of how to build out your config over time. Remember to put your global stuff at the bottom, since it reads top down with first match winning the option to set.

SSH – Start to Finish Architecture – Securing The Private Key

Our previous post showed how to generate the bare bones public/private key pair without using a passphrase. This is sometimes the desired configuration, but it is better to lock down the private key using a passphrase. When you generate the key pair, you can add a passphrase at the prompt that we just hit “enter” on last week, but you might want to change an existing passphrase or add a passphrase to a key that doesn’t already have one. The means for doing this is shown below:
ssh-keygen -f ~/.ssh/id_rsa -p

If the existing phrase is empty (like the one we generated last week,) this will prompt you for your new passphrase right away. If there is an existing passphrase, it will first prompt for that before prompting for the new one. Setting a passphrase on a private key is an important step to securing that key. If someone unauthorized to use that key managed to get a copy of it somehow, they won’t be able to use the key until they figure out the passphrase for it. While it is possible to brute force crack a key, if you use a decently long phrase that isn’t something commonly spoken or written, the chances of cracking it go down. Also note that SSH key passphrases allow for spaces, so you can literally write nonsense sentences, spaces and all. There is more that can be done to reduce the risk of someone using a stolen private key to do harm, but it’s on the client side, and there are caveats. We’ll cover that next week.

Now that we have a passphrase protecting our private key, what has changed in how ssh works? For starters, if you don’t use an Agent to load your keys to, every time you go to log into a server using this key, you will be prompted for the passphrase like you used to be prompted for a password. This makes convenience worse, not better. To use the agent, run ssh-add. If you’re using a standard key name such as id_rsa, id_dsa, or id_ecdsa, it will automatically find and load that key for you. For each key with a standard name it finds, it will prompt for the passphrase. You give it the phrase and it handles the rest. It acts on your behalf from then until it is told to unload a key or is stopped. When you go to login, the SSH client will see that the agent is running, and when prompted for the key by the server, it will pass that request through to the agent, which will provide proof that it knows the key, and thus you won’t be prompted. It’s like promptless SSH, but requires the extra step of loading the agent first.

If you get an error message when you run ssh-add, there is a chance that ssh-agent isn’t already running. If that is the case, you can start ssh-agent first, take the output it gives, and export those variables. For example:

ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-w8iG9Aq6KWLR/agent.1070; export SSH_AUTH_SOCK;
SSH_AGENT_PID=1071; export SSH_AGENT_PID;

If you have a key with a custom name such as id_rsa_2016, you can load these by passing their name, like so:
ssh-add /home/User_A/.ssh/id_rsa_2016

Using the agent is dangerous in a shared environment where other people have elevated privileges. Anyone with root can potentially pull the private key from memory while the agent is running on your behalf. You can unload the keys before locking your workstation if you’re paranoid enough, by using -D or -d as below:
ssh-add -D #Delete all identities
ssh-add -d /home/User_A/.ssh/id_rsa_2016 #Deletes just the id_rsa key from the agent list

You can also lock and unlock your agent using the “-x” and “-X” flags respectively, if you don’t want to completely unload for security’s sake. These will prompt you for a password to use for locking and unlocking the agent, if you choose to use them.

If you want to see which keys are loaded, you can list them with “ssh-add -l.” And if you need to be sure which public key matches the loaded private key, you can use “ssh-add -L.”

Finally, if you want to set a time limit on a key being loaded, you can use the -t flag to make it temporary. It requires a number (in seconds) for how long the key should remain loaded by the agent.

The rest of the flags are for more advanced stuff I will be covering separately, so that’s all we’ll cover for today. If you’ve kept up thus far, you’re pretty much at the level of the average SSH user at this point. (And there’s so much more to be covered.) Next week, we’ll go over some client configuration options to make session management easier.

SSH – Start to Finish Architecture – Our First Keys

Last week, we set the stage for the bare bones SSH installation and configuration for logging in with a password. This works for many people, but it isn’t very flexible. What if the user needs to run a workload remotely from a script? The script will get prompted for a password, and that isn’t handled gracefully without some helper such as an Expect wrapper. A perl call to Expect.pm, or a raw TCL based expect, or one of the myriad of other languages with an Expect module would add bloat to what would potentially be a lean and mean shell script, otherwise. Also, passwords are often cracked somewhat easily if the target system (Server_B) is configured poorly for handling repeated password attempts. Brute force attempts happen all the time on internet facing machines, even. So how do we get rid of the need for a password? We will use a public/private key pair to handle the authentication for us.

Remember that the best place to generate your public/private key pair is as the user on the machine you are coming FROM. In our example scenario, that would be User_A on Workstation_A. The key generation can be complex, but for now, we will just generate a key without a passphrase. Next week, we will cover using a passphrase on the key, why it is important to do this, and how to handle being able to still use scripts without getting prompted.

To generate our initial set of keys, we’ll use the ssh-keygen utility supplied by OpenSSH.

ssh-keygen -t rsa -b 4096 -C “User_A@Workstation_A initial key”

This should prompt for a path and name (usually the home_directory/.ssh/id_rsa) where the key will be generated.
It should also prompt for a passphrase, but we’re making that blank for now, so just hit “Enter” to move along.
Finally, it’ll do some calculations, print out a “bubble babble” output, and end. Once it’s done, you should see two new keys in the directory it gave. Do an “ls -l ~/.ssh” and see if you have an “id_rsa” and “id_rsa.pub” file, now (assuming you didn’t change the path or name when prompted.)

Okay, a few things are going on with this command. First, the flag “-t rsa” is telling the command to generate the keys using the RSA encryption algorithm. The “-b 4096” says to make that a 4096 bit RSA algorithm. The “-C” flag puts a comment in the file, which helps identify the purpose of the key and who owns it. This comment appears in the public key out of the pair. If you cat out that file, you should see the comment tag at the end of the key.

Once you have your key pair, understand that the “id_rsa” file is your private key. This should be locked down (usually with 600 permissions) and should not be copied to any other systems (barring some very rare exceptions.) The public key is “id_rsa.pub” and is what you need to push out to User_B at Server_B. The contents of this public key file actually go into a different file for that user. They should be placed into the ~User_B/.ssh/authorized_keys file. There are multiple ways to get this pushed out. Many people prefer to use the “ssh-copy-id” command to push this out, but I don’t like how it handles the authorized_keys file normally. It just appends to it blindly. I prefer to write a function that handles pushing the contents to the remote server along with catting out the remote server’s authorized_keys file, uniq the output, then write it back to the authorized_keys file. This procedure ensures only one copy of the public key ever gets pushed, no matter how many times you do this. The function details are in my “DSH – Distributed/Dancer’s Shell” product. If you don’t see a link for that yet, it’s because the product is in Alpha testing, and will be launched as a Beta soon. I’ll post a link when it goes live, for those who are interested.

Once the public key is on the remote server, make sure the permissions on the authorized_keys file are also 600. The .ssh directory should be 700. The home directory shouldn’t be readable by world. Then test your connection from User_A@Workstation_A to User_B@Server_B. It should “just work” and drop you at a shell prompt. If it doesn’t, you can check the logs on the remote server (usually /var/log/secure.log or /var/log/sshd.log or similar) for clues, and you can re-try the connection with “ssh -vvv” to get verbose output for other clues. Often it’s a permissions error, so double check those file permissions.

This will now allow User_A on Workstation_A to write scripts that can call “ssh User_B@Server_B ” for remote workloads to be run, without getting prompted for a password.

This isn’t really the best way to handle this, though, so we’ll look at how to secure the private key with a passphrase next week, and still be able to do this without getting prompted on every connection.

Thanks for sticking with me on this. There’s plenty more to come!