SSH – Start to Finish Architecture – Our First Keys

Last week, we set the stage for the bare bones SSH installation and configuration for logging in with a password. This works for many people, but it isn’t very flexible. What if the user needs to run a workload remotely from a script? The script will get prompted for a password, and that isn’t handled gracefully without some helper such as an Expect wrapper. A perl call to, or a raw TCL based expect, or one of the myriad of other languages with an Expect module would add bloat to what would potentially be a lean and mean shell script, otherwise. Also, passwords are often cracked somewhat easily if the target system (Server_B) is configured poorly for handling repeated password attempts. Brute force attempts happen all the time on internet facing machines, even. So how do we get rid of the need for a password? We will use a public/private key pair to handle the authentication for us.

Remember that the best place to generate your public/private key pair is as the user on the machine you are coming FROM. In our example scenario, that would be User_A on Workstation_A. The key generation can be complex, but for now, we will just generate a key without a passphrase. Next week, we will cover using a passphrase on the key, why it is important to do this, and how to handle being able to still use scripts without getting prompted.

To generate our initial set of keys, we’ll use the ssh-keygen utility supplied by OpenSSH.

ssh-keygen -t rsa -b 4096 -C “User_A@Workstation_A initial key”

This should prompt for a path and name (usually the home_directory/.ssh/id_rsa) where the key will be generated.
It should also prompt for a passphrase, but we’re making that blank for now, so just hit “Enter” to move along.
Finally, it’ll do some calculations, print out a “bubble babble” output, and end. Once it’s done, you should see two new keys in the directory it gave. Do an “ls -l ~/.ssh” and see if you have an “id_rsa” and “” file, now (assuming you didn’t change the path or name when prompted.)

Okay, a few things are going on with this command. First, the flag “-t rsa” is telling the command to generate the keys using the RSA encryption algorithm. The “-b 4096” says to make that a 4096 bit RSA algorithm. The “-C” flag puts a comment in the file, which helps identify the purpose of the key and who owns it. This comment appears in the public key out of the pair. If you cat out that file, you should see the comment tag at the end of the key.

Once you have your key pair, understand that the “id_rsa” file is your private key. This should be locked down (usually with 600 permissions) and should not be copied to any other systems (barring some very rare exceptions.) The public key is “” and is what you need to push out to User_B at Server_B. The contents of this public key file actually go into a different file for that user. They should be placed into the ~User_B/.ssh/authorized_keys file. There are multiple ways to get this pushed out. Many people prefer to use the “ssh-copy-id” command to push this out, but I don’t like how it handles the authorized_keys file normally. It just appends to it blindly. I prefer to write a function that handles pushing the contents to the remote server along with catting out the remote server’s authorized_keys file, uniq the output, then write it back to the authorized_keys file. This procedure ensures only one copy of the public key ever gets pushed, no matter how many times you do this. The function details are in my “DSH – Distributed/Dancer’s Shell” product. If you don’t see a link for that yet, it’s because the product is in Alpha testing, and will be launched as a Beta soon. I’ll post a link when it goes live, for those who are interested.

Once the public key is on the remote server, make sure the permissions on the authorized_keys file are also 600. The .ssh directory should be 700. The home directory shouldn’t be readable by world. Then test your connection from User_A@Workstation_A to User_B@Server_B. It should “just work” and drop you at a shell prompt. If it doesn’t, you can check the logs on the remote server (usually /var/log/secure.log or /var/log/sshd.log or similar) for clues, and you can re-try the connection with “ssh -vvv” to get verbose output for other clues. Often it’s a permissions error, so double check those file permissions.

This will now allow User_A on Workstation_A to write scripts that can call “ssh User_B@Server_B ” for remote workloads to be run, without getting prompted for a password.

This isn’t really the best way to handle this, though, so we’ll look at how to secure the private key with a passphrase next week, and still be able to do this without getting prompted on every connection.

Thanks for sticking with me on this. There’s plenty more to come!

Fun-Day Friday

Fridays will be “Fun-Day Friday,” at least for a while. I’ll talk about non-unix non-security stuff on this day, and today’s topic will be about one of my hobbies: Amateur Radio.

I’ve been a licensed HAM since November of 2008. In my initial test, I passed both my Technician and my General class exams, but failed the Extra when it was offered. I didn’t really expect to pass that one, but it gave me a pretty good idea of what kind of material was on it. I waited a few years before taking my Extra class exam again, because I wanted some time to get to know the hobby better before diving in that deep. I passed my Extra in 2012. On my to-do list is to get my Volunteer Examiner, but I keep getting life interruptions when I sit down to take it. It’s an open book, online test, so there’s no reason for me to not have it yet other than procrastination and interruptions in general. I’ll get it eventually.

In the grand scheme of things, I’ve enjoyed the hobby thus far. I don’t operate as much as other members of the local county club, and I get teased for this incessantly, but I enjoy the times I do operate, and that’s what matters.

HAM radio has a broad offering of ways to enjoy the hobby, so if you haven’t looked into it, you should. Some people enjoy rag chew on HF, others like to chase points in contests, and another set prefer to operate in only one mode or another, such as CW (morse code) or digital only modes where voice isn’t even carried.

The latest trend in our local club is DMR (digital mobile radio,) and the club just purchased a new DMR capable repeater for the locals to use. I picked up a new HT radio for the occasion, so I’ll probably cover a review on that next week. HAM radio will be one of the topics I bring up frequently on Fridays, but I’ll also cover some of my many other hobbies, such as knife throwing, playing music, book reviews, movie reviews, and so on. I eventually plan to spin these off onto their own blog space, but for now, I’ll share them here. Only on Fridays. You’ve now been forewarned.

Thanks for reading, and if you have any hobbies you love, post in the comments!

Hacker-Tool Hump-Day – USB Rubber Ducky

Every Wednesday (Hump Day) I intend to cover some topic related to hacking, penetration testing, making things do things they maybe weren’t meant to do in general, and so on. Eventually, Mondays will be more like what I envision “Blue Team” mindset should be, and Wednesdays will be more on what I envision “Red Team” mindset should look like. Since I’ve never had the opportunity to be on either kind of team at the place I’ve worked, I won’t say “this is how it is.” So if I use “Red Team/Blue Team” labels, understand this is from my perspective of how I understand them to be, from the outside looking in. I’ve mostly been in a Blue Team type of position (systems administration) but I’ve always tried to look at how to make things work in ways they maybe weren’t meant to work. This has helped me to be sure the stuff I run is run more securely. Any time I come up with a way to bypass, break, or otherwise torture a configuration, I look for ways to plug that hole.

Anyway, on to today’s topic. I’m a fan of Hak5 and the tools they create. Some people take a negative attitude toward them, but I think they’ve helped create some really nice tools for not only aggressive mentalities, but also for defensive ones.

Recently they held a sale in their Hak5 shop in celebration of one of their tools being shown on the show Mr. Robot. The tool in question is the USB Rubber Ducky. I bought one of these last year, and it has been useful from a SysAdmin perspective, so I thought I’d present it as my first topic. I bought one of their zippered kits during the sale, and it should be here by the time the next one of these posts goes live, so I might be covering other items from the kit at that point. I haven’t decided yet. The Ducky is available at the Hak5 shop, at this link:

USB Rubber Ducky Deluxe

So… the USB Rubber Ducky is a device that by default with stock firmware provided by the good folks at Hak5 simply emulates a USB keyboard device when you plug it in. It comes in a case with a swivel key chain piece, and you can take this apart to get to the Ducky internals. You need to do this, to do anything useful with it. The reason is simply this: the device has a microSD card that it reads a payload file from for the purposes of knowing what to “type.” The stock payload isn’t useful for anything more than amusement. You will need to be able to access this microSD card inside to put a useful payload on it. The default firmware doesn’t present a USB storage device for accessing the card, so you will have to take it apart to get that card out.

Take the swivel off, then use a large paperclip or a jewelers/glasses screwdriver to pop the two plastic pieces of the casing apart. There is a hole in the back opposite the USB connector. Just stick it in, give it a light twist, and it should come apart. The microSD card is sometimes stuck pretty good, so you may want a tool to catch the lip of the drive to get leverage for popping it out. It’s not glued in, I promise. The one that came with my first Ducky was 128 meg, so it’s not really useful for much more than keeping a payload on. The Ducky also came with a very small microSD card reader so that you can move your card to it and not have to buy a reader separately to get your payload onto the device. It’s formatted as vFAT, but I’m not sure if it can read the payload from an NTFS formatted card. I intend to test this.

There are alternative firmwares available. The only other one(s) I’ve tried are the Composite ones available as of the time of this writing, and upon initial testing, both seemed to behave identically as far as I could tell. The Composite firmware(s) present the Ducky as both a HID (keyboard) and a USB storage device. There used to be a version that would automatically launch the payload after a set amount of time, but the ones I tested seemed to have that functionality stripped. The version that has “4Cap” in the name lets you hit Caps Lock four times in a row to trigger the payload launch.

I tried the Composite firmware that had “Cap” in the name, expecting the payload to launch. I was disappointed. The only method that worked was actually pushing the button on the Ducky device itself to launch the payload.

I hopped onto the forums to ask about this, and after a few responses from forum members “winter_soldier” and “bored369” I was able to determine that the 4Cap hex file does indeed work as expected… on a Windows workstation. Another hearty “thanks” to winter_soldier and bored369 for the helpful responses on the forums. All of my testing had been done from Linux Mint, and only a press of the button on the Ducky would send the payload, there. I’ll research what the differences are to see if there’s a reason for this difference in behavior. The Mint laptop is a Lenovo ThinkPad W520, and the Windows laptop is a Lenovo Z50. I also noticed that if I do the caps lock key sequence to trigger this on Windows, it works the first time, but then starts flashing red on a second attempt without pulling it and plugging it back in. I can send the payload multiple times from one plug in if I use the button on the Ducky, instead. The button on the Ducky stops working if you use caps sequence to send the payload and it causes the flashing red light. I suspect this may be an error in the logic loop for handling the two different cases. Maybe the source is available for review. I’ll go looking to see if I can find it.

I also intend to try a case mod by drilling two holes into one side of the case. If I put a rod down in the hole to press the button, I won’t need to take the case apart, and if I use a small snip of strand from a novelty toy like the “glowing Christmas trees” you sometimes see on sale during the holidays, I can have a fiber to run down to the indicator light so I can see if it’s working (green) or if something went wrong (red) also without taking it apart.

So besides the actual USB Rubber Ducky, you also need to have a workstation to compile your payloads on, and update the firmware if needed. There are lots of examples of this on the Forums for Windows workstations, but I do my work from a Linux Mint workstation. I grabbed the encoder.jar from:

This encoder lets you specify a keyboard layout along with the Ducky Script file. Speaking of the Ducky Script file, you need to write your payload in Ducky Script, then use the encoder to compile it into a binary file that the Ducky reads upon start up. The specific file name should be “inject.bin” for the Ducky to read it. The version of java used to call the encoder should be 7 or higher.

An example test Ducky Script might look like this:

DELAY 1000
STRING #this is a test of my first payload

The “DELAY” is in milliseconds, so 1000 is “1 second.” You want a delay of 3 seconds or more with the stock firmware, most likely. Since I already had my Composite firmware in place, and it doesn’t launch until I push the button on the Ducky, I lowered my DELAY.

The “STRING” is the actual typed text that will get sent by the Ducky upon payload execution. Since the return key is a special key, the ENTER directive is needed to emulate that keypress. There are other meta keys that can be emulated. Review the documentation for what all are available, and visit the Forums if you’re having trouble with any of them. I seem to recall seeing a reported known bug about GUI vs. WINDOWS there a while back.

To encode a Ducky Script named “test.ducky” run the following:

java -jar encoder.jar -i ./ -o inject.bin -l us

After you have your payload, copy it to the microSD card, slap the card back into the Ducky, and the next time you plug it in, it should type whatever you told it to type in your script.

If you are bold enough to try out one of the alternative firmwares, visit the Wiki page below for instructions. I used the dfu-programmer software provided as a link from that site from my Mint workstation, and my instructions for getting it compiled will follow.

I had to install two development libraries first, since configure complained about them being missing:
sudo apt-get install libusb-dev libusb-1.0.0-dev

I still had problems compiling after this, so a little digging around on the internet using the might of my Google-FU, I came across instructions for “fixing” some broken code.

After installing the missing libraries, unpack the tarball:
tar xvfz ./dfu-programmer-0.5.4.tar.gz
cd dfu-programmer-0.5.4

Modify the file to change all references of “LDFLAGS” to “LIBS” instead:
sed -i.bak -e ‘s/LDFLAGS/LIBS/g’ ./

Run configure:

Before running make, modify the Makefile in the src directory to move the -lusb flags from LDFLAGS down to LIBS instead. If this sed statement is confusing, ask in the comments and I’ll explain all the pieces, but today’s topic is the Ducky, not “sed.”

sed -i.bak -ne ‘/^LDFLAGS =/{h;s/LDFLAGS = \(.*\)/LIBS = \1/;x;s/LDFLAGS =.*/LDFLAGS =/};/^LIBS = .*/{x;};p’

Now that you’ve “fixed” the problem children, run “make” per normal, and you should have a file “dfu-programmer” in your src directory.

You can grab one of the .hex firmware files from the github wiki links above, and use the following steps to update it.

Take the Ducky out of its case if you haven’t already (or if you haven’t modified the case as I described earlier.)
In order to put the Ducky into “DFU” mode, hold the button down, then insert it into the USB port. If you aren’t holding it down while you insert it, it’ll be in payload mode and will try to type the payload script. Wait a few moments for the Ducky to be recognized by the system, then use the dfu-programmer software to update the firmware as below:

This step isn’t necessary, but the wiki mentions it, so if you want to make a backup of the existing firmware, do this:
sudo ./dfu-programmer at32uc3b1256 dump >dump.bin

Before flashing a new firmware hex file to the Ducky, you need to erase it:
sudo ./dfu-programmer at32uc3b1256 erase

To drop your new firmware in place, use the “flash” command as shown below. I tried both versions of the Composite firmware, so both of these are shown, but you only run it against ONE .hex file. Pick your firmware and flash only that.
sudo ./dfu-programmer at32uc3b1256 flash –suppress-bootloader-mem Composite_Duck_S003.hex
sudo ./dfu-programmer at32uc3b1256 flash –suppress-bootloader-mem Composite_Duck_4cap.hex

Finally, you have to remember to reset the Ducky after a firmware update in order to make it run payloads again.
sudo ./dfu-programmer at32uc3b1256 reset

That’s it. After the firmware is updated, you should be able to remove the Ducky and reinsert it to test functionality. Remember that if you used a firmware like the Composite versions above that requires a push of the Ducky button, you won’t see the payload until you push the button. I recommend using a test payload that doesn’t press a bunch of meta keys, first. Open a text editor, then insert the Ducky. Press the button if needed, but watch that the payload actually gets sent to the text document to verify it is functioning properly.

Hak5 touts this as a device for social engineering by virtue of curiosity. Put your malicious payload on the Ducky, then drop it in a parking lot and wait. Your payload might call up a bunch of commands to offload a Windows password dump to a remote web server, for example. So how does this tool have any kind of legitimate use in a SysAdmin tool kit?

The answer is simple. There are two scenarios where I have personally already found this tool useful from an everyday SysAdmin perspective. The first is when dealing with a console session to a remote server, and the console does not allow copy/paste. When you are dealing with a blind console that doesn’t allow pasting commands into the terminal, you have to be meticulous in your typing. If you know all of the steps you need to get the task accomplished, you can carefully craft that out into a Ducky Script, compile it, and then make that console your focused window before plugging the Ducky in for your “I’m pasting my commands anyway, so take that!” workload.

The other scenario has to do with a long ssh private key passphrase that needed to be stored into an IAM solution, and people were present who should not see this passphrase. Instead of having to start over multiple times, because the laws of nature demand that I mistype the phrase at least four times before getting it right, I was able to put the passphrase into a Ducky Script, take the Ducky to the conference room where the IAM software was being configured, set the focus on the passphrase field in the software, then plug the Ducky in. Or at least, I would have if I’d had the Ducky with me that day. Instead I had to type it out. Multiple times. Because things. Since that was the “development” environment, I’ll be doing the “production” environment with the Ducky next time.

Thanks for reading this, and if you have any ideas on other legitimate uses of the USB Rubber Ducky as an every day SysAdmin tool, share in the comments!

SSH – Start to Finish Architecture – The Basic Setup

Once upon a long time ago in a place not too far away, the internet was born. In its early days, people used basic protocols to interact with each other and share information. One batch of these protocols allowed people to connect remotely to another machine and either login for an interactive shell, or pass remote commands to be run for processing. These were telnet, rsh, and rlogin. There were also rcp and ftp for file transfers. All of these were great until people started needing to lock their doors at night because the neighbors got too nosey. Security became a concern, and SSH was born.

Today, most people think of OpenSSH as the defacto software for SSH services. It contains both a server component (sshd), an sftp subsystem (to replace FTP) and a few clients for handling remote logins and file transfers (ssh and scp.) The truth is, there are plenty of packages to choose from, for both server side and client side communication with these new protocols. I may cover some of these, but this series is going to start with OpenSSH as a focus simply because it is the most popular at this time. We’ll start with a basic setup, and then progress through more complex setups to cover various needs based on scale of the operation needing this kind of access arrangement.

An initial explanation of how this works is a user on workstation A wants to be able to log in remotely to server B to run some kind of work load. Remember that every SSH session is composed of pieces of information at its most basic level. A user (User_A) on a workstation or server (Workstation_A) wants to connect remotely to a server (Server_B) as some user (User_B) to run a work load. We will use these labels moving forward to describe how to configure the software for each kind of scenario we want to cover. Eventually there will be a third system involved (either LDAP, or a Certificate Authority, or both) so we’ll toss in an _C label when we get to that. For now, just remember “_A” will represent “coming from” and “_B” will represent “going to” for directionality of each session flow.

So before User_A can connect to Server_B with SSH, Server_B needs to have an SSH service running. The software can be installed from source, or it can be installed from whatever package management system your server OS supports. RPM/yum and DPKG/apt-get are common for RedHat and Debian based systems, respectively. AIX has installp/RPM. HP-UX has swinstall. Just use the appropriate software for the system you are supporting, but understand that some of the more advanced features will require newer versions of OpenSSH, so if you try something and it doesn’t work, check your server version.

Now that OpenSSH sshd has been installed via whatever means you chose, User_A needs client software. It should be the same deal, here. Use whatever package management software you choose. There is an OpenSSH client via the Cygwin project for Windows for the “follow along” when your workstation is Windows, but other clients may work better for most of these scenarios. PuTTy and its derivatives, WinSCP, MobaXTerm are all excellent Windows clients. However, when we get to certificate based connections, these may not be sufficient. Just install OpenSSH client for now on whatever platform you’re coming from, and let’s move on.

Now Workstation_A and Server_B have the software needed to make a connection. If User_A wants to connect to Server_B as User_B, a “User_B” account will need to exist on the server, or at least be a user that can authenticate (LDAP or similar.) For now we will assume local accounts are in use. So to make a connection, User_A on Workstation_A will need to know the password for User_B on Server_B. Then typing the following will allow a connection to be established (assuming the account is active and not locked.)

As User_A from a terminal prompt:
ssh User_B@Server_B

It’s that simple. This is almost not worth covering, I know. However, every post in this series will take a step toward making things work more smoothly from an operations and security perspective, so stick with me. Next week we’ll cover setting up public/private key pairs for authentication without a password.

Welcome to the Lab

Welcome to the Lab.

On this site, I will share Unix tips and tricks for writing clean bash/ksh/posix shell code, sed, awk, perl, and other command line power tools.

I’ll show code snippets and examples for dealing with specific shell coding situations, such as input validation, cleaning up your work space, setting locks on files, multiplexing tasks using tmux and/or dsh, and a plethora of other day to day Unix Sysadmin tasks.

We will also explore developing best practices for hardening systems together.  You will be able to follow my journey through setting up a home lab using a mix of virtual machines and single board computers.  I’ll share links to the hardware I’m testing on so you can follow along and try the same set up if you like.

I’ll discuss general security topics, such as BCDR, Incident Response, and Governance documentation.

I’ll also share a small slice of related hobbies such as Amateur Radio.

Comments from the community are always welcome.

Get your popcorn at the door, sit back, and enjoy the show.