Fun-Day Friday – Book Review – Networking for Systems Administrators chapter 5

Last week, we actually did Chapters 3 and 4 together, but I only titled it Chapter 3.  Chapter 3 was the “IPv4” portion, and chapter 4 was the “IPv6” portion.  Today, we go over Chapter 5, for TCP/IP layer 4 Transport.

The Transport layer covers more than just TCP.  It includes TCP, UDP, ICMP, as well as some less commonly known protocols such as SCTP (stream control transmission protocol,) ESP (encapsulating security payload,) and AH (IPSec Authentication Header.)  Lucas mentions that there are dozens more, but these were the opening examples.

He goes into more depth about ICMP, TCP, and UDP, including roles, protocols, and troubleshooting.  After this, we get a bit of information on sockets, and how to configure the common services.

Finally, he covers the TCP connection state and the three way handshake, of SYN, SYN-ACK, and ACK, which is necessary for an ESTABLISHED session.  A little about the protocols file for defined numbers, and he wraps up the chapter.

Next week, we’ll look at Chapter 6, which is “viewing the network.”

AIX notifications

In line with my lack of a “proper” Monday post, I’m skipping the “proper” Wednesday post to bring you an AIX specific tool I was asked to research for an internal project this week at work.

Many of you are likely familiar with Linux’s “inotify” process, which can be used to trigger a response to a file change in real time.  This is great for filesystem events.  You might even be able to crowbar this into notifying on process death events by monitoring something out of /proc, if you’re brave enough to go poking around in there.

On AIX, there is no native “inotify.”  There is, however, a special file system called the “Autonomic Health Advisor File System (AHAFS.)”  This is available by default, and is easy to set up.

Our particular scenario was to monitor when a specific process dies.  We want a real time notification, and log of the event.  I spent a few hours reading up on how this works, then played with it until I was sure I understood the best way to use it, and this is what I came up with.

There is a sample perl script “” in the /usr/samples/ahafs/bin directory.  This can be used out of the box for most things.  In order to monitor a process, we need to know the path it is in.  In my example for this blog post, (not the actual service I needed to monitor,) we’ll say we want to monitor the apache httpd service.  It will be located at /usr/local/apache/bin/httpd for this example.

In order to monitor the death of this process, we would need to mimic that filesystem structure underneath the appropriate “Event Factory” directory.  In this case, that would be /aha/cpu/processMon.monFactory/ which means our final structure would be this:


Before we just go making the directory, though, we need to mount the ahafs filesystem to “/aha.”  This is as simple as:

sudo mkdir /aha
sudo mount -v ahafs /aha /aha

Once that’s done, we can build out the structure.  The “cpu/processMon.monFactory” was created by mounting, but we need the subdirectory structure.

sudo mkdir -p /aha/cpu/processMon.monFactory/usr/local/apache/bin

Now we can monitor at any time.  The monitor itself will sit in “wait” status until the process being monitored dies, then it will print some event information.  This means that we key off of this monitor process coming to completion before we do any notifications of our own.  My recommendation is to tee the output of the event notification with “-a” to append, so that you have a log you can review, then have it email your support email account.

/usr/samples/ahafs/bin/ -m /aha/cpu/processMon.monFactory/usr/local/apache/bin/apache.mon "CHANGED=YES;INFO_LVL=3" 2>&1 | tee -a /var/log/apache.monitor.log | mailx -s "Apache died on $( hostname )"

This should probably be placed within a script, and that script should be called after starting apache.  Of course, this is probably a terrible process to monitor, since httpd spawns off forked child processes all the time to handle requests, but it was an easy one to use as an example to get the point across on how to use this.

Also, it should be noted that for this specific scenario (PID dies, we want notification when it does,) if the process in question is one that doesn’t daemonize on its own, you can probably get away with just doing this:

echo "/usr/local/bin/my_process" | at now

When the process dies, the at job “dies” and you get an email notification from the at job termination itself.  You don’t get a local log appended, and you are relying solely on email for the notification, but it’s less work overall.

General Rant – Corporate Policies

We interrupt our regularly scheduled post on Sudo to bring you a General Rant rant.

Last week, I was oncall. We also received new laptops for our team. The current corporate policy states that we cannot have two laptops at the same time. The problem is that most members on our team actually run some flavor of Linux, which means we manage our own OS installation. I backed up all of my stuff on Friday when I came off of oncall, downloaded a new OS ISO to “burn” to a USB thumb drive, and dropped off the old laptop. I picked up my new one before leaving for the day. The new one doesn’t have a CD or DVD drive of any sort. This is why I had to use a USB drive.

Today (Saturday as I write this,) I began the process of installing the new OS. I interrupted the boot, went into the UEFI/BIOS to set the boot order, and… nothing. The USB drive doesn’t show as a boot option. It’s in a list at the bottom, but it’s not selectable as a boot device. I was going to wait until Monday to borrow the external DVD drive there, but I was informed that we bought one of those for my daughter at some point.  I borrowed hers, downloaded the ISO (again) and burned it to disk.

The drive we have here at the house?  Same problem.  It “shows” but doesn’t show as a bootable media option as it should.  It could be driver related, but it shouldn’t be.  I’ll try again on Monday with the one we have at work, but I may have to give this back to Corporate as “faulty” if the one we have there doesn’t work.  It’s also possible that they applied some kind of corporate policy to lock that out, and I don’t know the hoodoo voodoo to make it work from that perspective.

In the mean time, I’ve got plenty to work on for my home business(es) and will focus on those since this attempt was a bust.

It would have made more sense if they had let us have both laptops (new and old) at the same time for say… a week.  Due to the nature of my job, I can’t leave one sitting in their little build space while I remote in to set things up, since that would give them physical access.  Then again, “I got trust issues.”

Next rant…

I would build this with OpenBSD, but there are some software restrictions.  First, our Corporate VPN solution requires a specific agent that isn’t available on OpenBSD.  There is a third party client port available in the ports and packages, but the VPN service itself is configured to reject that.  There’s also a chance that the consoles we have that require Java won’t work with the Java clients available on OpenBSD.  I know that the OpenJDK has had issues in the past for some of them, so we download Oracle’s version, and I’m not aware of an Oracle provided OpenBSD build of Java.  Then again, I haven’t looked in a while.  I could be wrong, and one might be available.  I would have a higher peace of mind if I could run OpenBSD, rather than some flavor of Linux, but Linux is better than Windows any day of the week.

General Rant ending transmission.

Fun-Day Friday – Book Review – Networking for Systems Administrators chapter 3

This chapter is all about “Layer 3” (or the “Network Layer.”) Where Ethernet requires a MAC address to know where to send frames, Layer 3 in TCP/IP systems is the “IP address” layer. Addressing is either going to be IP version 4, or IP version 6. This is either a 32bit or 128bit number. Lucas mentions the Dynamic Host Configuration Protocol (DHCP) often used by workstations, versus statically setting an IP as you would do on a system that is designed to be a server.

He explains the notations used to represent the addresses. The most common notation for IPv4 is the dotted quad, which is four decimal numbers from 0 to 255, separated by a “dot.” He goes into how applying a “subnet mask” allows us to cut a network into smaller “subnet” networks. All addresses on the same subnet are able to talk to each other directly. Any address that wants to talk to an address outside of its subnet MUST go through a router first.

He covers the Classless Inter-Domain Routing notation, which applies the number of bits in the subnet mask to the end of the address to represent the network, rather than giving the IP and subnet addresses separately. There’s a handy chart for this on page 47.

There is a brief discussion about multi-homing, the loopback, and private networks with network address translations (NAT.) He covers a few tools to inspect, configure, and troubleshoot IP, including traceroute, ping, ifconfig, and netstat.

Next, Lucas introduces IPv6, which are often depicted as a colon separated list of alpha-numeric characters. The address is up to eight sets of four hexadecimal numbers. This represents 16 bits per set between the colons. Because this is cumbersome, shorthand notation may be used. Leading zeros from a hex section may be removed from the notation. This means a section of all zeros may be empty, so you get two colons back to back. Multiple sets of zeros may be “squeezed out” to just a single “double colon” set. Because of this special shorthand notation, only do a double colon ONCE per IP address.

IPv6 has an “autoconfiguration” facility, which allows clients to learn their IP address, as well as the router’s IP address. This facility works for a /64 network, and the protocol is “router discovery.” It doesn’t allow for assigning DNS servers, so DHCPv6 may still be necessary.

The “localhost” address in IPv6 has a special notation. The “::1” shorthand represents this loopback address.

Another special case in IPv6 is the “link-local” address. This is an auto-configured network address that begins with “fe8.” Each interface gets its own link-local, and they can be the same IP, so the OS attaches the interface name to the address. All IPv6 hosts on the same ethernet network can find each other through link-local communications. A link-local address usually appears with a “%” (percent sign) and the interface name or number at the end of the address.

The rest of the chapter includes the same troubleshooting, inspection, and configuration tools as the IPv4 sections, as well as some discussion on IPv6 tunneling (usually used to test IPv6 when your ISP doesn’t offer it yet,) as well as some discussion on how some operating systems decide which addressing to use by default, and when, where, and why you might choose to use IPv4 or IPv6 in your environment.

The next chapter will cover the TCP/IP “transport” layer (Layer 4.)

The Lab – Gear Check – Hak5 Bash Bunny – Serial

The last attack mode we haven’t covered is the SERIAL mode. This mode presents a serial device to the “victim” machine, which can be connected to via any standard serial console software tool. On Windows machines, PuTTY can handle serial connections. For Unix and Linux systems, a variety of options exist including the traditional “cu” and “tip” command line programs, “screen” (the tool used in Hak5 examples,) “kermit,” “minicom,” and so on. If you have a tool you’re comfortable with, you might stick a statically compiled version of it onto the drive and access it via the STORAGE mode using a HID mode control mechanism, perhaps.

There isn’t as much use for this, given the complexity it presents, except where the network is blocked by some kind of corporate software control, but serial is not. It seems more like a niche attack vector, but it’s better to have it available than not.

As you should be aware if you have experience connecting to serial consoles, a serial connection has some control setting that need to be set correctly before a serial session can be reliably established. The settings for the Bash Bunny’s serial session are as follows:

Baud Rate: 115200
Data Bits: 8
Parity Bit: No
Stop Bit: 1

This is often abbreviated as 115200/8N1 for some serial console software.

Again, it is almost always going to be easier to just use one of the network based attack modes, and ssh to the Bunny using plink (Windows) or openssh (Unix/Linux.)

Sudo Policy Fixes and Fails – The Cmnd Alias

So we’ve covered the User_Alias, Host_Alias, and Runas_Alias labels up to this point.  Today, we’re going to cover the Cmnd_Alias label.  This one takes a comma separated list of commands that the policy will allow.  It can contain defaults overrides, TAGs, command flags, and command arguments.  It can use special keywords in place of commands.  It is the most complex label in the policy, and it is critical to get this “right” if you don’t want to accidentally grant more privilege than you intended.

Most of our examples (after we go over the Defaults in detail) will be identifying problem policies, and most of those problems will relate to this label.

To define the label as an alias, we use the Cmnd_Alias entry like so:

Cmnd_Alias C_DOSTUFF = /usr/bin/stuff

This would grant policy to call the command “/usr/bin/stuff” (or just “stuff” if /usr/bin is in the PATH environment variable,) without restricting the flags or arguments that can be passed to it.  If we want to force it to never allow any arguments, we can change the policy like this:

Cmnd_Alias C_DOSTUFF = /usr/bin/stuff ""

This tells the policy to make the argument list “empty,” which means “no arguments are allowed.”   We can similarly restrict the flags that may be passed.

Cmnd_Alias C_DOSTUFF = /usr/bin/stuff [!-]* ""

This is the place where it is important to remember that “whitelisting” is the way to go.  Trying to “blacklist” often leads to heartache and pain killers.  The biggest example of a failed “blacklist” mentality is this:

Cmnd_Alias SHELLS = /bin/bash, /usr/bin/ksh, /bin/zsh
Cmnd_Alias SU = /bin/su

This is even mentioned in the man page, and yet, it is seen in the wild all too often.  This command appears to say “the user can run any command except the listed shells or su.”  However, by being able to run any command, the user is able to make a copy of any of the shells under a different name, and then use sudo to execute them.  This does not work the way people seem to think it does.  Even the man page has a “valid” entry that is dangerous.  The “valid” example it provides is this:

jill SERVERS = /usr/bin/, !SU, !SHELLS

On the surface, it looks like “jill” can execute anything within “/usr/bin” except the SU and SHELLS commands.  Since “cp” and “mv” and the like are in “/bin” but not “/usr/bin” on many systems, this looks like it might work, right?  Here’s the problem.  The “at” command often resides in “/usr/bin” so something as simple as this could be used to also bypass this problem:

echo "cp -p /bin/bash /usr/bin/myshell" | sudo at now

This is why it is always better to whitelist as much as possible, not give any kind of “blanket” policy with dumbed down “blacklist-like” restrictions.  My preference when dealing with Cmnd_Alias issues is to have the user write a script, (or write one for them) that handles the exact command that will need privileged access, then make that be owned by root and not writable by anyone.  Grant sudo policy to that script, instead.  You can control what flags are always passed, what arguments are always passed, and so on, if you do it this way.

We won’t look at the “Defaults” overrides this week, since we’re going to being covering Defaults next week, but we will look at the TAGs again.

So the defined TAGs are listed below.  The log related ones will be covered when we go over the Defaults, but I’ll list them, anyway.

PASSWD: | NOPASSWD:  -  Force a user to authenticate, or don't force authentication for this command.
EXEC: | NOEXEC:  -  Allow or attempt to prevent the command from forking a child process.
SETENV: | NOSETENV:  -  Override the value of the "setenv" Default on a per-command basis.
LOG_INPUT: | NOLOG_INPUT:  -  Override the value of the "log_input" Default on a per-command basis.
LOG_OUTPUT: | NOLOG_OUTPUT:  -  Override the value of the "log_output" Default on a per-command basis.

Remember that the “NOEXEC:” TAG only works if the binary was compiled as a shared object binary.  Statically compiled programs (such as “ed”) will not be blocked by this hooking mechanism.  This is why the special “sudoedit” command entry is recommended for any kind of file editing situation.  We covered this here.

I know this was a lot, but it’s not nearly enough.  We’ll go into more detail as we rip apart “bad sudo” policy examples later on.  Before we get to that, though, we need to look at the Defaults, and we’ll begin that next week.

Fun-Day Friday – Book Review – Networking for Systems Administrators chapter 2

Chapter 2 focuses specifically on how Ethernet works.

From defining broadcast domains to dealing with ethernet frame MAC addressing, he begins with the foundational basics.  He talks about duplex and speed for negotiating the connection, which both sides of the link need to agree upon.

He explains MTU (maximum transmission unit) and why mucking with reducing it below the default is almost certainly going to be problematic.  He also suggests ways to deal with situations where you can’t help but do so.

Lucas follows up with a quick explanation about the differences in category numbering for the wires that handle the transmissions.  The higher the number, the better, but higher costs can be prohibitive.  Work with what you can, but plan for higher when you can.

He moves into a quick explanation for using several troubleshooting tools, such as ping, arp, and “neighborhood discovery” (ND.)

Finally, he covers Virtual LANs (VLANs) to add a tag to an Ethernet Frame.  This allows traffic for multiple networks to flow over a single cable, without confusing where the packet should go.

He sprinkles in some more troubleshooting tools such as netstat, as well as tools to configure the ethernet layer (ifconfig and ethtool,) before closing the chapter out.

This chapter is important for the fundamentals of the “Data Link” layer (and to some extent the Physical Layer.)  Next week’s chapter covers “Layer 3” (Network Layer.)  This is mostly the “IP” layer in “TCP/IP” terms.

Thanks for reading!

The Lab – Gear Check – Hak5 Bash Bunny – Network

There are two different network attack modes that come with the Bash Bunny.  Only one of these should be used at a time, since they conflict.  The difference is the type of device it presents when active.

ECM_ETHERNET presents an Ethernet Control Model device, which is more useful when the target is Linux, Mac, or Android.  Probably other unix like systems, but testing will vet this out.

RNDIS_ETHERNET presents a Remote Network Driver Interface Specification device.  This is more useful on Windows, but will work on some Linux distributions.

Both of these present a gigabit false network to the “victim” machine, complete with DHCP server which serves up an IP in the to range.  The IP address of the Bunny itself will be to act as a gateway device.  Presenting this fake network is much like the connection you get from plugging a LAN Turtle into that USB port.  The LAN turtle presents a network, whether it bridges through to a plugged in RJ45 connection or not.  The Bash Bunny’s fake network is faster, however.

If you’re intended attack does not need to be connected to an actual network, the Bash Bunny will likely be a superior option.  If it needs to act as a man in the middle bridged network, the LAN Turtle is the superior option.  You can still tie networking in a host shared internet setup with the Bash Bunny, and instructions are provided to do so, but this requires a bit more work to get set up correctly.  To do this “manually” might take some time.  Of course, you could possibly combine attack modes and use a Ducky script to run some of the commands needed to set this up, as well.

You can connect to the Bash Bunny with SSH while it is in one of these two network attack modes.  The default root password is “hak5bunny” and root is able to do direct SSH login.  This is probably easier to use than the SERIAL mode we will cover next week.

Sudo Policy Fixes and Fails – The Runas Alias

Continuing our Sudo policy series, we’re going to focus on the Runas_Alias label this week.  In many regards, this looks much like the User_Alias in that it takes users and groups and the syntax is the same.

However, its purpose is much different.  Where the User_Alias identifies who CAN run this policy entry, the Runas_Alias defines which users and groups the policy may be executed under.

The typical default entry for this (much like the Host_Alias label) is “ALL” which allows the command to be executed under any user or group identified by the calling user.

Typical sudo calls look like this:

sudo my_command

This defaults to calling “my_command” as the root user.  The policy probably had “(ALL)” as the Runas_Alias label, as mentioned above.  We don’t always want to let a user run a command as root, though.  Sometimes we want to allow one user to call a command as another non privileged user, just because that process belongs to that user.  To do this, we set a Runas_Alias label that restricts the command to being executed under that user, not root, and not “ALL.”

With such a policy in place, the user has to know how to call sudo to execute under someone other than the default “root” user.  The “-u” flag allows this.

sudo -u otheruser my_command

Sometimes, we also need to call the command as a different group.  In order to do this, we can pass the “-g” flag.  A combination of both is valid.

sudo -u otheruser -g othergroup my_command

This allows for more granular control over the policies being written, and enforces safe policy that doesn’t grant access to root except when absolutely necessary.

This is the end goal of any good policy.  Only allow what is strictly necessary, and nothing more than that.

Fun-Day Friday – Book Review – Networking for Systems Administrators chapter 1

Continuing our review of “Networking for Systems Administrators” by Michael W. Lucas, we’ll roll right into Chapter 1.

This chapter focuses on thinking in layers.  There are different network layer models, including the 7 layer OSI model, but Lucas says you really only need 5 layers to represent the network.  There’s the 4 layers of the TCP/IP model, but he splits the lowest layer into the Physical and DataLink layers.  This matches the OSI model’s way of presenting those layers.

The OSI Session, Presentation, and Application layers are all lumped under the TCP/IP “Application Layer.”  Lucas calls this the “your stuff” layer, and it’s true.  The Network Admins won’t really care much beyond the Transport Layer when troubleshooting.

Speaking of troubleshooting, identifying the lowest layer that is broken is crucial for this.  Fix that layer first, and most (if not all) of the other layers will likely start working again.  And those that don’t, go with the next lowest layer, and work your way up.

The rest of the chapter covers specific troubleshooting techniques for each of the lower layers, with a promise for more in depth troubleshooting discussion later in the book.

This chapter is short, but critical for laying the ground work.  Understanding these layers is one of the most important things to know for network troubleshooting.