Book Review – Networking for Systems Administrators chapter 6

Since we aren’t doing “Monday this” and “Friday that” for a while, I thought I should leave off the usual title prefix.  I’m also continuing the chapter by chapter review for today to ease back into the writing.  This won’t be every Monday, but I need to mix these in every now and then to keep from letting it trail off before the reviews are finished.

This chapter focuses on viewing network connections.  This is useful for troubleshooting, diagnostics, and performance data gathering.  The chapter goes into details for displaying live ports, tcp/udp/both, filtering by state such as “established connections,” identifying the ports, and identifying the programs that own those ports.  The netstat command is discussed heavily, but lsof and sockstat make an appearance, as well.

As mentioned by the author, there is no common command for displaying which programs own which ports.  The lsof command is ported to many platforms, but is not always an option.

As an example of how to deal with this in AIX, (not specifically covered by the book,) you need to do this in two commands.  First, run the netstat command with the -A flag to get the socket identifier, then pass the identifier for the specific port to the “rmsock” command using tcbcb as the last parameter.  This will show you the program that owns that socket, even though you aren’t actually removing the socket at all.

Also, on openbsd you can use the fstat command, but this was not covered by the author.

Thanks for reading, and remember to check out the social media links from this site!

Returning next week.

My break has taken a bit longer than intended. I want to apologize for that, but rest assured, I’m coming back next week. This post is to give a heads up on the re-structuring of both UnixSecLab and Jack of all Hobbies.

Until further notice, the mailing list sign ups (one per site) will only be a summary feed of the previous week’s blog posts for that site. I may bring back the extra content, eventually, and I may continue to use the mailing list for the rare but occasional promotional advertisement for products or services I offer.

Also, until further notice, the minimum number of posts per week (starting NEXT week) will be ONE post on Monday (for UnixSecLab) and ONE post on Thursday (for Jack of all Hobbies.)

I’m trying to run two different sites with two different focus topics. I work an 8am to 5pm job with a one hour commute to and one hour commute from work each day. This means I lose two hours out of my day just driving on the interstate. I also have nine beautiful children that need some of my attention, a loving wife that needs some of it, and so on. My time is more limited than I imagined when I first started these ventures, so I’m having to restructure. I may be able to get into a groove and get more content in a week eventually, but for now I’m backing things down.

In lieu of the extra blog posts, I will be doing more on social media. Posting to Twitter(UnixSecLab), Twitter(Jack of all Hobbies), FaceBook, LinkedIn, and so on takes less of my time than writing a full detailed blog post would, so I’ll be more active there.

You can expect the first content blog post for UnixSecLab under the new format starting June 5th. The first Jack of all Hobbies blog post under the new format will be available June 8th.

I also have some exciting plans for the not too distant future, so stay tuned.

I hope to catch you all here and on social media, and as always, leave comments, suggestions, or other verbiage in the comments for this post.

Weekend Wrap Up – No real technical content today

This last weekend was crazy busy.  We’ve had heavy showers and mild storms hitting our area off and on most of the week, but this weekend we got hit hard.  Several towns and communities surrounding us have been heavily flooded due to the torrential rains.  A friend of the family showed their rain gauge measured more than seven (7) inches of rain in one day.  Some of the flooding was so bad that one lane of one of the roads literally washed away.  While 7 inches isn’t anywhere near what Queensland Australia got recently (they got almost a foot and a half or so,) the power of the flow was significant, and our county has been declared a disaster area.

Saturday morning, before the heavy stuff hit, I gave a presentation on Permaculture to a local community group.  It went well, and if you are interested in following that side of my life on a regular basis, check out my non-tech hobby page: Jack of all Hobbies.

Sunday morning at midnight, I headed to Little Rock.  This was right in the middle of the flooding mess.  My car stalled several times on my way in to work.  I was at work until almost noon, and I came home and crashed (sleep.)

Due to the crazy experiences we’ve had with the weather, the work schedule, and the presentation I gave, I’m cancelling today’s normal content to just provide an update on life.  Crazy busy.

Thanks again for reading, and hopefully I can get back on track quickly.

Fun-Day Friday – Book Review – Networking for Systems Administrators chapter 5

Last week, we actually did Chapters 3 and 4 together, but I only titled it Chapter 3.  Chapter 3 was the “IPv4” portion, and chapter 4 was the “IPv6” portion.  Today, we go over Chapter 5, for TCP/IP layer 4 Transport.

The Transport layer covers more than just TCP.  It includes TCP, UDP, ICMP, as well as some less commonly known protocols such as SCTP (stream control transmission protocol,) ESP (encapsulating security payload,) and AH (IPSec Authentication Header.)  Lucas mentions that there are dozens more, but these were the opening examples.

He goes into more depth about ICMP, TCP, and UDP, including roles, protocols, and troubleshooting.  After this, we get a bit of information on sockets, and how to configure the common services.

Finally, he covers the TCP connection state and the three way handshake, of SYN, SYN-ACK, and ACK, which is necessary for an ESTABLISHED session.  A little about the protocols file for defined numbers, and he wraps up the chapter.

Next week, we’ll look at Chapter 6, which is “viewing the network.”

AIX notifications

In line with my lack of a “proper” Monday post, I’m skipping the “proper” Wednesday post to bring you an AIX specific tool I was asked to research for an internal project this week at work.

Many of you are likely familiar with Linux’s “inotify” process, which can be used to trigger a response to a file change in real time.  This is great for filesystem events.  You might even be able to crowbar this into notifying on process death events by monitoring something out of /proc, if you’re brave enough to go poking around in there.

On AIX, there is no native “inotify.”  There is, however, a special file system called the “Autonomic Health Advisor File System (AHAFS.)”  This is available by default, and is easy to set up.

Our particular scenario was to monitor when a specific process dies.  We want a real time notification, and log of the event.  I spent a few hours reading up on how this works, then played with it until I was sure I understood the best way to use it, and this is what I came up with.

There is a sample perl script “aha.pl” in the /usr/samples/ahafs/bin directory.  This can be used out of the box for most things.  In order to monitor a process, we need to know the path it is in.  In my example for this blog post, (not the actual service I needed to monitor,) we’ll say we want to monitor the apache httpd service.  It will be located at /usr/local/apache/bin/httpd for this example.

In order to monitor the death of this process, we would need to mimic that filesystem structure underneath the appropriate “Event Factory” directory.  In this case, that would be /aha/cpu/processMon.monFactory/ which means our final structure would be this:

/aha/cpu/processMon.monFactory/usr/local/apache/bin/

Before we just go making the directory, though, we need to mount the ahafs filesystem to “/aha.”  This is as simple as:

sudo mkdir /aha
sudo mount -v ahafs /aha /aha

Once that’s done, we can build out the structure.  The “cpu/processMon.monFactory” was created by mounting, but we need the subdirectory structure.

sudo mkdir -p /aha/cpu/processMon.monFactory/usr/local/apache/bin

Now we can monitor at any time.  The monitor itself will sit in “wait” status until the process being monitored dies, then it will print some event information.  This means that we key off of this monitor process coming to completion before we do any notifications of our own.  My recommendation is to tee the output of the event notification with “-a” to append, so that you have a log you can review, then have it email your support email account.

/usr/samples/ahafs/bin/aha.pl -m /aha/cpu/processMon.monFactory/usr/local/apache/bin/apache.mon "CHANGED=YES;INFO_LVL=3" 2>&1 | tee -a /var/log/apache.monitor.log | mailx -s "Apache died on $( hostname )" support@my.example.com

This should probably be placed within a script, and that script should be called after starting apache.  Of course, this is probably a terrible process to monitor, since httpd spawns off forked child processes all the time to handle requests, but it was an easy one to use as an example to get the point across on how to use this.

Also, it should be noted that for this specific scenario (PID dies, we want notification when it does,) if the process in question is one that doesn’t daemonize on its own, you can probably get away with just doing this:

echo "/usr/local/bin/my_process" | at now

When the process dies, the at job “dies” and you get an email notification from the at job termination itself.  You don’t get a local log appended, and you are relying solely on email for the notification, but it’s less work overall.

General Rant – Corporate Policies

We interrupt our regularly scheduled post on Sudo to bring you a General Rant rant.

Last week, I was oncall. We also received new laptops for our team. The current corporate policy states that we cannot have two laptops at the same time. The problem is that most members on our team actually run some flavor of Linux, which means we manage our own OS installation. I backed up all of my stuff on Friday when I came off of oncall, downloaded a new OS ISO to “burn” to a USB thumb drive, and dropped off the old laptop. I picked up my new one before leaving for the day. The new one doesn’t have a CD or DVD drive of any sort. This is why I had to use a USB drive.

Today (Saturday as I write this,) I began the process of installing the new OS. I interrupted the boot, went into the UEFI/BIOS to set the boot order, and… nothing. The USB drive doesn’t show as a boot option. It’s in a list at the bottom, but it’s not selectable as a boot device. I was going to wait until Monday to borrow the external DVD drive there, but I was informed that we bought one of those for my daughter at some point.  I borrowed hers, downloaded the ISO (again) and burned it to disk.

The drive we have here at the house?  Same problem.  It “shows” but doesn’t show as a bootable media option as it should.  It could be driver related, but it shouldn’t be.  I’ll try again on Monday with the one we have at work, but I may have to give this back to Corporate as “faulty” if the one we have there doesn’t work.  It’s also possible that they applied some kind of corporate policy to lock that out, and I don’t know the hoodoo voodoo to make it work from that perspective.

In the mean time, I’ve got plenty to work on for my home business(es) and will focus on those since this attempt was a bust.

It would have made more sense if they had let us have both laptops (new and old) at the same time for say… a week.  Due to the nature of my job, I can’t leave one sitting in their little build space while I remote in to set things up, since that would give them physical access.  Then again, “I got trust issues.”

Next rant…

I would build this with OpenBSD, but there are some software restrictions.  First, our Corporate VPN solution requires a specific agent that isn’t available on OpenBSD.  There is a third party client port available in the ports and packages, but the VPN service itself is configured to reject that.  There’s also a chance that the consoles we have that require Java won’t work with the Java clients available on OpenBSD.  I know that the OpenJDK has had issues in the past for some of them, so we download Oracle’s version, and I’m not aware of an Oracle provided OpenBSD build of Java.  Then again, I haven’t looked in a while.  I could be wrong, and one might be available.  I would have a higher peace of mind if I could run OpenBSD, rather than some flavor of Linux, but Linux is better than Windows any day of the week.

General Rant ending transmission.

Fun-Day Friday – Book Review – Networking for Systems Administrators chapter 3

This chapter is all about “Layer 3” (or the “Network Layer.”) Where Ethernet requires a MAC address to know where to send frames, Layer 3 in TCP/IP systems is the “IP address” layer. Addressing is either going to be IP version 4, or IP version 6. This is either a 32bit or 128bit number. Lucas mentions the Dynamic Host Configuration Protocol (DHCP) often used by workstations, versus statically setting an IP as you would do on a system that is designed to be a server.

He explains the notations used to represent the addresses. The most common notation for IPv4 is the dotted quad, which is four decimal numbers from 0 to 255, separated by a “dot.” He goes into how applying a “subnet mask” allows us to cut a network into smaller “subnet” networks. All addresses on the same subnet are able to talk to each other directly. Any address that wants to talk to an address outside of its subnet MUST go through a router first.

He covers the Classless Inter-Domain Routing notation, which applies the number of bits in the subnet mask to the end of the address to represent the network, rather than giving the IP and subnet addresses separately. There’s a handy chart for this on page 47.

There is a brief discussion about multi-homing, the loopback, and private networks with network address translations (NAT.) He covers a few tools to inspect, configure, and troubleshoot IP, including traceroute, ping, ifconfig, and netstat.

Next, Lucas introduces IPv6, which are often depicted as a colon separated list of alpha-numeric characters. The address is up to eight sets of four hexadecimal numbers. This represents 16 bits per set between the colons. Because this is cumbersome, shorthand notation may be used. Leading zeros from a hex section may be removed from the notation. This means a section of all zeros may be empty, so you get two colons back to back. Multiple sets of zeros may be “squeezed out” to just a single “double colon” set. Because of this special shorthand notation, only do a double colon ONCE per IP address.

IPv6 has an “autoconfiguration” facility, which allows clients to learn their IP address, as well as the router’s IP address. This facility works for a /64 network, and the protocol is “router discovery.” It doesn’t allow for assigning DNS servers, so DHCPv6 may still be necessary.

The “localhost” address in IPv6 has a special notation. The “::1” shorthand represents this loopback address.

Another special case in IPv6 is the “link-local” address. This is an auto-configured network address that begins with “fe8.” Each interface gets its own link-local, and they can be the same IP, so the OS attaches the interface name to the address. All IPv6 hosts on the same ethernet network can find each other through link-local communications. A link-local address usually appears with a “%” (percent sign) and the interface name or number at the end of the address.

The rest of the chapter includes the same troubleshooting, inspection, and configuration tools as the IPv4 sections, as well as some discussion on IPv6 tunneling (usually used to test IPv6 when your ISP doesn’t offer it yet,) as well as some discussion on how some operating systems decide which addressing to use by default, and when, where, and why you might choose to use IPv4 or IPv6 in your environment.

The next chapter will cover the TCP/IP “transport” layer (Layer 4.)

Fun-Day Friday – Book Review – Networking for Systems Administrators chapter 2

Chapter 2 focuses specifically on how Ethernet works.

From defining broadcast domains to dealing with ethernet frame MAC addressing, he begins with the foundational basics.  He talks about duplex and speed for negotiating the connection, which both sides of the link need to agree upon.

He explains MTU (maximum transmission unit) and why mucking with reducing it below the default is almost certainly going to be problematic.  He also suggests ways to deal with situations where you can’t help but do so.

Lucas follows up with a quick explanation about the differences in category numbering for the wires that handle the transmissions.  The higher the number, the better, but higher costs can be prohibitive.  Work with what you can, but plan for higher when you can.

He moves into a quick explanation for using several troubleshooting tools, such as ping, arp, and “neighborhood discovery” (ND.)

Finally, he covers Virtual LANs (VLANs) to add a tag to an Ethernet Frame.  This allows traffic for multiple networks to flow over a single cable, without confusing where the packet should go.

He sprinkles in some more troubleshooting tools such as netstat, as well as tools to configure the ethernet layer (ifconfig and ethtool,) before closing the chapter out.

This chapter is important for the fundamentals of the “Data Link” layer (and to some extent the Physical Layer.)  Next week’s chapter covers “Layer 3” (Network Layer.)  This is mostly the “IP” layer in “TCP/IP” terms.

Thanks for reading!

Fun-Day Friday – Book Review – Networking for Systems Administrators chapter 1

Continuing our review of “Networking for Systems Administrators” by Michael W. Lucas, we’ll roll right into Chapter 1.

This chapter focuses on thinking in layers.  There are different network layer models, including the 7 layer OSI model, but Lucas says you really only need 5 layers to represent the network.  There’s the 4 layers of the TCP/IP model, but he splits the lowest layer into the Physical and DataLink layers.  This matches the OSI model’s way of presenting those layers.

The OSI Session, Presentation, and Application layers are all lumped under the TCP/IP “Application Layer.”  Lucas calls this the “your stuff” layer, and it’s true.  The Network Admins won’t really care much beyond the Transport Layer when troubleshooting.

Speaking of troubleshooting, identifying the lowest layer that is broken is crucial for this.  Fix that layer first, and most (if not all) of the other layers will likely start working again.  And those that don’t, go with the next lowest layer, and work your way up.

The rest of the chapter covers specific troubleshooting techniques for each of the lower layers, with a promise for more in depth troubleshooting discussion later in the book.

This chapter is short, but critical for laying the ground work.  Understanding these layers is one of the most important things to know for network troubleshooting.