Tag: AIX

Persistence through job control – inittab

The inittab file is often one of the first things loaded during the initialization process.  This file contains a list of processes that need to be run very early in the boot process, may need a “watchdog” to restart them on crash, and so on.  It also often contains the settings for defining and securing TTYs.  What we are concerned with is the “respawn” feature, since we want a place to drop a persistent process on the box.

Before we proceed, note that more modern varieties of Linux don’t use inittab, but have similar functions.  Red Hat Enterprise Linux (RHEL) version 6 has “upstart” which replaces the inittab with a directory containing individual files per process to be started.  RHEL version 7 has “systemd” which uses “unit” files to control just about everything.  We’ll cover those in next week’s article.  The “inittab” is a SystemV-ism, and OpenBSD doesn’t use it.  Instead of “inittab” the startup files begin in the “rc” scripts, and the TTYs are defined and secured in the “/etc/ttys” instead.  The reason for this is inittab is tied to the concept of run levels, and OpenBSD doesn’t really have those.

On AIX, the inittab is managed by a set of commands similar to many other base AIX commands (mk-, ch-, rm-, and ls- prefixed.)  The mkitab, chitab, rmitab, and lsitab commands create, change (modify,) remove, and list inittab entries (respectively.)  Let’s break down an inittab entry, using AIX’s inittab as a base line.

Each entry is colon “:” separated, similarly to /etc/passwd.  Lines are commented with a semi-colon “;” at the beginning of the line.  The first column is the label for the entry.  The second colomn is the run level that triggers the running of this command.  The third column is the “action” to take on the command.  This is where we define “run this one time at boot” vs. “run this again if it dies.”  The last column is “the command(s) to run.”  The reason I said “command(s)” is this can be a chain of commands piped together, not just a singular command.

To create an inittab entry (again, using AIX as an example,) we would do something similar to this:

mkitab "softaud:2:respawn:/usr/bin/audit start"

In the above example, we created an inittab entry labeled “softaud” that starts at runlevel 2, respawns on death, and calls the “/usr/bin/audit” command passing it “start” as input.  Since “audit” lives at “/usr/sbin/audit” on AIX, this entry is suspect.  If we were to leave a persistent script named “audit” in /usr/bin and then add this inittab entry, our script would be respawned on death, and would likely be overlooked by the systems administrators reviewing the file.  If we wanted to inject this after the existing entry already labeled as “audit” we could pass that mkitab command the “-i” flag followed by the label we want to append after.  Why “-i” for “append?”  You would have to ask IBM.

mkitab -i audit "softaud:2:respawn:/usr/bin/audit start"

The possible “actions” that can be passed to an inittab are listed in the man pages, but a few common ones are:
"once" - When the init command enters the run level specified for this record, start the process, do not wait for it to stop and when it does stop do not restart the process. If the system enters a new run level while the process is running, the process is not restarted.
"respawn" - If the process identified in this record does not exist, start the process. If the process currently exists, do nothing and continue scanning the /etc/inittab file.
"wait" - When the init command enters the run level specified for this record, start the process and wait for it to stop. While the init command is in the same run level, all subsequent reads of the /etc/inittab file ignore this object.

Some systems include a commands to make changes to inittab (such as we see with AIX.)  Other systems require you to modify the file directly using your favorite editor (vi, for example.)

Whatever your system requires, after a change is made, it only takes effect if you tell the “init” process to re-read the inittab.  To do this, you need the “telinit” command, and pass it the “q” option.

telinit q

Once it has been re-read, it will happily respawn the persistent script that was injected. A thorough audit of the inittab would involve going through the list of all commands in the file, checking that those processes exist at the locations given, doing a file type check, checksum check, and when possible, a “contents” check to see that they are legitimate.

AIX notifications

In line with my lack of a “proper” Monday post, I’m skipping the “proper” Wednesday post to bring you an AIX specific tool I was asked to research for an internal project this week at work.

Many of you are likely familiar with Linux’s “inotify” process, which can be used to trigger a response to a file change in real time.  This is great for filesystem events.  You might even be able to crowbar this into notifying on process death events by monitoring something out of /proc, if you’re brave enough to go poking around in there.

On AIX, there is no native “inotify.”  There is, however, a special file system called the “Autonomic Health Advisor File System (AHAFS.)”  This is available by default, and is easy to set up.

Our particular scenario was to monitor when a specific process dies.  We want a real time notification, and log of the event.  I spent a few hours reading up on how this works, then played with it until I was sure I understood the best way to use it, and this is what I came up with.

There is a sample perl script “aha.pl” in the /usr/samples/ahafs/bin directory.  This can be used out of the box for most things.  In order to monitor a process, we need to know the path it is in.  In my example for this blog post, (not the actual service I needed to monitor,) we’ll say we want to monitor the apache httpd service.  It will be located at /usr/local/apache/bin/httpd for this example.

In order to monitor the death of this process, we would need to mimic that filesystem structure underneath the appropriate “Event Factory” directory.  In this case, that would be /aha/cpu/processMon.monFactory/ which means our final structure would be this:

/aha/cpu/processMon.monFactory/usr/local/apache/bin/

Before we just go making the directory, though, we need to mount the ahafs filesystem to “/aha.”  This is as simple as:

sudo mkdir /aha
sudo mount -v ahafs /aha /aha

Once that’s done, we can build out the structure.  The “cpu/processMon.monFactory” was created by mounting, but we need the subdirectory structure.

sudo mkdir -p /aha/cpu/processMon.monFactory/usr/local/apache/bin

Now we can monitor at any time.  The monitor itself will sit in “wait” status until the process being monitored dies, then it will print some event information.  This means that we key off of this monitor process coming to completion before we do any notifications of our own.  My recommendation is to tee the output of the event notification with “-a” to append, so that you have a log you can review, then have it email your support email account.

/usr/samples/ahafs/bin/aha.pl -m /aha/cpu/processMon.monFactory/usr/local/apache/bin/apache.mon "CHANGED=YES;INFO_LVL=3" 2>&1 | tee -a /var/log/apache.monitor.log | mailx -s "Apache died on $( hostname )" support@my.example.com

This should probably be placed within a script, and that script should be called after starting apache.  Of course, this is probably a terrible process to monitor, since httpd spawns off forked child processes all the time to handle requests, but it was an easy one to use as an example to get the point across on how to use this.

Also, it should be noted that for this specific scenario (PID dies, we want notification when it does,) if the process in question is one that doesn’t daemonize on its own, you can probably get away with just doing this:

echo "/usr/local/bin/my_process" | at now

When the process dies, the at job “dies” and you get an email notification from the at job termination itself.  You don’t get a local log appended, and you are relying solely on email for the notification, but it’s less work overall.