For SystemV style systems, the next phase of the boot process after inittab is to kick off the rc scripts. This is often one of the last entries in inittab, even. The rc scripts on these systems typically begin with a script called “rc” that does some initial environmental setup, then it goes through and calls the different runlevel scripts based on which runlevel the system is booting into.
The rc scripts described here will be the same on both SystemV style systems, and “Upstart” init systems such as on Red Hat Enterprise Linux 6. The “systemd” affliction does things differently, and we’ll cover it next week.
These runlevel rc scripts live in a structure that varies from system to system, but is often either directly under /etc or under /etc/rc.d as the parent directory. The structure often looks like this:
/etc/rc.d/init.d
/etc/rc.d/rc#.d (where "#" is the runlevel number.)
The init.d directory contains the actual scripts that start, stop, restart, or show status of various services.
The rc#.d directories contain symbolic links which point to the scripts in the init.d directory. The names of these scripts determine whether to start or stop the script, and define which order they should be started or stopped.
For example, we might have a script called “httpd” that starts our web service. We want this to be one of the last things started, and one of the first that gets stopped, so we might have a structure like this:
/etc/rc.d/init.d/httpd (the actual start/stop script)
/etc/rc.d/rc2.d/S99httpd (symbolic link to ../init.d/httpd)
/etc/rc.d/rc2.d/K01httpd (symbolic link to ../init.d/httpd)
The “S99httpd” link says to “S”tart it, and the high number puts it as a lower priority when starting services. The “K01httpd” says to “K”ill it, (or stop it,) and the lower number gives it a higher priority when stopping services. The standard rc script that parses these directories will take use the name to figure out what order to do things in, and will pass either a “start” or a “stop” based on the “S” or “K” at the beginning of the name. The capitalization of the “S” and “K” are important. An easy way to disable one of these temporarily, is to rename it with a lower case character, but keeping the name the same, otherwise. This way you know what order it SHOULD be started or stopped if and when you want to re-enable it.
On complex runlevel systems, those numbers will vary depending on single or multi user mode, graphical environment mode, and so on. On AIX, there are only two runlevels, so most things will be in rc2.d. On BSD style systems, there is no actual concept of “runlevels” so much as there is a “local” rc file that has all of the settings inside of it, and the order is based on where they fall within the monolithic script.
In order to take advantage of the rc scripts for persistence, we would want to inject a call to our persistent shell within an existing script, or add one of our own. Remember to put it after networking stuff is started up.
When we suspect this has been done, the routine is similar to inittab inspection. Review all of the rc scripts, including “rc” itself, and for every call made, check that the file exists, is executable, and only contains what you expect it to contain. A comparison from a known clean system (such as a fresh install to another machine) is a fast way to check the common items. Anything that exists on our suspect machine, or any of the existing files that are different than originally delivered are worth digging deeper into. Use diff, sdiff, and the like to make fast work of this.