Pi-Top Kali – Getting Kali Installed

Over the last few weeks, I’ve gone through the process of installing Kali onto its own microSD card for use on the pi-top.  I started with the latest 2017 image, but recently the first 2018 image became available, so I did the procedure again.  This document will cover the latest image.

First, I downloaded the image from the site onto my Linux Mint laptop.  Next, I verified the checksum.

sha256sum ./kali-linux-2018.1-rpi3-nexmon.img.xz
1ce9fb1ab69c709046b3ddddfeff6481b484f19e8b2b61725cebfb6361953c08  ./kali-linux-2018.1-rpi3-nexmon.img.xz

This matched the checksum on the site.  After verification, I unpacked the xz archive with unxz.  This created the kali-linux-2018.1-rpi3-nexmon.img image file for installation.

unxz kali-linux-2018.1-rpi3-nexmon.img.xz

Next, I put a new 32G microSD card in to receive the image.  I checked dmesg output to verify that the card was recognized successfully.

[11115.585761] mmc0: new ultra high speed SDR50 SDHC card at address 0001
[11115.603234] mmcblk0: mmc0:0001 00000 29.8 GiB
[11115.606945]  mmcblk0: p1

In order to install, I unmounted the auto-mounted card and used dd to tranfer a copy of the image to the card.

sudo dd if=./kali-linux-2018.1-rpi3-nexmon.img of=/dev/mmcblk0 bs=512k
14000+0 records in
14000+0 records out
7340032000 bytes (7.3 GB, 6.8 GiB) copied, 388.375 s, 18.9 MB/s

I ejected and re-inserted the card to verify it is seen correctly.  Both partitions were picked up, and I unmounted these for the next step.  Because the image is much smaller than the size of the card, I needed to resize the linux partition to take advantage of the extra space.  I used GParted for this.

GParted has an issue with the e2fsck not being up to date enough.  I had to install the following two packages manually, before growing the EXT4 mount point:

sudo apt-get install ./e2fsprogs_1.43.8-1ubuntu1_amd64.deb ./e2fslibs_1.43.8-1ubuntu1_amd64.deb

Once GParted was working correctly, I used it to resize the linux partiton.

sudo gparted /dev/mmcblk0

I selected the EXT4 mount, chose “move/resize,” and resized it to take up all of the unallocated space.  I then applied the change, and closed the program.

I swapped microSD cards to put Kali into the pi-top and remove polarisOS.  Booting the machine worked fine on first try, so the image appears to be installed okay.

I had a few housekeeping tasks to take care of.

My first housekeeping task was to create a new password for the root user.  Second was to regenerate host keys.  Most pre-built images should go through these two basic steps, since they ship with pre-generated widely known shared keys and passwords.

To regenerate the host keys, I ran:
root@kali:~ rm /etc/ssh/ssh_host_*
root@kali:~ dpkg-reconfigure openssh-server
root@kali:~ service ssh restart

I was surprised to find that the newest version of kali actually didn’t ship with pre-installed ssh host keys.  When I ran the rm command, it failed with file not found, and a file listing of /etc/ssh confirmed there were none to remove.  I only needed the dpkg-reconfigure and service restart.

My next steps included updating the system, since Kali-Rolling updates very frequently.  In order to do this, I put the machine on the network, then ran the following:

ntpdate 0.us.pool.ntp.org
apt-get update
apt-get upgrade

The reason for the ntpdate first is that the machine doesn’t come with a configured NTP service, and the clock on the Pi puts the machine several months behind the real world.  It thought it was December 14 rather than current date.  The other two commands updated the APT cache, and then updated the system to latest.

My final task was to go through the process of updating the repository lists to be able to install the “pt-*” packages from polarisOS, but I ran into a dependency error on one of the packages, which prevented that from working properly.  I’ll continue to troubleshoot this issue before I go into the deep details of setting it up, but the error I ran into was that the python3-pt-idletime package was dependent upon python3 (< 3.6) where 3.6.4 is the current version in Kali.

Pi-Top Kali – First Boot PolarisOS

Last week we talked about the kit, how it went together, and the fact that the first boot was to the pi-top polarisOS microSD card included with the kit.  Today, I’ll go over a quick rambling of notes on how the second “First Boot” (after it charged) went, and what I poked around looking for in this OS.  This post will not include screenshots, but I may try to get a few and do a follow up later in the week that is screenshot heavy.

It is a graphical boot that brings up a LightDM Panel (auto log-in.)  This initial Panel is the “pi-topDASHBOARD” and contains the following three sections:

  • Learn. Play. Create (includes "pi-topCODER" launch and "CEEDuniverse" play buttons)
  • Quick Launch (includes application icons for common/popular applications.)
  • Notes (a notepad that can be typed into.)

Along the top of the Panel is a status bar that includes indicators for whether the network is working, battery status, date/time, and a selector for Settings.

Selecting any app from the Quick Launch (or pressing the “Panel” keyboard button) will make this initial panel go away, and present the typical X11 style desktop.

Pressing the “Panel” keyboard button will bring the panel view back.

Since the Battery Status only shows on the Panel bar, this is a good thing to have.

From the typical X11 style desktop view, there are a few apps pinned to the bottom task bar.

  • Chromium
  • File Manager (PCManFM)
  • LXTerminal
  • CEED Universe
  • 3D Slash
  • Mathematica
  • Wolfram

The far right of the status bar includes status icons for the network, volume, and time, as well as an “eject” icon for external media, and a “panel” icon that can be clicked instead of hitting the Panel button on the keyboard.

The primary menu icon is in the lower left corner and the menu includes categories for:

Out of the packages listed above, htop is a “top” replacement that has some enhanced features, CEED Universe is part of the pi-top “CEED” learning software, with online games and such, and the rest are pretty self explanatory (integrated development environments, programming languages, some games, web and email, and an office suite.)

The items of greatest interest to me from poking around with the intent of setting up Kali as the primary OS for this machine are the hardware related packages.  I want a battery status indicator, as well as power/terminal launch/panel control buttons to function from the keyboard.  In order to take advantage of these things, I took a look at the software repository settings, and found this:


deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
# Uncomment line below then 'apt-get update' to enable 'apt-get source'
#deb-src http://archive.raspbian.org/raspbian/ jessie main contrib non-free rpi


deb http://apt.pi-top.com/raspbian/ jessie main


deb http://archive.raspberrypi.org/debian/ jessie main ui
# Uncomment line below then 'apt-get update' to enable 'apt-get source'
#deb-src http://archive.raspberrypi.org/debian/ jessie main ui

All of the packages specifically for the pi-top hardware include a “pt-” prefix.  Using a mix of dpkg -S, dpkg -L, and apt-cache showpkg I was able to confirm that the pt-* packages are located at the raspbian repositories jessie main.  This should make it easy to install them for Kali to take advantage of the special keyboard keys, and maybe get a battery status indicator.

Pi-Top Kali – The Kit

Last week was all about the order and tracking.  Today will cover the unboxing, how the kit goes together, and some potential “gotchas” to watch for while putting your own kit together, if you’re following along.

The large shipping box contained two boxes.  The larger box had the shell/kit, and the smaller box held the Raspberry Pi 3.

pi-top case
Pi-Top Case

I opened the larger box first, and verified that the contents were okay.  Inside were the shell, a box containing the charging cable + plug adapters, some booklets on the pi-top itself as well as the inventor’s kit, and of course, a box containing the components to the inventor’s kit that came with the purchase.  I don’t plan to do anything with the inventor’s kit for a while, so I left that alone.

The first step in putting the kit together was to slide the keyboard panel down, then remove the screw holding the heat sink/GPIO bridge in place.

GPIO bridge + heat sink
GPIO bridge + heat sink

After that piece was removed, four screws were removed to take out the plastic card “stand in” for the Raspberry Pi 3.  This card has a little cushion on one side that helped protect the heat sink/GPIO bridge during shipping.  Once those were removed, I slid the hub to the right along its rail.

The next step was to unbox the Raspberry Pi 3 so that it could be installed.  What I did not do at first, but I recommend be done if you intend to use it at all, is to install the microSD card that came with a pre-installed pi-top polarisOS image.  Not installing the card before installing the Pi isn’t a deal breaker, but if you’re not going to swap cards, it’s less of a pain to do it now, rather than later.  If you do intend to swap cards, or you forget to install it before installing the Pi into the case, the little tool they include includes a groove for pulling/guiding the card into and out of the Pi.

To install the Pi, I had to line up the two USB prongs inside the case with the left most set of USB sockets on the Pi, then gently push up (toward the back of the case) to seat the board.  The next step was to put the screws back in to hold the Pi in place.  The holes in the Pi didn’t quite line up right with the holes for the screws, so a little effort may be needed in getting the Pi seated properly while installing these screws.

USB bridge
USB bridge
Pi Seated onto USB bridge
Pi Seated onto USB bridge

Once the screws were in place, I had to carefully slide the hub such that the audio plug and the HDMI plug slid into their respective sockets on the Pi.

Finally, I lined the heat sink/GPIO bridge up with the GPIO pins on the Pi, and carefully seated it until the pins on the other side seated into the smaller groove on the hub.  Once it was snugly in place, I installed the last (longest) screw to keep it seated.

Pi Installed
Pi Installed

With all of the components in play, and the microSD card installed, the last step was to slide the keyboard back into place and turn the machine on.  My first boot was to the pi-top polarisOS they included with the device, so that I could check that everything worked okay with their included software.

I noted that the battery indicator within polarisOS showed a 33% battery status, so I shut it back down and plugged it in.  It took a few hours to charge to full.  While charging the indicator light next to the USB ports on the back flashes.  When it is finished, it turns solid green.

Next week will cover the first boot, general poking around, and discovery of tools within polarisOS.

Pi-Top Kali – The Purchase

In the last update on this project, I went over the thought process that led to my decision to purchase the pi-top 2 kit.  Today, I’ll cover how the purchase, order tracking, and shipping tracking went.

I went to the site to place my order, since the AdaFruit reseller didn’t have this style available, yet.  The order itself was mostly straight forward, but the site had a few issues.  I chose to place the purchase as a “guest” because I didn’t see any need in creating an account with them.  My intent was to purchase this device, and then I had no reason to return.  There are only a few “modules” or add ons available through their site, and none of them really appealed to me, so this should have been a one and done deal.  I put in my credit card information, and then had to fill out some profile information such as name, address, and so on.

The first problem I ran into was the phone number field.  It was a required field, but no matter how I plugged in my phone number, it said “this number must be a valid international number,” (paraphrasing.)  There was a little flag icon next to the phone number field, but clicking it didn’t do anything.  It finally let me put in a number that was just 7 digits long, which I knew wasn’t correct, but didn’t care since the rest of the information was correct, and it wasn’t my intent to have to come back to the site after this purchase.

The purchase went through, and I got an order number.  I checked my email, and indeed, I had an order confirmation email, as well.  While reading the order confirmation, I ran across this statement:

Any inquiries about your order? Have any other questions for us? Visit our support page or contact us at support@pi-top.com, quoting your order number.

I read the support page link, and it stated that I would need an account if I wanted order tracking.  This was not mentioned during the order process prior to placing my order as a guest.  The instructions indicated that I should create the account “with the same email address as I used to place my order.”  I did so.  The profile page was not filled in all the way, so I went in and included the same information… name, address, phone number, and so on.

This time, the phone number field’s “flag” let me select the United States to indicate my country of origin.  There is a separate drop down field (both on the order page, and on the profile page for the account I had just set up) that asks for your country.  It would have made much more sense if the phone number field had keyed off of that, instead of requiring the kludgey flag clickbox thing, but it is what it is.

The next problem I ran into is that the order number did not show on this new account page.  I waited until the next day in case the linking might take some time.  When I didn’t see order information the next day, (Friday) I sent an email to the support email listed in the order confirmation email I had received.  This created yet another account that I had to set a password on, and the person that responded took two or three hours to respond.  Their question was to ask for which email I had used to order the package.  I felt this was an odd request, since I had used that same email to open my support request, but I sent a response within a minute of receiving this update, and waited.

I got no response the rest of that day.  I used the ticket system page itself to provide one more update as a “just in case my email reply didn’t go through” trigger, but still heard nothing over that weekend.  I just assumed that they were closed over weekends, and waited for Monday to roll back around in hopes of another update.

There were no status updates waiting for me Monday morning, so I sent another update request, since it should have been about noon U.K. time when I checked.  A couple of hours later, and I did get an update.  The update never did explain why the order status wasn’t showing on that account I set up, but it did include the tracking information, which is what I really wanted.  The package was already on a truck and out for delivery, so the shipping was very fast.

Based on my experiences, I can recommend that others create an account BEFORE purchasing, and be signed in DURING purchase of the kit.  I can’t guarantee that this will get you order tracking, but it has a better chance of it than the route I took.  All in all, the support experience wasn’t “great” but it was fast delivery (Ordered Thursday evening/Received Monday Afternoon,) and they did get me the tracking information before it was delivered.  Would I recommend this purchase?  So far, yes.

Pi-Top Kali – The Idea

As recently mentioned, I’ve been working on a few projects of late.  In preparation for an OpenSSH based class I might offer, I found myself wanting to offer a shorter class on OpenBSD’s VMM/VMD virtual machine hypervisor system.  In researching this VMM/VMD system, one of my tests involved booting a linux live disk.  I chose Kali for this.  Getting it to boot wasn’t straight forward, due to the lack of a graphics KVM style console.  The VMM/VMD hypervisor uses serial connections to the guest operating systems, so I had to find all of the bells and whistles to pass to the Kali boot loader to make it boot to a usable login prompt.

Secondary to the above, and partially why Kali was chosen, is the fact that my GCIH is half its lifetime old.  It’s a 4 year certification, and I’ve had it for 2 years, now.  I got a reminder that renewal is coming up, and I began the refresher research on what’s involved in keeping this certification maintained through the renewal process.  One option is to take another SANS course, and get a new certificate from it.  While I would love to do this at some point, their courses are very expensive.  I also have an itch to try a different certification provider, and one of those stands out above the rest, to me.  I’ve decided I will likely go for the PWK (Pentesting With Kali) class from Offensive Security, and take the OSCP (Offensive Security Certified Professional) exam and certification.  This certification has “teeth” in that you don’t memorize a question/answer pool in order to answer a bunch of questions that are similar, but not exactly the same.  Instead, they give you about 48 hours (2 days) for the total exam.  The first day is to do an actual penetration test of a 5 machine environment, and the second day is to give you time to do a professional quality write up/report of the pentest as if you were presenting the report to a client.  The cost is within reason, and my family supports me in this endeavor.  To that end, Kali is on my radar as a “use this frequently” system this year.

I have several options for running Kali moving forward, and I will cover many of them as I go on this journey.  I will eventually go over running it in virtual machine environments up to and including VMware Workstation, Oracle VirtualBox, as a ProxMox guest, and of course, through the serial console as an OpenBSD VMM/VMD guest.  I may or may not get around to covering running it as a live bootable USB stick, or as a physical install to a typical x86_64 laptop.  All of these are things I’m looking at, but the first thing I’ll cover is installation and use on a Raspberry Pi.

I’ve made a few attempts at using Kali on a Raspberry Pi before.  I had trouble getting the TFT displays working satisfactorily, and I benched those projects due to the level of hassle and my own time constraints.  I knew that when I circled back around to this idea, I’d want a bigger screen than either of those TFT displays offered.  I want the device to be portable enough that I can take it almost anywhere and set up shop, but I need a display that gives me enough work space to actually … work.

The smallest display I was willing to look at was the 7 inch displays available, but my wife has a 7 inch tablet, and it’s only a little larger than a modern day smart phone.  My latest failed Kali attempt was on my own tablet, where Kali Nethunter never seemed to get installed properly no matter how many times I went through the process.  I like this screen size, and there are a few 10 inch displays available.  I almost settled on a device that used one of these when I discovered that there is actually a kit that turns a Raspberry Pi into a laptop form factor.

The two versions of the kit available on AdaFruit are the first version of this product.  One is green, and one is grey, but the kit itself is otherwise the same.  The project site has an updated “pi-top 2” design, which moves the trackpad down below the keyboard, and gives room for the keyboard to be full size, which works better for me.  I never liked trackpads in general, because I tend to brush the thing while I’m typing, but I’m sure I’ll work around this limitation somehow.  This case is also green, with no grey option available.  I would prefer grey, but I can live with the green case as long as it is as functional as I hope it will be.

After all of the research I’ve done, I have decided on the pi-top as my next Kali attempt.  I’ve made the purchase for the pi-top 2 style case, and will cover the experience of how the order/tracking went, unboxing, setting it up, running the pi-top polarisOS that comes with it, and getting Kali installed and running on the new machine.

The order arrived today, but the write up for that will be next week.

Back from the ether – sort of

UnixSecLab fell off without warning or explanation last quarter.  There were several factors involved in this sudden disappearance, but I won’t list them all.  Some were family related, and some were “hey everyone, I have a cool new project in the works, and I want to announce it to the world in a big way when its done” related.  So here’s the skinny on what was relevant for last quarter that I didn’t report.

  • The first big project I started was for a class on OpenSSH.  I’m working on breaking down the man pages, re-organizing them into related / relevant sections, and writing up a presentation on each section to go into deep detail on even the most esoteric settings, plus discuss security implications of some of the potentially dangerous ones.  This has been bouncing around in my head for a while, and is part of why one of my first organized series of posts was SSH Start to Finish Architecture.  This project tapered off over the quarter due to the above mentioned family issues, and the inspiration for a smaller product offering…
  • The smaller second project was to develop a class on OpenBSD’s virtual machine hypervisor.  The VMM/VMD class idea was due to how new this software is to the OpenBSD ecosystem, and the lack of documentation on its use and set up outside of (the excellent) man pages.  The man pages do make it seem straight forward to use, but one of my first hurtles was getting an off the cuff live Linux CD running.  I chose Kali (since I’m also doing security related stuff on the side unrelated to the OpenSSH class I intend to use this for.)  The first hurtles involved figuring out how to make Kali boot to a root prompt in multi-user mode without getting hung up on trying to load the graphics.  It’s not a VMM/VMD issue, it’s a Linux boot options issue I had to research.
  • Since I started both of those (still in progress) projects, Michael W. Lucas has put a brand new edition of SSH Mastery into sponsorship, and I’ve learned that there may be another author working on a book about the OpenBSD hypervisor software.  This author’s Twitter is @pb_double.
  • Hak5 announced a new device that I will want to cover a bit near the end of the year, as well.  The Packet Squirrel is a nifty hardware man-in-the-middle device that has a switch similar to the Bash Bunny so that you can set it to different modes on the fly without having to reprogram it every time you want to use it.  It comes with three pre-programmed modes, including a raw tcpdump mode, an OpenVPN mode, and a DNS spoof mode.  Some setup is required for the last two.
  • I got a notice that my GCIH certification will expire in two years.  I knew this already, but it reminded me that I need to get some continuing education credits, and possibly look for a new certification, as well.  The SANS institute’s on demand classes are a steep price for an individual, and while obtaining a new SANS/GIAC cert would meet all of the requirements to renew the GCIH, I’m looking at other options.  One of those is the Offensive Security Certified Professional.  This is the cert for their PWK class (Penetration testing With Kali.)  From what I’ve been reading, it’s a rigorous class with a lab full of 50+ target machines, and the certification exam is a 5 machine live pentest.  A little under 24 hours are spent testing these machines, and then another 24 hours are given to finish and submit a report on findings.  I’m strongly leaning this direction.
  • Since I’m leaning that direction, I need to brush up on my offensive skills a bit.  I found an article that covers a bit on how to prepare for the OSCP.  It has some suggested links to online capture the flag sites, as well as some general advice and resources on brushing up.  The last two days I’ve done two full CTF machines from Over The Wire, and it was a lot of fun.  I completed Bandit and Leviathan.
  • Another new find (for me) is an online security training site that doesn’t cost anything for the classes.  Cybrary.it has a lot of good content, from what I can gather thus far, and it’s worth a look if you’re on a budget and trying to get a foothold into this space.

The posts will still be a little sporadic for a bit, but we’re back, and we’re going to catch up on some lost work.  I’ll share some tidbits of things I’ve learned while doing the CTFs (without doing any walkthroughs or mentioning any specific machines) as well as try to wrap up some of the dangling series posts from last year.

Happy New Year (2018) and thanks for sticking with us during the information drought!

Book Review – Networking for Systems Administrators chapter 8

There are 5 chapters left, including this one, and I would like to finish my chapter by chapter review before next year, so I’ll have to do more than one of these per month, at least one month.  I’ll probably do two a month to accelerate the process.  I apologize for the interruption to the current series, but we’ll return to it next week.

Chapter 8 is about DNS, or the Domain Name System.  As Michael W. Lucas states, there are books much larger than this one that cover this single topic alone, so this chapter is a very brief overview into the fundamentals of viewing it from a troubleshooting perspective.

The service runs on port 53, both UDP and TCP.  Many organizations only allow port 53 UDP traffic, but TCP is required for larger requests.  The chapter discusses how DNS servers keep a mapping of name and IP address relationships for translating requests between the two.

The name mappings are defined within zones.  Each layer of an address (read right to left) represents another zone.  For example, the “.net,” “.com,” and “.org” endings we often use are top level zones.  The book’s examples include “google.com” and “michaelwlucas.com” as child zones of the top level “.com” zone.  Any zone inside another zone is a child zone.  This could mean a1.www.mysite.noip makes www.mysite.noip a child zone of mysite.noip, which is in turn a child zone of .noip, in this scenario.

DNS servers are either authoritative, or recursive.  Authoritative nameservers contain the information for specific domains.  Recursive nameservers provide DNS lookups for clients.  These servers find the authoritative servers, then query it, then returns the result to the client.

Ideally, authoritative and recursive nameservers should be on separate machines.  This is for security reasons, as well as simplification of configuration.

Next, the author covers the DNS hierarchy, explaining how DNS is a distributed database, and how the queries work their way up the chain until a server is capable of providing an authoritative response.  Then, he goes into forward and reverse lookups.  A forward lookup is the response given when querying what IP(s) belong to a name.  A reverse lookup is the response when querying what name belongs to an IP, also known as an PTR record.  The protocol allows for multiple PTR records for the same name, but in practice, this can break things.

The next section covers the different types of records that are relevant to most situations, such as A (name to IPv4,) AAAA (name to IPv6,) SOA (start of authority,) PTR, CNAME (canonical name… name to name alias mapping,) and MX (mail exchange) records.

A brief discussion of caching follows, which explains that changes can take time to propagate.  Then he covers why checking DNS is important.  If a server is responding with incorrect or even inconsistent information, it will likely cause issues with other troubleshooting steps.

He suggests using “nslookup” on Windows and “host” on Unix systems, but the “host” command may not be available.  He covers both of these tools in detail before briefly introducing the more advanced “dig” and “unbound-host” commands.  Finally he explains the “hosts” file for local name to IP mappings that may override responses from DNS, depending on how a system is configured.

And I’ll wrap this review with a sentence that leads to a footnote, then the footnote attached.

“A few failed DNS requests can drag some server software to a crawl or make it entirely fail.”

“Should software be written such that it handles DNS failures gracefully? Of course. And in that world, I have a pony. No, wait — a unicorn. No, better still — a ponycorn!”

Persistence through job control – RC scripts

For SystemV style systems, the next phase of the boot process after inittab is to kick off the rc scripts.   This is often one of the last entries in inittab, even.  The rc scripts on these systems typically begin with a script called “rc” that does some initial environmental setup, then it goes through and calls the different runlevel scripts based on which runlevel the system is booting into.

The rc scripts described here will be the same on both SystemV style systems, and “Upstart” init systems such as on Red Hat Enterprise Linux 6.  The “systemd” affliction does things differently, and we’ll cover it next week.

These runlevel rc scripts live in a structure that varies from system to system, but is often either directly under /etc or under /etc/rc.d as the parent directory.  The structure often looks like this:

/etc/rc.d/rc#.d (where "#" is the runlevel number.)

The init.d directory contains the actual scripts that start, stop, restart, or show status of various services.

The rc#.d directories contain symbolic links which point to the scripts in the init.d directory.  The names of these scripts determine whether to start or stop the script, and define which order they should be started or stopped.

For example, we might have a script called “httpd” that starts our web service.  We want this to be one of the last things started, and one of the first that gets stopped, so we might have a structure like this:

/etc/rc.d/init.d/httpd (the actual start/stop script)
/etc/rc.d/rc2.d/S99httpd (symbolic link to ../init.d/httpd)
/etc/rc.d/rc2.d/K01httpd (symbolic link to ../init.d/httpd)

The “S99httpd” link says to “S”tart it, and the high number puts it as a lower priority when starting services.  The “K01httpd” says to “K”ill it, (or stop it,) and the lower number gives it a higher priority when stopping services.  The standard rc script that parses these directories will take use the name to figure out what order to do things in, and will pass either a “start” or a “stop” based on the “S” or “K” at the beginning of the name.  The capitalization of the “S” and “K” are important.  An easy way to disable one of these temporarily, is to rename it with a lower case character, but keeping the name the same, otherwise.  This way you know what order it SHOULD be started or stopped if and when you want to re-enable it.

On complex runlevel systems, those numbers will vary depending on single or multi user mode, graphical environment mode, and so on.  On AIX, there are only two runlevels, so most things will be in rc2.d.  On BSD style systems, there is no actual concept of “runlevels” so much as there is a “local” rc file that has all of the settings inside of it, and the order is based on where they fall within the monolithic script.

In order to take advantage of the rc scripts for persistence, we would want to inject a call to our persistent shell within an existing script, or add one of our own.  Remember to put it after networking stuff is started up.

When we suspect this has been done, the routine is similar to inittab inspection.  Review all of the rc scripts, including “rc” itself, and for every call made, check that the file exists, is executable, and only contains what you expect it to contain.  A comparison from a known clean system (such as a fresh install to another machine) is a fast way to check the common items.  Anything that exists on our suspect machine, or any of the existing files that are different than originally delivered are worth digging deeper into.  Use diff, sdiff, and the like to make fast work of this.

Persistence through job control – inittab replacements

Last week we looked at a traditional “inittab,” using AIX’s inittab as an example.  This week, we’ll look at some of the “inittab” replacements that have popped up on various flavors of Linux.

As we mentioned last week, “Upstart” replaced “inittab” in Red Hat Enterprise Linux version 6, and “SystemD” unit files replaced it in RHEL7.

For Upstart, instead of a single “inittab” file, there is a directory called “/etc/init” that contains individual files that each control a single program to run.  There is no specified order by name of file, and the configuration files don’t contain any ordering themselves, other than to say “start me after this other process.”  With inittab, the order is controlled by where it falls within the file, so you get less fine grained control with this method, but you can be sure that a process that depends on another process already being up will wait, at least.

An interesting note about Upstart is that it doesn’t just check “/etc/init.”  It can also live in “~/.init/” which would start jobs for a user rather than for the system, and these jobs wouldn’t run as children of PID 1.  This gives some flexibility in dropping persistence in less explored locations on the file system, which means more places to audit for these kinds of persistence scripts.

An Upstart init file contains directives that respond to “emitted events.”  The basic events are “starting, started, stopping, and stopped.”  You can use a custom event that can be called by the emitter manually.

The program for sending “emitter events” to the Upstart “inittab” scripts is “initctl,” rather than “telinit.”  Instead of a “telinit q” you would use “initctl <emitter event> <file name>” to stop, start, and so on.

For SystemD, there is no inittab, either.  Everything is a Unit file, and lives in the same directory structure for both “init” type processes and “rc” type processes.  For that reason, we won’t go into SystemD too much, today, but we will say this much that relates:

In the [Service] section of the Unit file, include the ExecStart directive to call the process you want to be started, the Restart directive to inform whether it should be “always” or “on-failure,” as well as a RestartSec directive to tell it to throttle the restart attempts so that it doesn’t restart as fast as possible.

SystemD also includes a different command from “telinit” for controlling the Unit files.  The command “systemctl” will allow for starting, stopping, and status of the services controlled by Unit files.

Digging through all of the Unit files for the Exec lines and whether and how they are restarted is the key to looking for potentially persistent shells dropped in this manner.  Remember, though, that just because a process isn’t set to “Restart” by directive doesn’t mean this isn’t calling a script that automatically loops on itself similarly to how we will cover other job types, later.

Persistence through job control – inittab

The inittab file is often one of the first things loaded during the initialization process.  This file contains a list of processes that need to be run very early in the boot process, may need a “watchdog” to restart them on crash, and so on.  It also often contains the settings for defining and securing TTYs.  What we are concerned with is the “respawn” feature, since we want a place to drop a persistent process on the box.

Before we proceed, note that more modern varieties of Linux don’t use inittab, but have similar functions.  Red Hat Enterprise Linux (RHEL) version 6 has “upstart” which replaces the inittab with a directory containing individual files per process to be started.  RHEL version 7 has “systemd” which uses “unit” files to control just about everything.  We’ll cover those in next week’s article.  The “inittab” is a SystemV-ism, and OpenBSD doesn’t use it.  Instead of “inittab” the startup files begin in the “rc” scripts, and the TTYs are defined and secured in the “/etc/ttys” instead.  The reason for this is inittab is tied to the concept of run levels, and OpenBSD doesn’t really have those.

On AIX, the inittab is managed by a set of commands similar to many other base AIX commands (mk-, ch-, rm-, and ls- prefixed.)  The mkitab, chitab, rmitab, and lsitab commands create, change (modify,) remove, and list inittab entries (respectively.)  Let’s break down an inittab entry, using AIX’s inittab as a base line.

Each entry is colon “:” separated, similarly to /etc/passwd.  Lines are commented with a semi-colon “;” at the beginning of the line.  The first column is the label for the entry.  The second colomn is the run level that triggers the running of this command.  The third column is the “action” to take on the command.  This is where we define “run this one time at boot” vs. “run this again if it dies.”  The last column is “the command(s) to run.”  The reason I said “command(s)” is this can be a chain of commands piped together, not just a singular command.

To create an inittab entry (again, using AIX as an example,) we would do something similar to this:

mkitab "softaud:2:respawn:/usr/bin/audit start"

In the above example, we created an inittab entry labeled “softaud” that starts at runlevel 2, respawns on death, and calls the “/usr/bin/audit” command passing it “start” as input.  Since “audit” lives at “/usr/sbin/audit” on AIX, this entry is suspect.  If we were to leave a persistent script named “audit” in /usr/bin and then add this inittab entry, our script would be respawned on death, and would likely be overlooked by the systems administrators reviewing the file.  If we wanted to inject this after the existing entry already labeled as “audit” we could pass that mkitab command the “-i” flag followed by the label we want to append after.  Why “-i” for “append?”  You would have to ask IBM.

mkitab -i audit "softaud:2:respawn:/usr/bin/audit start"

The possible “actions” that can be passed to an inittab are listed in the man pages, but a few common ones are:
"once" - When the init command enters the run level specified for this record, start the process, do not wait for it to stop and when it does stop do not restart the process. If the system enters a new run level while the process is running, the process is not restarted.
"respawn" - If the process identified in this record does not exist, start the process. If the process currently exists, do nothing and continue scanning the /etc/inittab file.
"wait" - When the init command enters the run level specified for this record, start the process and wait for it to stop. While the init command is in the same run level, all subsequent reads of the /etc/inittab file ignore this object.

Some systems include a commands to make changes to inittab (such as we see with AIX.)  Other systems require you to modify the file directly using your favorite editor (vi, for example.)

Whatever your system requires, after a change is made, it only takes effect if you tell the “init” process to re-read the inittab.  To do this, you need the “telinit” command, and pass it the “q” option.

telinit q

Once it has been re-read, it will happily respawn the persistent script that was injected. A thorough audit of the inittab would involve going through the list of all commands in the file, checking that those processes exist at the locations given, doing a file type check, checksum check, and when possible, a “contents” check to see that they are legitimate.