Follow me on Twitter:

Back to Basics: Unix System Stats Utilities

Posted: February 24th, 2010 | Author: | Filed under: Linux / Unix | Tags: , , , , , | No Comments »

Unix and Linux systems have forever been obtuse and mysterious for many people. They generally don’t have nice graphical utilities for displaying system performance information; you need to know how to coax the information you need. Furthermore, you need to know how to interpret the information you’re given. Let’s take a look at some common system tools that can provide tons of visibility into what the opaque OS is really doing.

Unfortunately, the same tools don’t exist universally across all Unix variants. A few commonly underused ones do, however, and that is what we’ll focus on first.

Disk Activity
A common source of “slowness” is disk I/O, or rather the lack of available I/O. On Linux especially, it may be a difficult diagnosis. Often the load average will climb quickly, but without any corresponding processes in top eating much CPU. Linux counts “iowait” as CPU time when calculating load average. I’ve seen load numbers in the tens of thousands, on more than one occasion.

The easiest way to see what’s happening to your disks is to run the ‘iostat’ program. Via iostat, you can see how many read and write operations are happening per device, how much CPU is being utilized, and how long each transaction takes. Many arguments are available for iostat, so do spend some time with the man page on your specific system. By default, running ‘iostat’ with no arguments produces a report about disk IO since boot. To get a snapshot of “now” add a numerical argument last, which will prompt iostat to gather statistics for that number of seconds.

Linux will show number of blocks read or written per second, along with some useful CPU statistics. This is one particularly busy server:

 avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 1.36    0.07    5.21   23.80    0.00   69.57
Device:   tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda       18.22     15723.35       643.25 65474958946 2678596632

Notice that iowait is at 23%. This means that 23% of the time this server is waiting on disk I/O. Some Solaris iostat output shows a similar thing, just represented differently(iostat -xnz):

    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
295.3   79.7 5657.8  211.0  0.0 10.3    0.0   27.4   0 100 d101
134.8   16.4 4069.8  116.0  0.0  3.5    0.0   23.3   0  90 d105

The %b (block) column shows that I/O to device d101 is 100% blocked waiting for the device to complete transaction. The average service time isn’t good either: disk reads shouldn’t take 27.4ms. Arguably, Solaris’s output is more friendly to parse, since it gives the reads per second in kilobytes rather than blocks. We can quickly calculate that this server is reading about 19KB per read by dividing the number of KB read per second by the number of reads that happened. In short: this disk array is being taxed by large amounts of read requests.

The ‘vmstat’ program is also universally available, and extremely useful. It, too, provides vastly different information among operating systems. The vmstat utility will show you statistics about the virtual memory subsystem, or to put it simply: swap space. It is much more complex than just swap, as nearly every IO operation involves the VM system when pages of memory are allocated.A disk write, network packet send, and the obvious “program allocates RAM” all impact what you see in vmstat.

Running vmstat with the -p argument will print out statistics about disk IO. In Solaris you get some disk information anyway, as seen below:

 kthr      memory            page            disk
 faults      cpu  r b w   swap
free  re  mf pi po fr de sr m0 m1 m2 m7
in   sy   cs us sy id  0 0 0 7856104 526824 386 2401 0 0 0  0  0  3  0  0  0
16586 22969 12576 8 9 83  1 0 0 7851344 522016 18 678 32 0  0  0  0  2
0  0  0 13048 11737 10197 7 6 86  0 0 0 7843584 514128 76 3330 197 0
0 0  0  2  0  0  0 4762 131492 4441 16 8 76

A subtle, but important differences between Solaris and Linux is that Solaris will start scanning for pages of memory that can be freed before it will actually start swapping RAM to disk. The ‘sr’ column, scan rate, will start increasing right before swapping takes place, and continue until some RAM is available. The normal things are available in all operating systems; these include: swap space, free memory, pages in and out (careful, this doesn’t mean swapping is happening), page faults, context switches, and some CPU idle/system/user statistics. Once you know how to interpret these items you quickly learn to infer what they indicate about the usage of your system.

The two main programs for finding “slowness” are therefore iostat and vmstat. Before the obligatory tangent into “what Dtrace can do for you,” here’s a few other tools that no Unix junkie should leave home without:

Lists open files (including network ports) for all processes
Lists all sockets in use by the system
Shows CPU statistics (including IO), per-processor

We cannot talk about system visibility without mentioning Dtrace. Invented by Sun, Dtrace provides dynamic tracing of everything about a system. Dtrace gives you the ability to ask any arbitrary question about the state of a system, which works by calling “probes” within the kernel. That sounds intimidating, doesn’t it?

Let’s say that we wanted to know what files were being read or written on our Linux server that has a high iowait percentage. There’s simply no way to know. Let’s ask the same question of Solaris, and instead of learning Dtrace, we’ll find something useful in the Dtrace ToolKit. In the kit, you’ll find a few neat programs like iosnoop and iotop, which will tell you which processes are doing all the disk IO operations. Neat, but we really want to know what files are being accessed so much. In the FS directory, the rfileio.d script will provide this information. Run it, and you’ll see every file that’s read or written, and cache hit statistics. There’s no way to get this information in other Unixes, and this is just one simple example of how Dtrace is invaluable.

The script itself is about 90 lines, inclusive of comments, but the bulk of it is dealing with cache statistics. An excellent way to start learning Dtrace is to simply read the Dtrace ToolKit scripts.

Don’t worry if you’re not a Solaris admin: Dtrace is coming soon to a FreeBSD near you. SystemTap, a replica of Dtrace, will be available for Linux soon as well. Until then, and even afterward, the above mentioned tools will still be invaluable. If you can quickly get disk IO statistics and see if you’re swapping the majority of system performance problems are solved. Dtrace also provides amazing application tracing functionality, and if you’re looking at the application itself, you already know the slowness isn’t likely being caused by a system problem.

Soon, I’ll publish a few Dtrace tutorials.

Some things have surely been left out – discuss below!

No Comments yet... be the first »

Back To Basics: Unix Differences in Performing Tasks

Posted: February 23rd, 2010 | Author: | Filed under: Linux / Unix | Tags: , , , , , | 3 Comments »

It has often been said that a skilled sysadmin can quickly come up to speed with any Unix system in a matter of hours. After all, the underlying principals are all the same. Fortunately, this is somewhat correct. Unfortunately, this also leads to people making changes on systems they do not understand, often times in suboptimal ways.

In this final Back to Basics With Unix piece, we’d like to spend some time talking about some common, routine sysadmin tasks and how they differ between Unix variants.

Sure, you can clunk around and change configuration files to mostly make something work on a foreign system. But will those changes remain after security patches get applied and stomp all over your work? Did you just change a file that was meant to never change, because there’s a separate file for local modifications? If you’re not familiar with “how it’s done” in that particular OS, it’s as likely as not.

Yes, I make fun of GUI configuration utilities. People that don’t understand systems often use them and “get by,” but they cannot fix things when they break, unless the GUI tool can do it for them. That said, they do have their place. When learning a new system, it often makes sense to use the provided configuration utilities, as you know without a doubt they will adjust the necessary setting they way the OS wants it done. Here’s a list of some handy general administration GUIs:

– AIX: smitty (does pretty much everything)
– FreeBSD: sysinstall (not recommended for use after the initial install, but it works)
– HP-UX: sam (like AIX’s smitty)
– Linux: system-config, webmin and many others (distro-dependant)
– Solaris: admintool, wbem (use with caution)

Often, these tools still don’t do what you need. They certainly don’t help you learn a system unless you take the time to examine what the tool actually changed. Let’s start off with the basics: gathering system information and managing hardware. It can be a nightmare to add a disk to a foreign system, so hopefully this list will get you steered in the proper direction.

Show hardware configuration:
– AIX: lsdev, lscfg, prtconf
– FreeBSD: /var/run/dmesg.boot, pciconf
– HP-UX: ioscan, model, getconf, print_manifest
– Linux: dmesg, lspci, lshw, dmidecode
– Solaris: prtconf, prtdiag, psrinfo, cfgadm

Note that ‘dmesg’ is a circular kernel buffer on most systems, and after the machine has been up for a while the boot information listing devices gets overwritten. FreeBSD thoughtfully saves it in dmesg.boot for you, but in other systems you’re left relying on the above-mentioned exploratory tools.

Add a new device (have the OS discover it without a reboot):
– AIX: cfgmgr
– FreeBSD: atacontrol, camcontrol
– HP-UX: ioscan, insf
– Linux: udev, hotplug (automatic)
– Solaris: devfsadm, disks, devlinks (all a hardlink to the same binary now)

If you connect a new internal disk and need it recognized, you should not need to reboot in the Unix world. The above commands will discover new devices and make them available. If you’re talking about SAN disks, the utilities are mostly the same, but there are other programs that make the process much easier and also allow for multipathing configurations.

Label and partition a disk:
– AIX: mkvg then mklv
– FreeBSD: fdisk or sysinstall
– HP-UX: pvcreate then lvcreate, or sam
– Linux: fdisk or others
– Solaris: format or fmthard

Of course, you’ll also want to create a file system on your new disk. This is newfs or mkfs everywhere, with the exception of AIX which forces you to use crfs. The filesystem tab file, which describes file systems and mount options, vary a bit as well. In Linux, FreeBSD, and HP-UX it is /etc/fstab, Solaris uses /etc/vfstab, and AIX references /etc/filesystems. We spent so much time on filesystems and hardware because that’s the generally the biggest hurdle when learning a new system, and when you’re needing to do it, often you’re in a hurry.

Other tasks may or may not be covered by GUI utilities in the various flavors of Unix, so here’s a few more that we deem crucial to understand.

Display IP information and change IP address permanently:
– AIX: ifconfig/lsattr; smitty or chdev
– FreeBSD: ifconfig; /etc/rc.conf
– HP-UX: ifconfig/lanadmin; set_params
– Linux: ‘ip addr’; /etc/sysconfig/network or /etc/network/interfaces
– Solaris: ifconfig; edit /etc/hosts, /etc/hostname.*

Linux will of course vary, but those two files cover the most popular distros.

When taking over a foreign system, we frequently want to two two things: install missing software (like GNU utilities), and verify that the system is up-to-date on security patches. Where to get packages and where to gete latest security patches varies too much to cover here—you’ll likely need to search to the OS in question—but the way you install packages and show installed patches is extremely useful to know.

List installed patches:
– AIX: instfix, oslevel
– FreeBSD: uname
– HP-UX: swlist
– Linux: rpm, dpkg
– Solaris: showrev

Install packages:
– AIX: smitty, rpm, installp
– FreeBSD: pkg_add, portinstall, sysinstall
– HP-UX: swinstall
– Linux: rpm, yum, apt, yast, etc.
– Solaris: pkgadd

As you can see, things vary immensely between the Unix variants. Even within all of Linux you can easily find yourself lost. Google is a friend to all sysadmins, but too often the conceptual questions go unanswered. Here’s a general rule of thumb, and something I’ve seen done incorrectly too many times: if you see a configuration file in /etc/, say syslog.conf, and there is an accompanying syslog.d directory, you are not supposed to edit the syslog.conf file directly. The same goes for pam.conf and pam.d. Each service will have their own file within the .d directory, and that is where they are configured.

The .d directory example is mostly applicable to Linux, but be sure to pay attention when you see similar multi-config layouts anywhere else. Future sysadmins using the system will thank you if the OS’s conventions are followed and it’s easy to identify customizations. It also means that your changes aren’t likely to be stomped over by updates.


Working With Unix Variant Differences

Posted: February 20th, 2010 | Author: | Filed under: Linux / Unix | Tags: , , , , | No Comments »

One thing is for certain: Unix is complicated. Linux does it one way, Solaris another, and all the BSDs, yet another. Fortunately there is some logic behind the differences. Some differences have to do with where the OS came from, and some were deign choices, intended to improve usability. In this article we’ll talk about a few major differences between the Unix variants, and tell you what you need to know about various differences in command-line utilities.


First, recall that Unix started off in research labs, and two main flavors came about: System V (SysV), and BSD. SysV (five, not “vee”) spawned from AT&T Unix, in their fourth version, SVR4. BSD, from Berkeley, is the competing Unix variant. They both derived from the same Unix from Bell labs, but quickly diverged. Despite POSIX efforts, there are still BSD and SysV systems today, and their functionality still diverges.

Most operating systems are pretty clearly associated with one or the other, and generalizations about BSD vs. SysV prove correct. FreeBSD is the main branch from the traditional BSD, soon followed by NetBSD and OpenBSD. Then OS X came about, which was loosely based on FreeBSD (but is very BSD-like). On the SysV side of the house, AIX, IRIX, and HP-UX were the main variants. In short: commercial entities focused on SysV, academics focused on BSD.

Linux, however, is an oddball. Linux certainly adopted many SysV methodologies, but these days it is also very BSD-like. Sun Solaris, too, is confusing. SunOS started off as BSD, but SunOS 4 was the last BSD version; SunOS 5.x (aka Solaris) is now SysV. The details are much crazier than I’ve alluded to here, and we probably don’t want yet another Unix history lesson. A fun place to start for further reading is the Wikipedia page on Unix_wars.

Fundamental Differences
It has been said that one can tell which system they are using based on two indicators: whether or not the system boots with inittab, and the format of their accounting file. Process accounting isn’t really used any longer, and most people don’t even know what it’s for, so that’s mostly moot. The boot system, however, is still critical to understand.

SysV booting means you use inittab. The init program, when run by the kernel, will check /etc/inittab for the initdefault entry, and then boots to the runlevel defined there. Entering a runlevel means that each startup script in the directory will be run in order. Sequentially, and slowly. Sun was so annoyed with this they implemented a mechanism to fire up services in parallel, among other things, with the Service Management Facility (SMF). Ubuntu Linux implemented Upstart, which basically works around the sequential nature of init scripts too.

BSD booting means that init simply runs /etc/rc, and that’s all. Well, it used to. Soon BSD systems implemented rc.local, so that software and sysadmins alike could implement changes without fear of harming the critical system startup routines. Then /etc/rc.d/ was implemented, so that each script could live separately, just like SysV init scripts. Traditionally, BSD-style scripts didn’t take arguments, because there are no runlevels, and they only run once: on startup. There are still no runlevels in BSD, but the startup scripts generally take “start” and “stop” arguments, to allow sysadmins and package management tools to restart services easily.

Command Arguments
The most frustrating, and quickest to surface differences between SysV and BSD, are in the traditional utilities. Some common commands take very different arguments, and even have some very different functionality. This isn’t so important if you’re in Linux now, as it generally supports both, but once you find yourself in BSD-land, you’re up for some confusion.

The first command people usually run into is ‘ps.’ The arguments differ:

  • SysV: ps –elf
  • BSD: ps aux

Linux supports both, BSD does not. Often we may want to list all processes owned by a particular user. In BSD, you must run, “ps aux |grep username” but in SysV you can run, “ps –u username.” Just plain ‘ps’ will list your own processes in both flavors.

Another commonly noticed difference is with the ‘du’ command. Not because some older systems don’t support the –h argument to provide human-readable output, but because they display different things.

  • SysV: shows the amount available in 512-byte blocks
  • BSD: nice output showing size in bytes and percentage used

Printing in BSD is always confusing for SysV users, and vice-versa. Again this isn’t as common, since newer OSes support both, but it’s noteworthy nonetheless. BSD systems traditionally used lpr, lpq, and lprm to administer print jobs, whereas SysV had lp, lpstat, and cancel. Most systems adopted the BSD style, since lpr-ng (next generation) provided these commands, and CUPS subsequently adopted the BSD variants.

Other programs, such as du, who, ln, tr and more will have slight differences between SysV and BSD. Heck, the differences between the various Unix standards are confusing enough that a single Unix variant may have multiple directories of utilities. Take a look at Solaris’s /usr/ucb, /usr/xpg4, and /usr/xpg6 directories. Each standard they support, which has differences from POSIX, is documented and implemented in a separate location. Too bad Linux doesn’t comply with any standards.

In the end, the differences outlined here are probably the only ones anyone would ever notice. The nuances between du, for example, may be applicable for people writing shell scripts for systems administration procedures. The differences do turn up often enough to be mentionable, but in reality this level of work requires reading manual pages so often that they’d figure it out quickly. User-level utilities are “similar enough” with the exception of ps.

There are so many other differences in system maintenance procedures that those are more frequently focused on. Once the ‘ps’ hurdle is out of the way, and you understand how the system boots, the main problems are more conceptual, as in “how do I add a user.” These vary by OS, and also by distribution of Linux.

Come back next week to learn about the different ways Unix-like operating systems facilitate systems administration tasks.

No Comments yet... be the first »