Follow me on Twitter:

Understanding Fibre Channel

Posted: March 15th, 2010 | Author: | Filed under: SAN 101 | Tags: , , , | 1 Comment »

As we dive deeper into SAN technology, it’s Fibre Channel’s turn to be examined. FC is the underpinning of all SAN technologies these days, as it won the protocol war roughly 25 years ago.

FC wouldn’t be much use without something on top of it, namely SCSI. FC is the low-level transport that ships data, but hosts are normally talking SCSI as far as they’re concerned. The hubs, switches, and HBAs in a SAN all speak FC, while the applications that use SAN storage continue to use familiar protocols, like SCSI.

The idea behind FC was to create a high throughput, low latency, reliable and scalable protocol. Ethernet wouldn’t quite cut it for highly-available storage needs. FC can currently operate of speeds up to 10Gb/s (10GFC) for uplinks, and 4Gb for standard host connections. FC also provides small connectors. As silly as it sounds, SCSI cables become unruly after time, and small strands of fibre are certainly easier to manage. The equipment required to connect to a FC SAN (multiple HBAs for each host, fibre, and switches) is extremely expensive, and was the main reason SAN technologies took so long to become widely adopted.

In reality, two different protocols, or topologies, make up the FC protocol. FC supports all topologies, but the behavior of the protocol changes depending on the topology. The following three types of topologies are supported:

  • PTP (point to point): normally used for DAS configurations.
  • FC-AL (FC Arbitrated Loop): Fabric Loop ports, or FL ports on a switch, and NL_Ports (node loop) on an HBA, support loop operations.
  • FC-SW (FC Switched): the mode when operating on a switched SAN.

FC-AL operation is quite scary, but sometimes a device doesn’t support FC-SW operations, and there’s no choice. A hub has no choice but to operate in FC-AL mode, and therefore attached hosts must as well. When a device joins an FC-AL, or when there’s any type of error or reset, the loop must reinitialize. All communication is temporarily halted during this process, so it can cause problems for some applications. FC-AL is limited to 127 nodes due to the addressing mechanism, in theory, but in reality closer to 20. FC-AL is mostly relegated to niche uses now, including but not limited to internal disk array communications and internal storage for high-end servers.

FC switches can be connected any way you please, since the FC protocol avoids the possibility of a loop by nature. Ethernet isn’t so lucky. The addressing scheme used does impose a limit of 239 switches though. FC switches use FSPF, a link-state protocol like OSPF in the IP world, to ensure loop-free and efficient connectivity.

FC networks are generally designed in one of two ways: either one big star, or one big star with edge switches hanging off it. These are commonly known as “core-only” and “core-edge” configurations. Normally a SAN will contain two of these networks, and each host’s HBA or storage device’s controller will attach to each. Keeping these networks separate isn’t as necessary as it is with FC-AL topologies, but even with FC-SW setups it still provides complete isolation and assurance that a problem in one fabric won’t impact the other. An FSPF recalculation, for example, could cause a brief interruption in service.

As previously mentioned, there are different port types in a SAN, and it can get confusing. Let’s try to clear up some of that terminology:

  • N_Port: Node Port; the node connection point; end points for FC traffic
  • F_Port: Fabric Port; a switch-connected port, that is a “middle point” connection for two N_Ports
  • NL_Port: Node Loop Port; connects to others via their NL_Ports, or to a switched fabric via a single FL_Port; or NL_port to F_Port to F_Port to N_Port (through a switch)
  • FL_Port: Fabric Loop Port; a shared point of entry into a fabric for AL devices; example  NL_Port to FL_Port to F_Port to N_Port
  • E_Port: Expansion Port; used to connect multiple switches together via ISL (inter-switch links)
  • G_Port: Generic Port; can switch between F_Port and E_Port operation depending on how it’s connected
  • TE_Port: Trunked Expansion Port; link aggregation of multiple E_Ports for higher throughput

You’ll generally only see F_Ports and FL_Ports when looking at a single SAN switch, and knowing the difference helps. FL means that you’re talking FC-AL, and there’s a device attached that is either a hub, something that can’t do anything but FC-AL, or something strange. Ports will automatically configure themselves as an FL_Port if the attached device is Loop-only, otherwise it will be an F_Port. It’s also worth noting that some brands of FC switches don’t allow you to have an E_Port unless you pay a higher licensing fee. It’s something to think about if you ever plan to connect multiple switches together.

FC Layers
FC has its own layers, so in fact, calling it “like Ethernet” isn’t quite accurate, even if it helps for understanding. They are:

  • FC-0: The interface to the physical media; cables, etc
  • FC-1: Transmission protocol or data-link layer, encodes and decodes signals
  • FC-2: Network Layer; the core of FC
  • FC-3: Common services, like hunt groups
  • FC-4: Everything! Protocol mapping for SCSI, iSCSI, FCP, IP, and others

The bulk of FC is really in FC-2. FC-PH refers to FC-0 through FC-2, which are strangely dubbed the physical layers.

FC also supports its own naming and addressing mechanism, which sheds light on the previously mentioned limitations in FC-AL and FC-SW topologies. Next time, we’ll discuss the header format for FC-2 as well as FC address assignment and name resolution.

In a Nutshell:

  • FC is the transport mechanism, and SCSI or even IP rid atop FC
  • FC-AL is a loop, where all connected devices see each other, and a re-initialization takes out the entire SAN
  • Port types reveal what is actually happening, and knowing what they stand for can aid in topology visualization when looking at a switch’s configuration

1 Comment »

Networking 101: Understanding Layers

Posted: March 7th, 2010 | Author: | Filed under: Networking 101 | Tags: , , , , | No Comments »

Continuing our journey, it’s time to take a trip up the OSI Reference Model, and learn what this mysterious thing is all about. The network stack is of great significance, but not so much that it’s the first thing you should learn. The networking 101 series has waited to ensue the “layers” discussion for good reason. Many so-called networking classes will start by teaching you to memorize the name of every layer and every protocol contained within this model. Don’t do that. Do realize that layers 5 and 6 can be completely ignored, though.

The International Standards Organization (ISO) developed the OSI (Open Systems Interconnection) model. It divides network communication into seven layers. Layers 1-4 are considered the lower layers, and mostly concern themselves with moving data around. Layers 5-7, the upper layers, contain application-level data. Networks operate on one basic principle: “pass it on.” Each layer takes care of a very specific job, and then passes the data onto the next layer.

The physical layer, layer 1, is too often ignored in a classroom setting. It may seem simple, but there are aspects of the first layer that oftentimes demand significant attention. Layer one is simply wiring, fiber, network cards, and anything else that is used to make two network devices communicate. Even a carrier pigeon would be considered layer one gear (see RFC 1149). Network troubleshooting will often lead to a layer one issue. We can’t forget the legendary story of CAT5 strung across the floor, and an office chair periodically rolling over it leading to spotty network connectivity. Sadly, this type of problem exists quite frequently, and takes the longest to troubleshoot.

Layer two is Ethernet, among other protocols; we’re keeping this simple, remember. The most important take-away from layer 2 land is that you should understand what a bridge is. Switches, as they’re called nowadays, are bridges. They all operate at layer 2, paying attention only to MAC addresses on Ethernet networks. The common fledgling network admin always seem to mix up layers two and three. If you’re talking about MAC address, switches, or network cards and drivers, you’re in the land of layer 2. Hubs live in layer 1 land, since they are simply electronic devices with zero layer 2 knowledge. Layer two will have it’s own section in Networking 101, so don’t worry about the details for now, just know that layer 2 translates data frames into bits for layer 1 processing.

On the other hand, if you’re talking about an IP address, you’re dealing with layer 3 and “packets” instead of layer 2’s “frames.” IP is part of layer 3, along with some routing protocols, and ARP (Address Resolution Protocol). Everything about routing is handled in layer 3. Addressing and routing is the main goal of this layer.

Layer four, the transport layer, handles messaging. Layer 4 data units are also called packets, but when you’re talking about specific protocols, like TCP, they’re “segments” or “datagrams” in UDP. This layer is responsible for getting the entire message, so it must keep track of fragmentation, out-of-order packets, and other perils. Another way to think of layer 4 is that it provides end-to-end management of communication. Some protocols, like TCP, do a very good job of making sure the communication is reliable. Some don’t really care if a few packets are lost–UDP is the prime example.

And arriving at layer seven, we wonder what happened to layer 5 and 6. They’re useless. A few applications and protocols live there, but for the understanding of networking issues, talking about these provides zero benefit. Layer 7, our friend, is “everything.” Dubbed the “Application Layer,” layer 7 is simply application-specific. If your program needs a specific format for data, you will invent some format that you expect the data to arrive in, and you’ve just created a layer 7 protocol. SMTP, DNS, FTP, etc, etc are all layer 7 protocols.

The most important thing to learn about the OSI model is what it really represents. Pretend you’re an operating system on a network. Your network card, operating at layers 1 and 2, will notify you when there’s data available. The driver handles the shedding of the layer 2 frame, which reveals a bright, shiny layer 3 packet inside (hopefully). You, as the operating system, will then call your routines for handling layer 3 data. If the data has been passed to you from below, you know that it’s a packet destined for yourself, or it’s a broadcast packet (unless you’re also a router, but never mind that for now). If you decide to keep the packet, you will unwrap it, and reveal a layer 4 packet. If it’s TCP, the TCP subsystem will be called to unwrap and pass the layer 7 data onto the application that’s listening on the port it’s destined for. That’s all!

When it’s time to respond to the other computer on the network, everything happens in reverse. The layer 7 application will ship its data onto the TCP people, who will stick additional headers onto the chunk of data. In this direction, the data gets larger with each progressive step. TCP hands a valid TCP segment onto IP, who give its packet to the Ethernet people, who will hand it off to the driver as a valid Ethernet frame. And then off it goes, across the network. Routers along the way will partially disassemble the packet to get at the layer 3 headers in order to determine where the packet should be shipped. If the destination is on the local Ethernet subnet, the OS will simply ARP for the computer instead of the router, and send it directly to the host.

Grossly simplified, sure; but if you can follow this progression and understand what’s happening to every packet at each stage, you’re just conquered a huge part of understanding networking. Everything gets horribly complex when you start talking about what each protocol actually does. If you are just beginning, please ignore all that stuff until you understand what the complex stuff is trying to accomplish. It makes for a much better learning endeavor! In future Networking 101 articles we will begin the journey up the stack, examining each layer in detail by discussing the common protocols and how they work.

No Comments yet... be the first »

Networking 101: More Subnets, and IPv6

Posted: February 27th, 2010 | Author: | Filed under: Networking, Networking 101 | Tags: , , , | 3 Comments »

What’s the point of creating subnets anyways? How do I remember those strange looking subnet masks? How the heck does this work with those crazy looking IPv6 addresses? This edition of Networking 101 will expand on the previous Subnets and CIDR article, in the interest of promoting a thorough understanding of subnetting.

An oft-asked question in networking classes is “why can’t we just put everyone on the same subnet and stop worrying about routing?” The reason is very simple. Every time someone needs to talk, be it to a router or another host, they have to send an ARP request. Also, there’s broadcast packets that aren’t necessarily limited to ARP, which everyone hears. When there are only 255 devices on a /24 subnet, the amount of broadcast packets are fairly limited. It is important to keep this number low, because every time a packet destined for a specific host or a broadcast address is seen, the host must handle the packet. A hardware interrupt is created, and the kernel of the operating system must read enough of the packet to determine whether or not it cares about it.

Broadcast storms happen at times, mainly because of layer 2 topology loops. We’ll explain layer 2 topology issues in excruciating (actually, enlightening) detail in a future issue. When thousands of packets hit a computer at a time, slow and fast computers alike can become very slow. The kernel spends so much time handling interrupts that it doesn’t have much left for dealing with “trivial” things like making sure your web browser process gets a chance to run. So that, my friends, is why subnets are very important. This is also known as a broadcast domain, because it limits the amount of broadcasts that you will hear.

The natural follow-up question normally involves a host’s notion of a broadcast address and netmask. We hopefully understand that a host needs to understand what computers are on the same subnet. Those IP addresses can be spoken to directly, making a router unnecessary. When the netmask or broadcast address is incorrectly configured, you’ll quickly find that some hosts are unreachable.

The most common erroneous configuration happens when someone configure an IP address without specifying the netmask and broadcast address. For some reason, most operating systems don’t take the liberty of updating these things, even though one can be determined from the other. If you run ‘ifconfig eth0 netmask’ you might expect that everything is ready to go. Unfortunately, it’s very likely that your broadcast address was set to It largely depends on the router’s configuration, but normally this results in all broadcast packets being dropped. Conversely, if the netmask is configured incorrectly, the computer wouldn’t know where the subnet starts and begins. If a computer thinks a host is on the same subnet when it actually isn’t, it will attempt to ARP for it instead of the router. Routers can be configured to handle this and pretend they are the host (called Proxy Arp), but normally the result is unreachable hosts.

Understand how the netmask is configured, to avoid this problem. Figuring out the network and broadcast address isn’t very difficult when you remember that the netmask simply means “cover some bits,” but deciphering netmask representation can induce a double-take. The netmask for a /24 network is, that’s easy. But what does mean? The best way to decipher it is to begin with the masked off part. Comparing it to the /24, which had three octets masked, we see that has two octets masked, and part of another. We know it’s between a /16 and a /24. We have to understand binary, and realize how many bits are masked. The last 16 bits are clearly part of the network portion. The third octet, 240, allows 16 IP addresses beyond the mask, so it must mean that four bits are left (2^4=16). The four remaining bits, plus the 16 bits used for the first two octets means that we’re dealing with a /20!

What about We’re definitely in a land smaller than the /24 subnet. If we look at the remaining bits in the last octet, we can see that there are eight IP addresses available. Remember that only 2^3 can make eight, so we’re using all but three bits in the network portion. This is a /29 network. Of course, the easy ones are pretty clear: allows half as many host addresses in the last octet compared to the /24 network, so it’s a /25.

On the topic of confusing netmasks, IPv6 addresses certainly have a place. The netmask isn’t really an issue–the same concept applies, just with larger numbers to remember. The real problem lies within the address representation itself; the IETF seemed to take pride in creating confusion. Typically an IPv6 address is represented in hex, or base-16. Our old friend IPv4 could represent an IP address in hex too, which would look like B.B.B.B for the address Unfortunately, IPv6 isn’t quite that nice looking. To represent 128 bits, IPv6 normally breaks up the address into eight 16-bit segments.

An IPv6 address looks like: 2013:4567:0000:CDEF:0000:0000:00AD:0000. It does get a bit easier. For example, leading zeros are not written, and contiguous quads of zeros get collapsed to ::. Trailing zeros ,however, must be shown. This is a bit confusing, but the rules always allow for a non-ambiguous IP address. Leading zeros in each quad can always be removed, but the collapsing of contiguous blocks of zeros can only happen once per address. The above address with collapsed zeros will look like: 2013:4567:0000:CDEF::AD:0000. IPv6 provides 2^128 addresses, more than enough to allocate roughly 1000+ IP addresses per square meter of the earth.

If you remember the rules of binary, the address representation rules with IPv6, and a few simple subnets for reference, you’ll be Master of Subnets – the one who everyone asks for help.


Back to Basics: Unix System Stats Utilities

Posted: February 24th, 2010 | Author: | Filed under: Linux / Unix | Tags: , , , , , | No Comments »

Unix and Linux systems have forever been obtuse and mysterious for many people. They generally don’t have nice graphical utilities for displaying system performance information; you need to know how to coax the information you need. Furthermore, you need to know how to interpret the information you’re given. Let’s take a look at some common system tools that can provide tons of visibility into what the opaque OS is really doing.

Unfortunately, the same tools don’t exist universally across all Unix variants. A few commonly underused ones do, however, and that is what we’ll focus on first.

Disk Activity
A common source of “slowness” is disk I/O, or rather the lack of available I/O. On Linux especially, it may be a difficult diagnosis. Often the load average will climb quickly, but without any corresponding processes in top eating much CPU. Linux counts “iowait” as CPU time when calculating load average. I’ve seen load numbers in the tens of thousands, on more than one occasion.

The easiest way to see what’s happening to your disks is to run the ‘iostat’ program. Via iostat, you can see how many read and write operations are happening per device, how much CPU is being utilized, and how long each transaction takes. Many arguments are available for iostat, so do spend some time with the man page on your specific system. By default, running ‘iostat’ with no arguments produces a report about disk IO since boot. To get a snapshot of “now” add a numerical argument last, which will prompt iostat to gather statistics for that number of seconds.

Linux will show number of blocks read or written per second, along with some useful CPU statistics. This is one particularly busy server:

 avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 1.36    0.07    5.21   23.80    0.00   69.57
Device:   tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda       18.22     15723.35       643.25 65474958946 2678596632

Notice that iowait is at 23%. This means that 23% of the time this server is waiting on disk I/O. Some Solaris iostat output shows a similar thing, just represented differently(iostat -xnz):

    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
295.3   79.7 5657.8  211.0  0.0 10.3    0.0   27.4   0 100 d101
134.8   16.4 4069.8  116.0  0.0  3.5    0.0   23.3   0  90 d105

The %b (block) column shows that I/O to device d101 is 100% blocked waiting for the device to complete transaction. The average service time isn’t good either: disk reads shouldn’t take 27.4ms. Arguably, Solaris’s output is more friendly to parse, since it gives the reads per second in kilobytes rather than blocks. We can quickly calculate that this server is reading about 19KB per read by dividing the number of KB read per second by the number of reads that happened. In short: this disk array is being taxed by large amounts of read requests.

The ‘vmstat’ program is also universally available, and extremely useful. It, too, provides vastly different information among operating systems. The vmstat utility will show you statistics about the virtual memory subsystem, or to put it simply: swap space. It is much more complex than just swap, as nearly every IO operation involves the VM system when pages of memory are allocated.A disk write, network packet send, and the obvious “program allocates RAM” all impact what you see in vmstat.

Running vmstat with the -p argument will print out statistics about disk IO. In Solaris you get some disk information anyway, as seen below:

 kthr      memory            page            disk
 faults      cpu  r b w   swap
free  re  mf pi po fr de sr m0 m1 m2 m7
in   sy   cs us sy id  0 0 0 7856104 526824 386 2401 0 0 0  0  0  3  0  0  0
16586 22969 12576 8 9 83  1 0 0 7851344 522016 18 678 32 0  0  0  0  2
0  0  0 13048 11737 10197 7 6 86  0 0 0 7843584 514128 76 3330 197 0
0 0  0  2  0  0  0 4762 131492 4441 16 8 76

A subtle, but important differences between Solaris and Linux is that Solaris will start scanning for pages of memory that can be freed before it will actually start swapping RAM to disk. The ‘sr’ column, scan rate, will start increasing right before swapping takes place, and continue until some RAM is available. The normal things are available in all operating systems; these include: swap space, free memory, pages in and out (careful, this doesn’t mean swapping is happening), page faults, context switches, and some CPU idle/system/user statistics. Once you know how to interpret these items you quickly learn to infer what they indicate about the usage of your system.

The two main programs for finding “slowness” are therefore iostat and vmstat. Before the obligatory tangent into “what Dtrace can do for you,” here’s a few other tools that no Unix junkie should leave home without:

Lists open files (including network ports) for all processes
Lists all sockets in use by the system
Shows CPU statistics (including IO), per-processor

We cannot talk about system visibility without mentioning Dtrace. Invented by Sun, Dtrace provides dynamic tracing of everything about a system. Dtrace gives you the ability to ask any arbitrary question about the state of a system, which works by calling “probes” within the kernel. That sounds intimidating, doesn’t it?

Let’s say that we wanted to know what files were being read or written on our Linux server that has a high iowait percentage. There’s simply no way to know. Let’s ask the same question of Solaris, and instead of learning Dtrace, we’ll find something useful in the Dtrace ToolKit. In the kit, you’ll find a few neat programs like iosnoop and iotop, which will tell you which processes are doing all the disk IO operations. Neat, but we really want to know what files are being accessed so much. In the FS directory, the rfileio.d script will provide this information. Run it, and you’ll see every file that’s read or written, and cache hit statistics. There’s no way to get this information in other Unixes, and this is just one simple example of how Dtrace is invaluable.

The script itself is about 90 lines, inclusive of comments, but the bulk of it is dealing with cache statistics. An excellent way to start learning Dtrace is to simply read the Dtrace ToolKit scripts.

Don’t worry if you’re not a Solaris admin: Dtrace is coming soon to a FreeBSD near you. SystemTap, a replica of Dtrace, will be available for Linux soon as well. Until then, and even afterward, the above mentioned tools will still be invaluable. If you can quickly get disk IO statistics and see if you’re swapping the majority of system performance problems are solved. Dtrace also provides amazing application tracing functionality, and if you’re looking at the application itself, you already know the slowness isn’t likely being caused by a system problem.

Soon, I’ll publish a few Dtrace tutorials.

Some things have surely been left out – discuss below!

No Comments yet... be the first »

Back To Basics: Unix Differences in Performing Tasks

Posted: February 23rd, 2010 | Author: | Filed under: Linux / Unix | Tags: , , , , , | 3 Comments »

It has often been said that a skilled sysadmin can quickly come up to speed with any Unix system in a matter of hours. After all, the underlying principals are all the same. Fortunately, this is somewhat correct. Unfortunately, this also leads to people making changes on systems they do not understand, often times in suboptimal ways.

In this final Back to Basics With Unix piece, we’d like to spend some time talking about some common, routine sysadmin tasks and how they differ between Unix variants.

Sure, you can clunk around and change configuration files to mostly make something work on a foreign system. But will those changes remain after security patches get applied and stomp all over your work? Did you just change a file that was meant to never change, because there’s a separate file for local modifications? If you’re not familiar with “how it’s done” in that particular OS, it’s as likely as not.

Yes, I make fun of GUI configuration utilities. People that don’t understand systems often use them and “get by,” but they cannot fix things when they break, unless the GUI tool can do it for them. That said, they do have their place. When learning a new system, it often makes sense to use the provided configuration utilities, as you know without a doubt they will adjust the necessary setting they way the OS wants it done. Here’s a list of some handy general administration GUIs:

– AIX: smitty (does pretty much everything)
– FreeBSD: sysinstall (not recommended for use after the initial install, but it works)
– HP-UX: sam (like AIX’s smitty)
– Linux: system-config, webmin and many others (distro-dependant)
– Solaris: admintool, wbem (use with caution)

Often, these tools still don’t do what you need. They certainly don’t help you learn a system unless you take the time to examine what the tool actually changed. Let’s start off with the basics: gathering system information and managing hardware. It can be a nightmare to add a disk to a foreign system, so hopefully this list will get you steered in the proper direction.

Show hardware configuration:
– AIX: lsdev, lscfg, prtconf
– FreeBSD: /var/run/dmesg.boot, pciconf
– HP-UX: ioscan, model, getconf, print_manifest
– Linux: dmesg, lspci, lshw, dmidecode
– Solaris: prtconf, prtdiag, psrinfo, cfgadm

Note that ‘dmesg’ is a circular kernel buffer on most systems, and after the machine has been up for a while the boot information listing devices gets overwritten. FreeBSD thoughtfully saves it in dmesg.boot for you, but in other systems you’re left relying on the above-mentioned exploratory tools.

Add a new device (have the OS discover it without a reboot):
– AIX: cfgmgr
– FreeBSD: atacontrol, camcontrol
– HP-UX: ioscan, insf
– Linux: udev, hotplug (automatic)
– Solaris: devfsadm, disks, devlinks (all a hardlink to the same binary now)

If you connect a new internal disk and need it recognized, you should not need to reboot in the Unix world. The above commands will discover new devices and make them available. If you’re talking about SAN disks, the utilities are mostly the same, but there are other programs that make the process much easier and also allow for multipathing configurations.

Label and partition a disk:
– AIX: mkvg then mklv
– FreeBSD: fdisk or sysinstall
– HP-UX: pvcreate then lvcreate, or sam
– Linux: fdisk or others
– Solaris: format or fmthard

Of course, you’ll also want to create a file system on your new disk. This is newfs or mkfs everywhere, with the exception of AIX which forces you to use crfs. The filesystem tab file, which describes file systems and mount options, vary a bit as well. In Linux, FreeBSD, and HP-UX it is /etc/fstab, Solaris uses /etc/vfstab, and AIX references /etc/filesystems. We spent so much time on filesystems and hardware because that’s the generally the biggest hurdle when learning a new system, and when you’re needing to do it, often you’re in a hurry.

Other tasks may or may not be covered by GUI utilities in the various flavors of Unix, so here’s a few more that we deem crucial to understand.

Display IP information and change IP address permanently:
– AIX: ifconfig/lsattr; smitty or chdev
– FreeBSD: ifconfig; /etc/rc.conf
– HP-UX: ifconfig/lanadmin; set_params
– Linux: ‘ip addr’; /etc/sysconfig/network or /etc/network/interfaces
– Solaris: ifconfig; edit /etc/hosts, /etc/hostname.*

Linux will of course vary, but those two files cover the most popular distros.

When taking over a foreign system, we frequently want to two two things: install missing software (like GNU utilities), and verify that the system is up-to-date on security patches. Where to get packages and where to gete latest security patches varies too much to cover here—you’ll likely need to search to the OS in question—but the way you install packages and show installed patches is extremely useful to know.

List installed patches:
– AIX: instfix, oslevel
– FreeBSD: uname
– HP-UX: swlist
– Linux: rpm, dpkg
– Solaris: showrev

Install packages:
– AIX: smitty, rpm, installp
– FreeBSD: pkg_add, portinstall, sysinstall
– HP-UX: swinstall
– Linux: rpm, yum, apt, yast, etc.
– Solaris: pkgadd

As you can see, things vary immensely between the Unix variants. Even within all of Linux you can easily find yourself lost. Google is a friend to all sysadmins, but too often the conceptual questions go unanswered. Here’s a general rule of thumb, and something I’ve seen done incorrectly too many times: if you see a configuration file in /etc/, say syslog.conf, and there is an accompanying syslog.d directory, you are not supposed to edit the syslog.conf file directly. The same goes for pam.conf and pam.d. Each service will have their own file within the .d directory, and that is where they are configured.

The .d directory example is mostly applicable to Linux, but be sure to pay attention when you see similar multi-config layouts anywhere else. Future sysadmins using the system will thank you if the OS’s conventions are followed and it’s easy to identify customizations. It also means that your changes aren’t likely to be stomped over by updates.


Networking 101: Subnetting – Slice Up 32-bits

Posted: February 20th, 2010 | Author: | Filed under: Networking 101 | Tags: , , , , | No Comments »

Welcome to networking 101, edition two. This time around we’ll learn about subnets and CIDR, hopefully in a more manageable manner than some books present it.

But first, let’s get one thing straight: there is no Class in subnetting. In the olden days, there was Class A, B, and C networks. These could only be divided up into equal parts so VLSM, or Variable Length Subnet Masks, were introduced. The old Class C was a /24, B was a /16, and A was a /8. That’s all you need to know about Classes. They don’t exist anymore.

An IP address consists of a host and a network portion. Coupled with a subnet mask, you can determine which part is the subnet, how large the network is, and where the network begins. Operating systems need to know this information in order to determine what IP addresses are on the local subnet and which addresses belong to the outside world and require a router to reach. Neighboring routers also need to know how large the subnet is, so they can send only applicable traffic that direction. Divisions between host and network portions of an address are completely determined by the subnet mask.

Classless Internet Domain Routing (CIDR), pronounced “cider,” represents addresses using the network/mask style. What this really means is that an IP address/mask combo tells you a lot of information:

network part / host part

The above string of 32-bits represents a /16 network, since 16 bits are masked.

Throughout these examples (and in the real world), certain subnet masks are referred to repeatedly. They are not special in any way; subnetting is a simple string of 32 bits, masked by any number of bits. It is, however, helpful for memorizing and visualizing things to start with a commonly used netmask, like the /24, and work from there.

Let’s take a look at a standard subnetting table, with a little bit different information:

Subnet mask bits

Number of /24 subnets

Number of addresses

Bits stolen

































Because of the wonders of binary, it works out that a /31 has two IP addresses available. Imagine the subnet: If we picture that in binary, it looks like:

00000010.00000010.00000010.00000000 (
11111111.11111111.11111111.11111110 (31)

The mask is “masking” the used bits, meaning that the bits are used up for network identification. The number of host bits available for tweaking is equal to one. It can be a 0 or a 1. This results in two available IP addresses, just like the table shows. Also, for each additional bit used in the netmask (stolen from the network portion), you can see that the number of available addresses gets cut in half.

Let’s figure out the broadcast address, network address, and netmask for The netmask is simple: that’s (26 bits of mask means 6 bits for hosts, 2^6 is 64, and 255-64 is 192). You can find subnetting tables online that will list all of this information for you, but we’re more interested in teaching people how to understand what’s happening. The netmask tells you immediately that the only part of the address we need to worry about is the last byte: the broadcast address and network address will both start with 192.168.0.

Figuring out the last byte is a lot like subnetting a /24 network, but you don’t even need to think about that, if it doesn’t help you. Each /26 network has 64 hosts. The networks run from .0 to .64, .65 to .128, .128 to .192, and from .192 to .256. Our address,, falls into the .192 to .256 netblock. So the network address is And the broadcast address is even simpler: 192 is 11000000 in binary. Take the last six bits (the bits turned “off” by the netmask), turn them “on”, and what do you get? To see if you got this right, now compute the network address and broadcast address for (Network address:; broadcast

It can be hard to visualize these things at first, and it helps to start with making a table. If you calculated that you wanted subnets with six hosts in each of them, (eight, including the network and broadcast address that can’t be used) then you can start making the table. The following is,, and the final subnet of

Subnet Number

Network Address

First IP

Last IP

Broadcast Address





In reality, you’re much more likely to stumble upon a network where there’s three /26’s and the final /26 is divided up into two /27’s. Being able to create the above table mentally will make things much easier.

That’s really all you need to know. It gets a little trickier with larger subnets in the /16 to /24 range, but the principal is the same. It’s 32 bits and a mask. Do, however, realize that there are certain restrictions governing the use of subnets. We cannot allocate a /26 starting with If we utter the IP/mask of to most operating systems, they will just assume we meant This is because the /26 space requires 64 addresses, and they must start at a natural bit boundary for the given mask. In the above table, what would mean? It means you meant to say

Those tricky ones do demand a quick example. Remember how the number of IP addresses in a subnet gets halved when you take another bit from the network side to create a larger mask? The same concept works in reverse. If we have a /25 that holds 128 hosts, and steal a bit from the host (netmask) portion, we now have a /24 that holds 256. Google for a “subnet table” to see the relationship between netmasks and network sizes all at once. If a /16 holds 65536 addresses, a /17 holds half as many, and a /15 holds twice as many. It’s tremendously exciting! Practice, practice, practice. That’s what it takes to understand how this works. Don’t forget, you can always fall back to counting bits.

The next step, should you want to understand more about subnets, is to read up on some routing protocols. We’ll cover some of them soon, but in the next installment of Networking 101, we’re starting our trip up the OSI model.

No Comments yet... be the first »