Follow me on Twitter:

Are Cisco Flex Links the End of STP?

Posted: June 18th, 2010 | Author: | Filed under: Networking | Tags: , , , | 2 Comments »

Cisco Flex Links gives network operators a simple, reliable, and more scalable method of layer 2 redundancy. The Spanning Tree Protocol (STP) is not destined for the scrap bin, but it will certainly fall out of favor with many enterprise networks.

Flex Links are a pair of layer 2 interfaces configured to act as a backup of each other. Configuring Flex Links is very simple, but it’s a manual process. Spanning tree can configure itself if you just enable it, albeit likely a sub-optimal configuration, but a working one nonetheless. Flex Links, on the other hand, require manual setup and layout of your layer 2 network. If you don’t want to leave anything to chance, then Flex Links are preferred over STP.

The benefits of FlexLinks include:

  • simplicity, which equals stability.
  • instant failover.
  • rudimentary load balancing capabilities, so one link isn’t wastefully idle.
  • load balancing works across switches in a stack, including port channels.

Flex Links’ primary operating mode is just like spanning tree: one on, one off. With per-VLAN spanning tree, a trunk port can have some VLANs enabled and some blocked at the same time, so on the surface it seems that STP is superior. In reality, you can configure Flex Links to load balance VLANs, and we’ll show you how shortly.


Conceptually, you configure Flex Links by telling one link it’s the active link, and another that it’s the backup of that

Flex Links Design Map

primary (active) one. Without configuring VLAN load balancing, it will completely disable the backup, and if the active link goes down the backup will take over.

For example, to configure port gi1/0/1 as a active link, and gi1/0/2 as the backup, you’d run:

Switch# configure terminal
Switch(conf)# interface gigabitethernet1/0/1
Switch(conf-if)# switchport backup interface gigabitethernet1/0/2

That’s all there is to configuring the basic mode, which gets you failover but no load balancing. Before talking about load balancing, let’s take a look at preemption and “mac address-table move update.”


Preemption, that is, the preferred port for forwarding traffic, is also configurable. This is most often used in combination with multiple links that have differing bandwidth capacities. If you wish to ensure that port 1, a primary port that has more bandwidth, will return to the active link when it comes back up, you would set:  interface preemption mode bandwidth andswitchport backup interface preemption delay. The delay is used to set the amount of time (in seconds) to wait before allowing port 1 to preempt port 2 and begin taking over traffic again.

MAC Address-Table Move Update

Enabling the MAC address-table move update feature allows for rapid convergence when a primary link goes down and the backup takes over traffic forwarding duties. Without this feature enabled, neighboring switches may continue to forward traffic for a short time to a dead port, since they have learned MAC addresses associated with that link.

When move update is enabled, the switch containing Flex Links will broadcast an update packet to let other switches know what happened, and they will in turn un-learn that false MAC address mapping.

On the switch with Flex Links, simply configure:

Switch(conf)# mac address-table move update transmit

All switches, including ones with Flex Links, need to receive these updates. This is not enabled by default, so you’ll need to run the following command on all of your devices:

Switch(conf)# mac address-table move update receive

To see the status and verify that “move update” is enabled, run: show mac address-table move update. Checking the status of your Flex Links is much the same: show interfaces [interface-id] switchport backup.

Load Balancing

Flex Links should be configured such that both ports are forwarding traffic at the same time. This way, you get load balancing in addition to redundancy. The limitation is that only one port can be forwarding a single VLAN at a time. If we have VLANs 1-200, we need to choose which VLANs are forwarded primarily through which port. The most simple configuration, ignoring traffic requirements, would be that VLANs 1-100 use port 1, and VLANs 101-200 use port 2.

Before we get into configuring preferred VLANs, let’s talk about multicast. Multicast, of course, becomes an issue with this type of setup. If a port passed an IGMP join, and the switch is part of a multicast group, when the port goes down the switch will no longer be able to receive multicast traffic for that group. The quick fix is to make both Flex Links always be part of learned groups, with the command: switchport backup interface gigabitEthernet 1/0/12 multicast fast-convergence.

Now, on to VLAN load balancing. It is quite easy; just specify which VLANs you prefer on which links:

Switch(config-if)#switchport backup interface gigabitEthernet1/0/2 prefer vlan 101-200.

If you have VLANs 1-200 on the switch, show interfaces switchport backup will show you:

Vlans Preferred on Active Interface: 1-100
Vlans Preferred on Backup Interface: 101-200

If a link goes down, VLANs that are preferred on that interface will be moved to the other link in the pair. Likewise, when a link returns to service, its preferred VLANs are blocked on the backup and returned to the preferred link.

Be sure to run show interfaces switchport backup detail to see the full status, including link speeds, preemption modes, the MAC address-table move update status.

In summary, the simplicity of Flex Links make it a better choice for carrier and core enterprise networks over the ubiquitous spanning tree protocol. Link-level redundancy is had via STP, but with Flex Links you have more control and better load balancing capabilities. This certainly means that it takes longer to configure since you are planning the layer 2 network manually, but when you need a stable no-surprises link-layer network, Flex Links are definitely the way to go.


Networking 101: Understanding Layers

Posted: March 7th, 2010 | Author: | Filed under: Networking 101 | Tags: , , , , | No Comments »

Continuing our journey, it’s time to take a trip up the OSI Reference Model, and learn what this mysterious thing is all about. The network stack is of great significance, but not so much that it’s the first thing you should learn. The networking 101 series has waited to ensue the “layers” discussion for good reason. Many so-called networking classes will start by teaching you to memorize the name of every layer and every protocol contained within this model. Don’t do that. Do realize that layers 5 and 6 can be completely ignored, though.

The International Standards Organization (ISO) developed the OSI (Open Systems Interconnection) model. It divides network communication into seven layers. Layers 1-4 are considered the lower layers, and mostly concern themselves with moving data around. Layers 5-7, the upper layers, contain application-level data. Networks operate on one basic principle: “pass it on.” Each layer takes care of a very specific job, and then passes the data onto the next layer.

The physical layer, layer 1, is too often ignored in a classroom setting. It may seem simple, but there are aspects of the first layer that oftentimes demand significant attention. Layer one is simply wiring, fiber, network cards, and anything else that is used to make two network devices communicate. Even a carrier pigeon would be considered layer one gear (see RFC 1149). Network troubleshooting will often lead to a layer one issue. We can’t forget the legendary story of CAT5 strung across the floor, and an office chair periodically rolling over it leading to spotty network connectivity. Sadly, this type of problem exists quite frequently, and takes the longest to troubleshoot.

Layer two is Ethernet, among other protocols; we’re keeping this simple, remember. The most important take-away from layer 2 land is that you should understand what a bridge is. Switches, as they’re called nowadays, are bridges. They all operate at layer 2, paying attention only to MAC addresses on Ethernet networks. The common fledgling network admin always seem to mix up layers two and three. If you’re talking about MAC address, switches, or network cards and drivers, you’re in the land of layer 2. Hubs live in layer 1 land, since they are simply electronic devices with zero layer 2 knowledge. Layer two will have it’s own section in Networking 101, so don’t worry about the details for now, just know that layer 2 translates data frames into bits for layer 1 processing.

On the other hand, if you’re talking about an IP address, you’re dealing with layer 3 and “packets” instead of layer 2’s “frames.” IP is part of layer 3, along with some routing protocols, and ARP (Address Resolution Protocol). Everything about routing is handled in layer 3. Addressing and routing is the main goal of this layer.

Layer four, the transport layer, handles messaging. Layer 4 data units are also called packets, but when you’re talking about specific protocols, like TCP, they’re “segments” or “datagrams” in UDP. This layer is responsible for getting the entire message, so it must keep track of fragmentation, out-of-order packets, and other perils. Another way to think of layer 4 is that it provides end-to-end management of communication. Some protocols, like TCP, do a very good job of making sure the communication is reliable. Some don’t really care if a few packets are lost–UDP is the prime example.

And arriving at layer seven, we wonder what happened to layer 5 and 6. They’re useless. A few applications and protocols live there, but for the understanding of networking issues, talking about these provides zero benefit. Layer 7, our friend, is “everything.” Dubbed the “Application Layer,” layer 7 is simply application-specific. If your program needs a specific format for data, you will invent some format that you expect the data to arrive in, and you’ve just created a layer 7 protocol. SMTP, DNS, FTP, etc, etc are all layer 7 protocols.

The most important thing to learn about the OSI model is what it really represents. Pretend you’re an operating system on a network. Your network card, operating at layers 1 and 2, will notify you when there’s data available. The driver handles the shedding of the layer 2 frame, which reveals a bright, shiny layer 3 packet inside (hopefully). You, as the operating system, will then call your routines for handling layer 3 data. If the data has been passed to you from below, you know that it’s a packet destined for yourself, or it’s a broadcast packet (unless you’re also a router, but never mind that for now). If you decide to keep the packet, you will unwrap it, and reveal a layer 4 packet. If it’s TCP, the TCP subsystem will be called to unwrap and pass the layer 7 data onto the application that’s listening on the port it’s destined for. That’s all!

When it’s time to respond to the other computer on the network, everything happens in reverse. The layer 7 application will ship its data onto the TCP people, who will stick additional headers onto the chunk of data. In this direction, the data gets larger with each progressive step. TCP hands a valid TCP segment onto IP, who give its packet to the Ethernet people, who will hand it off to the driver as a valid Ethernet frame. And then off it goes, across the network. Routers along the way will partially disassemble the packet to get at the layer 3 headers in order to determine where the packet should be shipped. If the destination is on the local Ethernet subnet, the OS will simply ARP for the computer instead of the router, and send it directly to the host.

Grossly simplified, sure; but if you can follow this progression and understand what’s happening to every packet at each stage, you’re just conquered a huge part of understanding networking. Everything gets horribly complex when you start talking about what each protocol actually does. If you are just beginning, please ignore all that stuff until you understand what the complex stuff is trying to accomplish. It makes for a much better learning endeavor! In future Networking 101 articles we will begin the journey up the stack, examining each layer in detail by discussing the common protocols and how they work.

No Comments yet... be the first »

Networking 101: Subnetting – Slice Up 32-bits

Posted: February 20th, 2010 | Author: | Filed under: Networking 101 | Tags: , , , , | No Comments »

Welcome to networking 101, edition two. This time around we’ll learn about subnets and CIDR, hopefully in a more manageable manner than some books present it.

But first, let’s get one thing straight: there is no Class in subnetting. In the olden days, there was Class A, B, and C networks. These could only be divided up into equal parts so VLSM, or Variable Length Subnet Masks, were introduced. The old Class C was a /24, B was a /16, and A was a /8. That’s all you need to know about Classes. They don’t exist anymore.

An IP address consists of a host and a network portion. Coupled with a subnet mask, you can determine which part is the subnet, how large the network is, and where the network begins. Operating systems need to know this information in order to determine what IP addresses are on the local subnet and which addresses belong to the outside world and require a router to reach. Neighboring routers also need to know how large the subnet is, so they can send only applicable traffic that direction. Divisions between host and network portions of an address are completely determined by the subnet mask.

Classless Internet Domain Routing (CIDR), pronounced “cider,” represents addresses using the network/mask style. What this really means is that an IP address/mask combo tells you a lot of information:

network part / host part

The above string of 32-bits represents a /16 network, since 16 bits are masked.

Throughout these examples (and in the real world), certain subnet masks are referred to repeatedly. They are not special in any way; subnetting is a simple string of 32 bits, masked by any number of bits. It is, however, helpful for memorizing and visualizing things to start with a commonly used netmask, like the /24, and work from there.

Let’s take a look at a standard subnetting table, with a little bit different information:

Subnet mask bits

Number of /24 subnets

Number of addresses

Bits stolen

































Because of the wonders of binary, it works out that a /31 has two IP addresses available. Imagine the subnet: If we picture that in binary, it looks like:

00000010.00000010.00000010.00000000 (
11111111.11111111.11111111.11111110 (31)

The mask is “masking” the used bits, meaning that the bits are used up for network identification. The number of host bits available for tweaking is equal to one. It can be a 0 or a 1. This results in two available IP addresses, just like the table shows. Also, for each additional bit used in the netmask (stolen from the network portion), you can see that the number of available addresses gets cut in half.

Let’s figure out the broadcast address, network address, and netmask for The netmask is simple: that’s (26 bits of mask means 6 bits for hosts, 2^6 is 64, and 255-64 is 192). You can find subnetting tables online that will list all of this information for you, but we’re more interested in teaching people how to understand what’s happening. The netmask tells you immediately that the only part of the address we need to worry about is the last byte: the broadcast address and network address will both start with 192.168.0.

Figuring out the last byte is a lot like subnetting a /24 network, but you don’t even need to think about that, if it doesn’t help you. Each /26 network has 64 hosts. The networks run from .0 to .64, .65 to .128, .128 to .192, and from .192 to .256. Our address,, falls into the .192 to .256 netblock. So the network address is And the broadcast address is even simpler: 192 is 11000000 in binary. Take the last six bits (the bits turned “off” by the netmask), turn them “on”, and what do you get? To see if you got this right, now compute the network address and broadcast address for (Network address:; broadcast

It can be hard to visualize these things at first, and it helps to start with making a table. If you calculated that you wanted subnets with six hosts in each of them, (eight, including the network and broadcast address that can’t be used) then you can start making the table. The following is,, and the final subnet of

Subnet Number

Network Address

First IP

Last IP

Broadcast Address





In reality, you’re much more likely to stumble upon a network where there’s three /26’s and the final /26 is divided up into two /27’s. Being able to create the above table mentally will make things much easier.

That’s really all you need to know. It gets a little trickier with larger subnets in the /16 to /24 range, but the principal is the same. It’s 32 bits and a mask. Do, however, realize that there are certain restrictions governing the use of subnets. We cannot allocate a /26 starting with If we utter the IP/mask of to most operating systems, they will just assume we meant This is because the /26 space requires 64 addresses, and they must start at a natural bit boundary for the given mask. In the above table, what would mean? It means you meant to say

Those tricky ones do demand a quick example. Remember how the number of IP addresses in a subnet gets halved when you take another bit from the network side to create a larger mask? The same concept works in reverse. If we have a /25 that holds 128 hosts, and steal a bit from the host (netmask) portion, we now have a /24 that holds 256. Google for a “subnet table” to see the relationship between netmasks and network sizes all at once. If a /16 holds 65536 addresses, a /17 holds half as many, and a /15 holds twice as many. It’s tremendously exciting! Practice, practice, practice. That’s what it takes to understand how this works. Don’t forget, you can always fall back to counting bits.

The next step, should you want to understand more about subnets, is to read up on some routing protocols. We’ll cover some of them soon, but in the next installment of Networking 101, we’re starting our trip up the OSI model.

No Comments yet... be the first »

What the Heck is a TCAM?

Posted: February 16th, 2010 | Author: | Filed under: Networking | Tags: , , , | 1 Comment »

Let’s talk about TCAM hardware, Cisco SDM templates, and try to answer that elusive question: “why do I have to reboot my router to enable certain features, which in turn disables others?”

First, CAM stands for Content Addressable Memory. A CAM is a special type of memory; some would say the opposite of RAM. With normal computer memory (RAM) the operating system provides an address, and receives the data stored at the supplied address. With a CAM, the operating system supplies the data, and the CAM returns a list of addresses where the data is stored, if it finds any. Furthermore, a CAM searches the entire memory in one operation, so it is considerably faster than RAM.

CAMs are very expensive, so they aren’t normally found in PCs. Even router vendors will sometimes skimp, opting to instead implement advanced software-based searching algorithms to plod through RAM. Most commonly, CAMs and TCAMs are found in network processing devices, including Intel IXP cards and various routers or switches. The most commonly implemented CAMs are called binary CAMs. They search only for ones and zeros; a simple operation. MAC address tables in switches commonly get stored inside binary CAMs. You can bet that any

A Renesas TCAM

switch capable of forwarding Ethernet frames at line-speed gigabit is using CAMs for lookups. If they were using RAM, the operating system would have to remember the address where everything is stored. With CAMs, the operating system can find what it needs in a single operation. In this case desired data is the switchport that data should be sent out, based on the given MAC address, i.e. the essence of a MAC table. Some older Cisco switches running CatOS even opted to call this table the cam table, thereby causing great confusion across the land. Bridge table, forwarding table, mac-address table, cam table; it’s all the same.

Finally, a TCAM is a Ternary CAM. This allows the operating system to match a third state, “X.” The X state is a mask, which means you don’t care what it is. This naturally lends itself to networking, since netmasks operate this way. To calculate a subnet address we mask the bits we don’t care about, and then apply the logical AND operation to the rest. Being able to do this in hardware is a great benefit for routers. Additionally, routers can store their entire routing table in these TCAMs, allowing for very quick lookups. A router with routing tables in TCAMs can find the next-hop destination in a single operation every time instead of trying to search through a tree (or other data structure) in RAM.

Hardware can sometimes seem magic, but it isn’t always transparent. When configuring routers most people will run into a situation where enabling a new feature will require that the Cisco SDM (Switching Database Manager) template be changed. This template is actually a method Cisco uses to assign specific applications to specific TCAM resources.

Some routers will allow you to manually specify how much TCAM space you want to allocate to a specific feature. Others aren’t so nice. They make you choose from a few restrictive templates, which allocate the resources automatically based on a few predetermined settings. For example, on the Cisco 3750, we recently wanted to enable policy-based routing (PBR) to implement a layer 3 jail. The basic idea with template-only routers is that you have to choose where you want most of the optimizations, and compromise on the rest.

For this platform, there are four templates: default, routing, PBR, and VLAN. Each of these tries to allow for a bit more resources allocated to the specified task. For policy routing, we’d have to choose “routing” or “PBR,” which in turn limits the amount of unicast MAC addresses that can be held in TCAMs. Likewise, selecting a VLAN template will make PBR impossible, but allow for more VLAN database information to be held in TCAMs. There are always compromises when we need to use more advanced features. Keeping true with the spirit of router operating systems, there’s also some mysterious side-effects when a new template is chosen. On our specific router, if the PBR template is chosen, the router will become unable to support VPN routing/forwarding tables (VRF). The next unsightly gotcha is that with the IOS version that supports IPv6, you cannot even enable PBR. There is no template to allow both policy routing and IPv6.

Perhaps the main idea of TCAM allocation still isn’t clear. Just because, for example, 8K is allocated to routing tables, this doesn’t mean that you can only have a routing table of that size. There’s always the fallback of process switching. Process switching means that everything will be done by the processor instead of in hardware (TCAMs). Processor intervention is not desirable, mostly because it is much slower than hardware lookups. Also, the processor is supposed to be used for things like sending logs to a syslog server and controlling SSH sessions. If a router doing process switching gets really busy, it may be unable to service your console access attempts. Generally speaking, the more expensive the router, the less it will use the processor.

Hardware is finite, and we always need more. More expensive routers don’t always suffer from the constant struggle for TCAMs because they have enough to support most features that currently exist. Unfortunately, most companies won’t want to purchase the latest and greatest router with seemingly endless hardware resources unless they can justify the added cost by showing a current need for them. So, most of us are stuck having to adjust TCAM allocations.

Further reading: an interesting blog from Plixer.

1 Comment »