Follow me on Twitter:

Managing Virtual Machine and Cloud Sprawl

Posted: February 15th, 2010 | Author: | Filed under: IT Management, Linux / Unix | Tags: , , | No Comments »

Virtualization (in the cloud or locally) is great; that much we can all agree on. Virtual machines (VMs) can tend to grow out of control, however, now that it’s so easy to create them. This should not be all that surprising, but many small to medium businesses are also dabbling in VMs, and they are suddenly overwhelmed by the VM growth.

Each VM is another server that an administrator must manage. Security updates must be applied and global configuration changes now need to be propagated to all these new machines. While it’s easy to create 3-4 (or more) servers on one physical piece of hardware, you’ll certainly struggle if you aren’t already set up to scale.

Unfettered Growth
The number of physical machines in a small company may drop dramatically; maybe 40%, when virtualization is implemented. Unfortunately, the number of OS instances will generally increase by two-fold or more at the same time. The power and cooling savings are realized, as was promised by virtualization, but taking 20 servers to 12 servers, for example, will means you may soon have 40 OS instances to manage.

You Need It

Puppet, from Reductive Labs

The reasons for VM proliferation depend on your culture, but the most common reason is that delegating control of an entire OS is easier than managing an application for customers. IT customer, be they engineers, application developers, or smaller IT units within an organization, frequently need more access then cenral IT is willing to give. The easy solution: give them a server of their own. Test environments, too, are best served by virtual machines.

To keep hardware (and power and cooling) costs down, many companies implement policies about the implementation of new services. New applications and servers need to be run on VMs first, unless it’s really requires its own server. Policies such as these are good, in that they limit wastefulness, but they do tend to exacerbate VM sprawl.

Sprawl aside; it’s worth noting that higher utilization levels on your servers does not mean that they’ll use an appreciably larger amount of power. In fact, the power savings claims are really true, and can be even greater if your utilization is low and you use VirtualCenter’s power management features. VMWare can migrate VMs to fewer servers if utilization isn’t high enough, and actually power off unnecessary servers. This works best with Dell hardware, but other large vendors are supported as well. Imagine: all your VMs migrating to a few blades in a blade server during the nighttime, and then as utilization increases during the day, blades quickly boot up and take the load as needed. Granted, I don’t personally know any enterprise environments that are brave enough to try it yet, but in theory the concept is wonderful.

Something magical happens when a company grows to around 50 operating systems. It’s too many to manage by simply logging in and running commands, so people start to write scripts. In Windows land, if it hasn’t already happened, you must implement Active Directory. For the Unix/Linux servers, configuration management becomes even more important. Writing a script that SSH’s to each server and runs a command doesn’t scale, no matter how hard people want it to. You need a real configuration management system (such as puppet or cfengine) to ensure that servers are configured exactly how you want, and that they will remain that way.

If you already operate in a large environment with good automated installations and configuration management systems, chances are scaling 100-fold won’t be a problem. Barring scaling issues with the management software itseld, that is. A good network-booting deployment system is only half the battle, because every server isn’t going to be configured identically. If you’re “doing it right,” you should be able to arbitrarily reinstall any server, walk away, and know that it’ll come back up patched and running all the services it’s supposed to. Servers, or rather the OS that runs on them, should be truly disposable.

Management of a “golden image” is promised by VMWare, probably because ITIL mentions it, but it doesn’t really help in practice. You have to create your images (somehow). There’s no mechanism to update a golden image with security patches and apply them to existing systems; you’ll generally have to reinstall the OS instances. And that’s what you should do periodically, but without some kind of configuration management system, you’ll also be manually installing and configuring the services that the VMs used to provide in order to restore service functionality.

VM growth, therefore, is no different from server growth. It may be easier and cheaper, but from the OS management viewpoint, you’re doing the same thing. Likewise, the availability of your services is also in danger. Running five VMs on a single piece of hardware means that a hardware failure takes out five servers instead of one. VMWare and Xen can both be clustered and run from shared storage, such that a hardware failure will result in the VMs immediately (instantly, even) being migrated to other servers. The problem is that VMotion requires the most expensive VMWare license, and a VirtualCenter server. Shared storage isn’t as big of an issues these days with iSCSI, but its still another aspect that must be configured. We’ll cover this issue in-depth in a future article, focusing on Xen and RHEL Clustering Services.

The point is: dealing with VM sprawl is no different than dealing with scaling up to support more physical servers. Use whatever mechanisms are available on your given platforms, and “do it right.” A VM is, and always will be, just another server.

No Comments yet... be the first »

Manage Devices and Configurations with Cisco SDM

Posted: February 15th, 2010 | Author: | Filed under: Networking, Security | Tags: , , | No Comments »

Ever wanted to make something “just work” in a secure and reliable way? We, too, have often thought that common configurations should just be selectable. The Cisco Security Device Manager(SDM) is a Java-based Web application for managing Cisco devices. It implements many management features aside from just security-related tasks, and it’s quite interesting. In this article we’ll explain what it can do, and why you might want to take it for a test drive.

Network admins can use SDM to generate Cisco TAC approved configurations with the click of a few buttons. It’s not just limited to simple configurations either. Some tricky configuration tasks such as QoS and VPNs also become easier with the SDM because it ensure that configuration errors don’t exist. In short, you can deploy new devices and services much quicker by using the SDM.

As the name implies, SDM also intently focuses on security. A feature called “one-click lockdown” will set your router up as Cisco recommends—a good starting point for new routers. Also, the security audit function of the SDM will check your configuration and offer up a surprisingly large set of recommendations for hardening security. Many are things that most administrators don’t worry about, but with the SDM you can easily click “fix it” for each item after reading a description. There’s no reason to leave any possible vulnerability open when you have a quick, easy GUI manager pointing out what should change.

Cisco SDM user interface

The SDM is also a management console that gives you a real-time look at your device. It provides a nice interface for viewing system logs, firewall logs, and even real-time performance statistics. You probably already gather performance data via SNMP for historical charting, but being able to see the real-time information while you’re logged into the device manager, where you can also make changes to the configuration, is quite convenient.

SDM is available for most IOS-based routers running 12.2 and above. It is install by downloading a zip file from Cisco and copying it to the router’s flash memory. It’s then accessed from your Web browser (Firefox or IE required, as well as certain Java versions).

Making it Work

First, we must point out that using the SDM requires that you enable the HTTP server on your device. Yes, most Cisco security holes involve the Web server, and yes, a Web spider can easily DoS your router if it starts crawling Web pages and runs it out of RAM. Fortunately, both of these are negligible if you don’t allow access to the Web server from external networks. So first things first, enable: ip http secure-server, then configure ACLs to limit access properly.

After unzipping the file downloaded from Cisco, you can browse to: https://$server/flash/sdm.shtml

Then, login with a highly privileged account (level 15 is required). Up comes the Java applet, and you’re in! It couldn’t be easier than that.


At the top, you’ll see things like Wizard, Advanced, and Monitor. The left had side lists things you can do in Wizard mode, and includes things such as VPN, Firewall, and LAN configuration options.

At the top you’ll also see a “deliver” button, which is another way of saying “commit.” All changes made within the SDM are committed to flash and merged into the running configuration when deliver is clicked.

Various configuration menus exist, most of which make the task at hand slightly easier. For the advanced administrator, it means you can just select options quickly without remembering the specific syntax. More junior admins can make previously confusing concepts work with little effort as well, and then look at the configuration that was generated.

The neatest feature is the security audit. When run, it will gather information about your device and then provide a list of problems. A nice “fix it” check box next to each item can be clicked, or you can elect to choose “fix all.” Beware that Cisco’s idea of security is basically very locked down. Selecting “fix all,” for example, will disable SNMP. It’s true that exposing SNMP to the external world is unwise, but you really do need it enabled for internal access.

You can also configure ACLs and interface parameters from within the GUI. Interfaces can be configured completely via the SDM, and the really nice part is that it lists all available setting for the particular interface. You’ll see check boxes for every option, along with a nice description of each option. ACLs can also be configured, and the GUI presents a nice view of which services will be allowed, and in which direction, on each interface.

In advanced mode, you can easily change many things, including OSPF and BGP settings. It’s just a matter of a few clicks to add another OSPF process ID or add another network to an existing one. Being able to see networks each OSPF process advertises and configure passive interfaces in a single well laid out window is very exciting.

In Monitor mode, you can see which interfaces are down, how much CPU is being utilized, and how much RAM is being taken up by which processes. Very useful information, sure to put a smile on your face the first time you see it.

The SDM does not support everything you’d want to do on a router, but the majority of common tasks are covered. It’s definitely a time-saver, learning tool, and convenience crutch all in one. Don’t feel bad using the SDM; convenience always outweighs prestige, assuming you can do it via the command line too. Enable the “show changes before delivering config” option to see what commands the SDM is about to run, and you’ll avoid surprises and possibly learn something at the same time.

No Comments yet... be the first »