CiRBA Blog

Virtual Fit for Purpose

Having just emerged from some serious white paper writing it seems like a great time for a new post.  We released three new papers last week, and before I put my pen down I will keep the momentum going for a few more paragraphs so I can chime in on what we, and the market, have been up to.  And I will avoid the urge to make lame excuses as to why it has been so long since I have posted - having firmly established a pattern of infrequent blog posts in the past I can safely ride along on a finely crafted wave of low expectations. I should also mention that two of my colleagues - Gerry Smith and Chuck Tatham - will be joining me in posting to this blog to share their insights from now on.

The three new papers cover some interesting topics: one is a total update of our virtualization paper (which was over one year old), one is a discussion of virtualizing applications onto mainframes, and one is a smaller paper on advanced workload analysis.  The last two papers cover features being released in version 4.6 of our product, which we announced April 22. 

The new release has some interesting features around personality-based benchmarking and contention risk analysis.  Although this all sounds wonderfully esoteric, the new capabilities actually serve two simple purposes: to understanding what a workload will look like when it is virtualized, and to determine how high to stack workloads before the risk becomes unacceptable.  Neither of these topics is trivial, particularly when cross-platform transitions are involved (hence the advanced benchmarking) or when stacking ratios are high (hence the risk analysis).  And both of these conditions seem to be arising more and more.

Which brings me to the market.  Vendors seem to be releasing bigger and bigger systems, and in many cases it makes a lot of sense to use larger, vertically scaled nodes over smaller “commodity” servers.  I’m sure HP’s new DL785 systems will swallow up workloads like they’re going out of style, and this model joins existing offerings from IBM, Unisys and Sun that pack a surprising amount of horsepower in a single x86-based system.  The extreme end of this trend is, of course, the mainframe platform, and we also see a surprising amount of interest in understanding what applications will look like on mainframes.  Perhaps even more interesting, this demand is coming both from organizations that want to use their spare capacity as well as shops that currently have no mainframe footprint at all.  Interesting times indeed.

Which makes the aforementioned features very important.  When analyzing workloads between platforms simple MHz-based estimations of relative performance are out the window.  It becomes all about the workload personalities, and the relative capabilities of each platform to host a particular type of workload.  For example, if you are calculating PI to thousands of digits then your best bet would probably be to run it in a screaming x86 or RISC-based system with a high per-core compute performance.  If, on the other hand, you are analyzing an OLTP database workload, the relative performance of a mainframe may win the day.  These types of decisions are becoming more and more common, and many large organizations are undertaking “fit for purpose” projects in order to figure out what their chosen platform is for each type of workload.

This fit for purpose applies not just to the individual servers, but also to the types of virtual pools and clusters being built.  Critical, transaction-oriented workloads will often require a different kind of resource pool than non-critical or batch-oriented workloads.  Building different pools to different performance specifications allows IT organizations to provide multiple hosting options to internal and external customers, and resources pools that are designed to run critical applications may be more expensive to be hosted in than lower-spec batch pools.  This variety of options essentially amounts to virtual fit for purpose, and the ability to create such infrastructure and offer multi-tier hosting options to end consumers will certainly spell doom for one size fits all virtualization planning.

I would go on but I would be repeating words that are written elsewhere.  Every topic has a best document to be in, and if this discussion were any longer it would no longer be suitable for a blog posting.  Think of it as literary fit for purpose…

Evolution of the Virtualization Ecosystem

It’s been a while since my last post – we got busy in the period leading up to VMworld and the dust is only now settling.  The good news is that it was a great show, and aside from all the intriguing conversations we had we also managed to walk away with the Best of VMworld Gold Award for Capacity Planning and Consolidation.  Since gold is about the only color we don’t have in our analysis maps, the award was a great addition to our booth :)

But enough about us – let’s talk about everyone else.  I think it would have been difficult to not notice the sheer number of vendors focusing on this space, and the fact that it is becoming a market unto itself.  There were solutions on display that seek to add value from every conceivable angle, and any shortcomings in VMware’s offering were being filled by multiple vendors.  If one takes the view that the virtualization game will be won or lost based on the ecosystem, not the hypervisor, then this bodes well for VMware.

If, however, one looks at the sustainability of this ecosystem then it paints a somewhat different picture.  The announcement of ESX 3i, which eliminates the Linux service console, undoubtedly sent a shockwave through the vendors that rely on that component to make their solutions work.  Any vendor running scripts on the service console or relying on access beyond the APIs will be in for hard times if VMware continues in this direction.  And, unfortunately for them, VMware should continue in this direction, as any product that doesn’t restrict access to approved, maintainable interfaces will become untenable in the future.

Which brings all of this back to that familiar argument about the downside of de-facto monopolies.  Even if your product uses the published APIs, your future is dependent on them.  And with one dominant player in this particular market, that creates a dangerous situation for ISVs.  We at CiRBA sleep at night because we are not dependent on this level of access, and even more so because we span all virtualization technologies and can even help provide mobility between them (which I’m sure helps our customers sleep at night as well).  But for other elements of the evolving ecosystem the story is much like that in nature: a little more diversity would probably be a good thing...

Hold the Fudge Please

For anyone getting serious about virtualization, one of the first questions that comes up is how to account for the overhead introduced by the virtualization technology being used. Any time you place a layer of abstraction between resource demand and supply there is a high likelihood that some overhead will result.

When we look at common server virtualization technologies there is significant variability in the level of overhead.  Above-kernel technologies such as Solaris Zones are based on a containment model, which is fairly efficient since the device drivers and scheduler activity are controlled, not virtualized. On the other hand, below kernel technologies such as VMware tend to replace what would have been a fairly direct link to the hardware with virtual device drivers, thus generating CPU activity whenever I/O occurs. Paravirtualization technologies (like Xen) and partitioning technologies (like LPARs) have different characteristics yet again, with LPARs having some very sophisticated I/O virtualization capabilities

Of course there is currently a lot of focus on VMware, and how to determine how high you can safely pile the workloads.  Because it is easy to do, it is a common practice to use simple fudge factors to account for overhead when stacking workloads. Overhead numbers in the 15% range are common, and fudge factors upwards of 30% are not unheard of in some situations. Of course, this is like surgery with a sledgehammer, and is no replacement for a proper model of overhead.

To properly analyze this, let’s consider a workload pattern for a physical server:

This is a server that uses spends most of the day between 0 and 25% utilization and peaks up as high as 50% late in the day. (This is a quartile-based view, which I won’t get into too much here.) To determine what this will look like in a VMware environment, the next step is to look at the I/O rates on this server:

From these curves we can see that this server is not I/O intensive for most of the day but has some reasonably sustained I/O activity after 7pm. In VMware, this I/O will pass through virtual drivers, which effectively pass the operations on to the underlying device drivers, but generate CPU while doing so. We can therefore use these curves to estimate the true overhead on the CPU, and attribute this overhead to the corresponding times of day.

The result of using this approach is a virtual CPU curve that looks like this:

When compared against the original curve it is clear to see that the overhead is skewed heavily toward the periods of significant I/O activity, and that periods of low I/O do not contribute disproportionately to the overall utilization levels.  (This curve accounts for overhead caused by network I/O, disk I/O and general scheduler overhead, and is exaggerated to help illustrate the effect.) 

When applied in practice this helps bring an additional level of accuracy to virtualization analysis and helps prevent layering fudge factors on top of fudge factors, which tends to lead to sticky situations that can undermine your attempts to trim down your data center. 

An Ounce of Analysis is Worth a Pound of Memory

Time to roll up our sleeves and talk about memory, and in particular the sharing of memory between virtual machines.  In many virtualization initiatives targeting existing environments the CPU utilization of the target servers is fairly low, particularly in Windows environments.  In an earlier post I went into some of the reasons why this is the case, but this time I want to talk about how this may cause the emphasis of a virtualization project to shift toward memory.

Low CPU utilization often implies high stacking ratios, especially if the workloads aren't overly "peaky" in nature.  But in practice this simply means that something else will become the bottleneck, and because it is uncommon for servers with low utilization to have high I/O rates, the next bottleneck is often memory.  In general, the memory requirement for a process is not related to its activity level unless it is competing with other applications, a situation that is rectified at the OS level by memory paging.  But in most steady-state production servers enough memory is added to prevent paging, thus giving the application a nice big space to spread out and laze around as it does nothing.

When stacking applications together in virtual machines, an opportunity exists to share read-only memory pages between VMs, but this opportunity will be squandered if they are virtualized the wrong way.  Most users of Solaris 10 Zones are familiar with the "sparse root" model and the advantages it gives with respect to memory sharing.  Simply put, the more similar you make each zone the more memory sharing will occur, and this is often a major consideration when determining what apps to put together on the same physical system.

What most people don't know is that VMware does this too, albeit in a different way.  If VMs are sharing identical pages of memory VMware will eventually notice this (using checksums) and start sharing them.  But this will only happen if the VMs are running similar stuff (i.e. the same OS, Rev, service pack, app revs), and if every VM is different then there won't be much to share.  And the difference can be significant - over half the memory in use can be shared with the right configuration, effectively doubling the potential virtualization ratio for a given environment.

Unfortunately most Windows virtualization initiatives ignore this and focus primarily on utilization information.  This tends to create configurations that have relatively low memory sharing potential (since this is not an optimization criteria), and consequently have lower virtualization ratios.  By adding configuration uniformity as a criteria when analyzing servers it can have a surprising effect on the level of virtualization attained, and all it takes is a some foresight in the planning phase.

Are You Virtually Compliant?

In order to help prevent technical anarchy on a grand scale, I find myself frequently pointing out to those undertaking virtualization initiatives that it is not merely a sizing exercise.  First and foremost it is an exercise in constructing a coherent infrastructure that puts the right things together and keeps the right things apart, especially when dealing with production systems.  You don't build a sophisticated IT infrastructure over many years to simply have it "randomized" into VMs based on utilization.

Some counter by asking "Why not?  If the systems are in their own VMs then what difference does it make?"  Rather than go into the details of how problematic it can be to combine systems with different availability levels, different DR strategies, etc., I find it easiest to think of things from a security perspective.

System-level security has traditionally been based on the premise that you cannot see the local storage without first authenticating with the system at the OS level (i.e. logging on).  Network-based storage authenticates at the system level and thus also prevents one from accessing it without being "on the system".  This access control is like a castle wall that protects everything inside it, and is the first line of defense in any IT security strategy.

Virtualization is a different paradigm, and as these things go, new paradigms often upset existing assumptions.  Most virtualization technologies store their VM images in files, and this encapsulation enables many of the benefits of virtualization by providing portability, state management, interoperability and a whole list of other advantages.  But it also means that those with access to these files can bypass all authentication mechanisms on that system; by having a complete file (and memory) image of a virtual machine, you can peer inside it with little effort.  The castle wall is suddenly made of very thin paper.

In fact, most VM vendors provide utilities to "mount" a VM image as a disk, providing convenient access to its contents.  This means that the old fear of someone "walking off with a disk" has now become a fear of someone "copying a file", which is considerably easier to do.  But more to the point of this topic, it means that any administrator or user with access to these images becomes a "super super" user, with access to the contents of systems that they may not otherwise be privy to.  That fact that a single user exists that can access (and modify) the data of any app on a virtual cluster is enough to kill pretty much any compliance audit.

There are, of course, ways to prevent this.  Some vendors support encryption of the VM images, although this will undoubtedly cause performance to tank.  Some provide affinity and anti-affinity rules to influence the movement of images within a pool of physical systems, but offer no practical means to establish and maintain these rules.  (Not to mention the fact that they are already in the same pool may make it too late to achieve true isolation.)

The best solution, in my opinion, is to simply avoid combining systems that should never go together.  Seems simple enough - this is what you spent the last 10 years doing, so why stop now.  And the best way to accomplish this is to establish in advance the business constraints that govern a virtualization project and ensure that these constraints are honoured throughout the process.  And then look at sizing...

Utilization v Moore's Law

A lot of organizations look to consolidation and virtualization as a way to increase utilization.  While this is certainly true, there is another very interesting way to look at it: consolidation and virtualization are necessary tools in the battle against falling utilization.

The reason?  When it comes time to refresh servers, you simply cannot buy gear that is nearly as slow as the stuff you are replacing.  Moore's Law dictates that the number of transistors on a chip will double every 18 months, which roughly translates into the doubling of compute speed.  This has certain limits, and lately we have seen a shift away from increasing speeds and toward "multi-core" strategies, but the net effect is roughly the same.

This means that on a 3 year lease cycle you will be replacing systems with ones that are roughly 4 times faster.  The increase in app response is no doubt appreciated by end users, but from a utilization perspective this means that the utilization of these systems drops significantly with every hardware refresh. 

When viewed on aggregate this has interesting implications.  If we approximate Moore's Law to say that speeds increase by about 58% per year, and we consider a pool of 300 servers that have been purchased over 3 years, then the aggregate compute capacity would be:

100+158+252 = 510 units of power (realtive to the power of the year 1 servers)

If on the 4th year you refresh the first 100 with servers that are 4x more powerful then the pool becomes:

158+252+400 = 810 units of power

This is, interestingly, a 58% increase in the compute capacity over the previous year, meaning that upgrading only a portion of a server pool will still track to Moore's law on the larger set.

More importantly, however, it means that you need to achieve at least a 4:1 consolidation ratio on any new servers just to keep your utilization levels stable (ignoring application growth).  If you are replacing 6 year old servers then you need a 16:1 ratio, and for 9 year old gear you would have to achieve a 64:1 ratio just to keep things the same.  Needless to say this can become quite challenging.

The upshot of all this is that doing nothing will typically cause utilization levels to drop over time, and the single-digit utilization levels that many organizations experience (particularly on Wintel servers) are undoubtedly contributed to by this effect.  It also means that anyone looking to use consolidation or virtualization to increase utilization levels better be very careful about the expectations that they set.  The incessant shrinkage of transistors is working against you...

In Praise of Mainframes

I've had a number of discussions recently regarding the sizing of servers for use in virtualization initiatives.  On one hand, keeping the cpu count low has the advantage of keeping software costs down, and allows an infrastructure to be constructed of low-cost, commoditized servers.  On the other hand, using larger servers has an advantage in that it provides better economies of scale when combining workloads, and also provides higher headroom for peak demands.  This has understandably been a hot topic of late.

When I try to put this in perspective I can't help but turn my thoughts to the mainframe world, where virtualization and proper capacity planning have been mainstays for decades.  This is the epitome of the scale up model, and few if any mainframe environments are suffering from the sprawl and underutilization that we are seeing in the opens systems world.

And there is, in my opinion, a significant underlying reason for this. It is what I call reverence for the platform, and, put simply, it is the fact that people respect big, expensive systems more than small, cheap ones. Why do we rarely see VM sprawl on mainframes? Because people don't mess with them.

This respect is important.  How important? Just watch any movie.

Nobody goes on a quest to seek advice from "blade rack"

Nobody tries to defeat the enemy by knocking out their "midrange"

Nobody crawls through air ducts to gain access to "file and print"

They go to Mainframe

They go to the biggest, most vertically scaled thing they can find.

In fact, the only prominent example of a horizontally scaled technology in a movie that I can think of is the Borg. But it was evil – more like a grid with attitude. And that may have been a TV show.

So next time you are sizing servers for virtualization be sure to pay some attention to the ultimate non-technical consideration: reverence for the platform.  And who knows - if a bunch of actors come walking through the server room they may just try wreck your stuff instead…

Virtualization: Technical Challenge or Business Challenge?

Many view virtualization as a technical challenge, and focus on determining how many applications can be combined together on as little hardware as possible.  I would challenge this notion, and say that it is instead a business challenge, which largely revolves around determining which applications should be placed together from a business perspective.

The difference is important, and any confusion between the two can be explained quite simply.  Many virtualization projects start out in the lab, which is relatively free of business constraints.  This tends to cause early virtualization investigations to focus on the technical aspects of virtualization, and the criteria used to combine applications is almost universally driven by workload considerations.

Once you go beyond the lab, however, the "real world" production environments are riddled with constraints, including change restrictions, availability commitments, business owner relationships, compliance rules, etc..  To charge into such environments using workload as the primary consideration for virtualization is to run the risk of constructing a completely dysfunctional infrastructure.  Consider a simple case: by combining applications that have non-overlapping maintenance windows onto a single physical server you may never be able to do hardware maintenance again.

This is but one example, and to list all the business constraints that govern production environments would be daunting.  But for a start I always recommend incuding the following basic constraints when analyzing production environments:

• physical location
• operational environment
• application tier
• availabilty target

There are many others, but these are a good start.  It is rare that one would seek to combine a production server and its DR counterpart on the same server, and combining servers from different application tiers on the same system can create correlated workloads that slow transaction response times.  Location is also usually a big constraint, although this is often lifted if the virtualization is part of a larger data center consolidation.  Keeping availability levels consistent between VMs is always wise, and allows you to size and spec hardware appropriately, particularly wrt redundancy levels and UPS requirements.

These are specific examples, but are part of a general trend.  As virtualization moves out of the lab and into production, the challenge becomes less and less about the the specific virtualization technology and how to stack workloads on it, and more about the target environment and the proper consideration of all the constraints within it.

Branching Late

We met with a number of technology analysts in the last week to discuss version 4.0 of our product and one conversation stands out as particularly interesting in my mind. The discussion centered on how to arrive at the best strategies for consolidation and/or virtualization for a particular set of servers that are part of a larger environment.

The analysts were describing cases they've seen where a strategy was decided upon early, and that this strategy dictated the end result and prevented the exploration of other options. They referred to this as "branching early", where a path is committed to early in the process, even if it not optimal. An example would be to say "we're going to take those servers at that location and virtualize them all" without proper consideration of what is on the servers or whether location is a true constraint on the virtualization process.

I contrasted this with the preferred approach of "branching late" and holding off on committing to a path for a set of servers until you having performed various types of what-if analysis. For example, the servers in question can first be analyzed for database and app server consolidation potential, which helps avoid putting I/O intensive apps on virtualized platforms (and also saves on expensive VM licenses). They can also be kept in a larger pool and a location constraint can be applied or lifted, allowing exploration of other options by effectively allowing an analyst to say "now let's see how much consolidation we can do if we allow ourselves to go across locations."

By keeping target servers grouped together and using constraint models to analyze the possibilities, rather than breaking them up along specific lines and committing to a strategy prematurely, this "late branching" technique is a much more effective way to determine the "optimal transfer function" for a set of servers.

We met with a number of technology analysts in the last week to discuss version 4.0 of our product and one conversation stands out as particularly interesting in my mind. The discussion centered on how to arrive at the best strategies for consolidation and/or virtualization for a particular set of servers that are part of a larger environment.

The analysts were describing cases they've seen where a strategy was decided upon early, and that this strategy dictated the end result and prevented the exploration of other options. They referred to this as "branching early", where a path is committed to early in the process, even if it not optimal. An example would be to say "we're going to take those servers at that location and virtualize them all" without proper consideration of what is on the servers or whether location is a true constraint on the virtualization process.

I contrasted this with the preferred approach of "branching late" and holding off on committing to a path for a set of servers until you having performed various types of what-if analysis. For example, the servers in question can first be analyzed for database and app server consolidation potential, which helps avoid putting I/O intensive apps on virtualized platforms (and also saves on expensive VM licenses). They can also be kept in a larger pool and a location constraint can be applied or lifted, allowing exploration of other options by effectively allowing an analyst to say "now let's see how much consolidation we can do if we allow ourselves to go across locations."

By keeping target servers grouped together and using constraint models to analyze the possibilities, rather than breaking them up along specific lines and committing to a strategy prematurely, this "late branching" technique is a much more effective way to determine the "optimal transfer function" for a set of servers.

Inaugural Post

Welcome to my new blog. 

It is going live as part of an update to our website, and I plan to use it to post observations and commentary on data center management and optimization, with a particular focus on server consolidation and virtualization.  People are always saying to me "you should write that down....it's an interesting perspective..."  Well now I can.

January 16th marks an important date for our company as we ship version 4.0 of our Data Center Intelligence product.  We think this highly unique and powerful software will change the way server consolidation and virtualization analysis will be done in many data centers. To coincide with the release I decided to join the Blogging ranks.

Be sure to visit again...