CiRBA Blog

Hyper-V: Hyper Hype or Hyper Important?

As we get closer to Microsoft’s release of Hyper-V the “which will be better” or “who will win” noise is increasing. VMware is the gorilla with significant traction and in market experience. Microsoft has a history of not being first, but slowly taking massive share of markets it decides are important.  And of course Citrix, a company with perhaps the most virtualization pedigree of the group.

We are seeing more and more articles like this one from SearchServerVirtualization.com that addresses future pricing wars. Depending on the expert or pundit you get wildly different views of which technology is better, whether price matters and if it’s possible to disrupt what VMware has already achieved. One thing is for certain, more choice is better for the market and customers.  Competition keeps pricing sharp, drives more aggressive R&D and ensures an eye to quality customer service and support. 

The wildcard is what kind of mark Microsoft will make this fall.  There is no question that their offerings will be less mature and have fewer features than the VMware offerings.  That said they do have some other major advantages, not the least of which is a server OS footprint that is unrivaled.  This “distribution mechanism” is a massive advantage.  Ultimately customers will be weighing a variety of criteria like features, price and performance.   The big question is, how close does Microsoft have to come to start clawing significant share.  They will earn a certain amount of share just because of their OS position in the market, but how much they take and how fast will be determined by the markets acceptance of the feature differential.  If Hyper-V is 60% as featured and capable as VMware?....70%?   Combined with a price advantage and the association with the Windows server OS should allow Microsoft to take share even with a feature deficit.

CiRBA Version 5.0

Today we announced Version 5.0 of our Data Center Intelligence Product.   The theme around this release is “Dynamic” Data Center Intelligence and it was developed in response to the reality that  “New School data centers” or highly virtualized environments are far more dynamic….it’s what makes virtualization powerful, liberating, cost effective, but also scary.  To maintain service levels and contain risk it is important to keep a different kind of eye on the environment. As our CTO, Andrew Hillier likes to say “spontaneity in the data center is a bad thing” and with workloads decoupled from the physical there is far more dynamism possible in the data center.  Over the past 2 years we have focused on building software that would provide the absolute best, lowest risk answers to our customers and partners as they guide their organizations or clients through sensitive transformations, consolidation and virtualization initiatives. 

Version 5.0 is all about making those optimal answers available continuously…..dynamically.  As organizations move toward taking full advantage of virtualization the “right answers” become an everyday or even every hour necessity.  CiRBA’s rule-based analytics engine can now watch over infrastructure and constantly notify of new opportunities for efficiency, whitespace issues or risks that are discovered in an environment.   

We are looking forward to engaging our customers and partners in the many opportunities made available through Dynamic Data Center Intelligence.

Virtualization: Balancing Risk vs. Reward

One of our customers (Underwriters Laboratories) spoke with CIO Magazine a couple of weeks ago and the resulting story made for an interesting overview of their success in reducing servers, complexity and cost through virtualization.   

One topic that I have found to be increasingly important to IT leadership, but not covered in this article, is balancing the kinds of consolidation results that Underwriters Laboratories identifies, with increases in risk.

It reminded me of a conversation I had some time ago with CiRBA board member and insurance company CIO Mike Woeller.  During our conversation, I passionately described how our software allows IT to more quickly virtualize, consolidate  and become much more efficient and agile.

Mike acknowledged the importance of making efficient use of infrastructure but stated “quite frankly Gerry; it is a distant second to reliability.  I lose my day job when things don’t work”.  It was a good reminder that data centers are so tremendously complex today, that keeping the lights on can be an enormous challenge unto itself.   

The simple fact is that virtualization, while releasing us from many of the limitations of physically bound computing, introduces new complexity and challenges that can also raise the risk profile.

One of our customers recently discovered during analysis that security and permissions could have created a disastrous situation post virtualization.  Settings that were once isolated to one server suddenly had implications for everything else that was to be combined on the same host.  This of course highlights the need to plan carefully prior to leaping.

How does risk factor into your decisions around virtualization and consolidation?

Virtual Fit for Purpose

Having just emerged from some serious white paper writing it seems like a great time for a new post.  We released three new papers last week, and before I put my pen down I will keep the momentum going for a few more paragraphs so I can chime in on what we, and the market, have been up to.  And I will avoid the urge to make lame excuses as to why it has been so long since I have posted - having firmly established a pattern of infrequent blog posts in the past I can safely ride along on a finely crafted wave of low expectations. I should also mention that two of my colleagues - Gerry Smith and Chuck Tatham - will be joining me in posting to this blog to share their insights from now on.

The three new papers cover some interesting topics: one is a total update of our virtualization paper (which was over one year old), one is a discussion of virtualizing applications onto mainframes, and one is a smaller paper on advanced workload analysis.  The last two papers cover features being released in version 4.6 of our product, which we announced April 22. 

The new release has some interesting features around personality-based benchmarking and contention risk analysis.  Although this all sounds wonderfully esoteric, the new capabilities actually serve two simple purposes: to understanding what a workload will look like when it is virtualized, and to determine how high to stack workloads before the risk becomes unacceptable.  Neither of these topics is trivial, particularly when cross-platform transitions are involved (hence the advanced benchmarking) or when stacking ratios are high (hence the risk analysis).  And both of these conditions seem to be arising more and more.

Which brings me to the market.  Vendors seem to be releasing bigger and bigger systems, and in many cases it makes a lot of sense to use larger, vertically scaled nodes over smaller “commodity” servers.  I’m sure HP’s new DL785 systems will swallow up workloads like they’re going out of style, and this model joins existing offerings from IBM, Unisys and Sun that pack a surprising amount of horsepower in a single x86-based system.  The extreme end of this trend is, of course, the mainframe platform, and we also see a surprising amount of interest in understanding what applications will look like on mainframes.  Perhaps even more interesting, this demand is coming both from organizations that want to use their spare capacity as well as shops that currently have no mainframe footprint at all.  Interesting times indeed.

Which makes the aforementioned features very important.  When analyzing workloads between platforms simple MHz-based estimations of relative performance are out the window.  It becomes all about the workload personalities, and the relative capabilities of each platform to host a particular type of workload.  For example, if you are calculating PI to thousands of digits then your best bet would probably be to run it in a screaming x86 or RISC-based system with a high per-core compute performance.  If, on the other hand, you are analyzing an OLTP database workload, the relative performance of a mainframe may win the day.  These types of decisions are becoming more and more common, and many large organizations are undertaking “fit for purpose” projects in order to figure out what their chosen platform is for each type of workload.

This fit for purpose applies not just to the individual servers, but also to the types of virtual pools and clusters being built.  Critical, transaction-oriented workloads will often require a different kind of resource pool than non-critical or batch-oriented workloads.  Building different pools to different performance specifications allows IT organizations to provide multiple hosting options to internal and external customers, and resources pools that are designed to run critical applications may be more expensive to be hosted in than lower-spec batch pools.  This variety of options essentially amounts to virtual fit for purpose, and the ability to create such infrastructure and offer multi-tier hosting options to end consumers will certainly spell doom for one size fits all virtualization planning.

I would go on but I would be repeating words that are written elsewhere.  Every topic has a best document to be in, and if this discussion were any longer it would no longer be suitable for a blog posting.  Think of it as literary fit for purpose…

Evolution of the Virtualization Ecosystem

It’s been a while since my last post – we got busy in the period leading up to VMworld and the dust is only now settling.  The good news is that it was a great show, and aside from all the intriguing conversations we had we also managed to walk away with the Best of VMworld Gold Award for Capacity Planning and Consolidation.  Since gold is about the only color we don’t have in our analysis maps, the award was a great addition to our booth :)

But enough about us – let’s talk about everyone else.  I think it would have been difficult to not notice the sheer number of vendors focusing on this space, and the fact that it is becoming a market unto itself.  There were solutions on display that seek to add value from every conceivable angle, and any shortcomings in VMware’s offering were being filled by multiple vendors.  If one takes the view that the virtualization game will be won or lost based on the ecosystem, not the hypervisor, then this bodes well for VMware.

If, however, one looks at the sustainability of this ecosystem then it paints a somewhat different picture.  The announcement of ESX 3i, which eliminates the Linux service console, undoubtedly sent a shockwave through the vendors that rely on that component to make their solutions work.  Any vendor running scripts on the service console or relying on access beyond the APIs will be in for hard times if VMware continues in this direction.  And, unfortunately for them, VMware should continue in this direction, as any product that doesn’t restrict access to approved, maintainable interfaces will become untenable in the future.

Which brings all of this back to that familiar argument about the downside of de-facto monopolies.  Even if your product uses the published APIs, your future is dependent on them.  And with one dominant player in this particular market, that creates a dangerous situation for ISVs.  We at CiRBA sleep at night because we are not dependent on this level of access, and even more so because we span all virtualization technologies and can even help provide mobility between them (which I’m sure helps our customers sleep at night as well).  But for other elements of the evolving ecosystem the story is much like that in nature: a little more diversity would probably be a good thing...

Hold the Fudge Please

For anyone getting serious about virtualization, one of the first questions that comes up is how to account for the overhead introduced by the virtualization technology being used. Any time you place a layer of abstraction between resource demand and supply there is a high likelihood that some overhead will result.

When we look at common server virtualization technologies there is significant variability in the level of overhead.  Above-kernel technologies such as Solaris Zones are based on a containment model, which is fairly efficient since the device drivers and scheduler activity are controlled, not virtualized. On the other hand, below kernel technologies such as VMware tend to replace what would have been a fairly direct link to the hardware with virtual device drivers, thus generating CPU activity whenever I/O occurs. Paravirtualization technologies (like Xen) and partitioning technologies (like LPARs) have different characteristics yet again, with LPARs having some very sophisticated I/O virtualization capabilities

Of course there is currently a lot of focus on VMware, and how to determine how high you can safely pile the workloads.  Because it is easy to do, it is a common practice to use simple fudge factors to account for overhead when stacking workloads. Overhead numbers in the 15% range are common, and fudge factors upwards of 30% are not unheard of in some situations. Of course, this is like surgery with a sledgehammer, and is no replacement for a proper model of overhead.

To properly analyze this, let’s consider a workload pattern for a physical server:

This is a server that uses spends most of the day between 0 and 25% utilization and peaks up as high as 50% late in the day. (This is a quartile-based view, which I won’t get into too much here.) To determine what this will look like in a VMware environment, the next step is to look at the I/O rates on this server:

From these curves we can see that this server is not I/O intensive for most of the day but has some reasonably sustained I/O activity after 7pm. In VMware, this I/O will pass through virtual drivers, which effectively pass the operations on to the underlying device drivers, but generate CPU while doing so. We can therefore use these curves to estimate the true overhead on the CPU, and attribute this overhead to the corresponding times of day.

The result of using this approach is a virtual CPU curve that looks like this:

When compared against the original curve it is clear to see that the overhead is skewed heavily toward the periods of significant I/O activity, and that periods of low I/O do not contribute disproportionately to the overall utilization levels.  (This curve accounts for overhead caused by network I/O, disk I/O and general scheduler overhead, and is exaggerated to help illustrate the effect.) 

When applied in practice this helps bring an additional level of accuracy to virtualization analysis and helps prevent layering fudge factors on top of fudge factors, which tends to lead to sticky situations that can undermine your attempts to trim down your data center. 

An Ounce of Analysis is Worth a Pound of Memory

Time to roll up our sleeves and talk about memory, and in particular the sharing of memory between virtual machines.  In many virtualization initiatives targeting existing environments the CPU utilization of the target servers is fairly low, particularly in Windows environments.  In an earlier post I went into some of the reasons why this is the case, but this time I want to talk about how this may cause the emphasis of a virtualization project to shift toward memory.

Low CPU utilization often implies high stacking ratios, especially if the workloads aren't overly "peaky" in nature.  But in practice this simply means that something else will become the bottleneck, and because it is uncommon for servers with low utilization to have high I/O rates, the next bottleneck is often memory.  In general, the memory requirement for a process is not related to its activity level unless it is competing with other applications, a situation that is rectified at the OS level by memory paging.  But in most steady-state production servers enough memory is added to prevent paging, thus giving the application a nice big space to spread out and laze around as it does nothing.

When stacking applications together in virtual machines, an opportunity exists to share read-only memory pages between VMs, but this opportunity will be squandered if they are virtualized the wrong way.  Most users of Solaris 10 Zones are familiar with the "sparse root" model and the advantages it gives with respect to memory sharing.  Simply put, the more similar you make each zone the more memory sharing will occur, and this is often a major consideration when determining what apps to put together on the same physical system.

What most people don't know is that VMware does this too, albeit in a different way.  If VMs are sharing identical pages of memory VMware will eventually notice this (using checksums) and start sharing them.  But this will only happen if the VMs are running similar stuff (i.e. the same OS, Rev, service pack, app revs), and if every VM is different then there won't be much to share.  And the difference can be significant - over half the memory in use can be shared with the right configuration, effectively doubling the potential virtualization ratio for a given environment.

Unfortunately most Windows virtualization initiatives ignore this and focus primarily on utilization information.  This tends to create configurations that have relatively low memory sharing potential (since this is not an optimization criteria), and consequently have lower virtualization ratios.  By adding configuration uniformity as a criteria when analyzing servers it can have a surprising effect on the level of virtualization attained, and all it takes is a some foresight in the planning phase.

Are You Virtually Compliant?

In order to help prevent technical anarchy on a grand scale, I find myself frequently pointing out to those undertaking virtualization initiatives that it is not merely a sizing exercise.  First and foremost it is an exercise in constructing a coherent infrastructure that puts the right things together and keeps the right things apart, especially when dealing with production systems.  You don't build a sophisticated IT infrastructure over many years to simply have it "randomized" into VMs based on utilization.

Some counter by asking "Why not?  If the systems are in their own VMs then what difference does it make?"  Rather than go into the details of how problematic it can be to combine systems with different availability levels, different DR strategies, etc., I find it easiest to think of things from a security perspective.

System-level security has traditionally been based on the premise that you cannot see the local storage without first authenticating with the system at the OS level (i.e. logging on).  Network-based storage authenticates at the system level and thus also prevents one from accessing it without being "on the system".  This access control is like a castle wall that protects everything inside it, and is the first line of defense in any IT security strategy.

Virtualization is a different paradigm, and as these things go, new paradigms often upset existing assumptions.  Most virtualization technologies store their VM images in files, and this encapsulation enables many of the benefits of virtualization by providing portability, state management, interoperability and a whole list of other advantages.  But it also means that those with access to these files can bypass all authentication mechanisms on that system; by having a complete file (and memory) image of a virtual machine, you can peer inside it with little effort.  The castle wall is suddenly made of very thin paper.

In fact, most VM vendors provide utilities to "mount" a VM image as a disk, providing convenient access to its contents.  This means that the old fear of someone "walking off with a disk" has now become a fear of someone "copying a file", which is considerably easier to do.  But more to the point of this topic, it means that any administrator or user with access to these images becomes a "super super" user, with access to the contents of systems that they may not otherwise be privy to.  That fact that a single user exists that can access (and modify) the data of any app on a virtual cluster is enough to kill pretty much any compliance audit.

There are, of course, ways to prevent this.  Some vendors support encryption of the VM images, although this will undoubtedly cause performance to tank.  Some provide affinity and anti-affinity rules to influence the movement of images within a pool of physical systems, but offer no practical means to establish and maintain these rules.  (Not to mention the fact that they are already in the same pool may make it too late to achieve true isolation.)

The best solution, in my opinion, is to simply avoid combining systems that should never go together.  Seems simple enough - this is what you spent the last 10 years doing, so why stop now.  And the best way to accomplish this is to establish in advance the business constraints that govern a virtualization project and ensure that these constraints are honoured throughout the process.  And then look at sizing...

Utilization v Moore's Law

A lot of organizations look to consolidation and virtualization as a way to increase utilization.  While this is certainly true, there is another very interesting way to look at it: consolidation and virtualization are necessary tools in the battle against falling utilization.

The reason?  When it comes time to refresh servers, you simply cannot buy gear that is nearly as slow as the stuff you are replacing.  Moore's Law dictates that the number of transistors on a chip will double every 18 months, which roughly translates into the doubling of compute speed.  This has certain limits, and lately we have seen a shift away from increasing speeds and toward "multi-core" strategies, but the net effect is roughly the same.

This means that on a 3 year lease cycle you will be replacing systems with ones that are roughly 4 times faster.  The increase in app response is no doubt appreciated by end users, but from a utilization perspective this means that the utilization of these systems drops significantly with every hardware refresh. 

When viewed on aggregate this has interesting implications.  If we approximate Moore's Law to say that speeds increase by about 58% per year, and we consider a pool of 300 servers that have been purchased over 3 years, then the aggregate compute capacity would be:

100+158+252 = 510 units of power (realtive to the power of the year 1 servers)

If on the 4th year you refresh the first 100 with servers that are 4x more powerful then the pool becomes:

158+252+400 = 810 units of power

This is, interestingly, a 58% increase in the compute capacity over the previous year, meaning that upgrading only a portion of a server pool will still track to Moore's law on the larger set.

More importantly, however, it means that you need to achieve at least a 4:1 consolidation ratio on any new servers just to keep your utilization levels stable (ignoring application growth).  If you are replacing 6 year old servers then you need a 16:1 ratio, and for 9 year old gear you would have to achieve a 64:1 ratio just to keep things the same.  Needless to say this can become quite challenging.

The upshot of all this is that doing nothing will typically cause utilization levels to drop over time, and the single-digit utilization levels that many organizations experience (particularly on Wintel servers) are undoubtedly contributed to by this effect.  It also means that anyone looking to use consolidation or virtualization to increase utilization levels better be very careful about the expectations that they set.  The incessant shrinkage of transistors is working against you...

In Praise of Mainframes

I've had a number of discussions recently regarding the sizing of servers for use in virtualization initiatives.  On one hand, keeping the cpu count low has the advantage of keeping software costs down, and allows an infrastructure to be constructed of low-cost, commoditized servers.  On the other hand, using larger servers has an advantage in that it provides better economies of scale when combining workloads, and also provides higher headroom for peak demands.  This has understandably been a hot topic of late.

When I try to put this in perspective I can't help but turn my thoughts to the mainframe world, where virtualization and proper capacity planning have been mainstays for decades.  This is the epitome of the scale up model, and few if any mainframe environments are suffering from the sprawl and underutilization that we are seeing in the opens systems world.

And there is, in my opinion, a significant underlying reason for this. It is what I call reverence for the platform, and, put simply, it is the fact that people respect big, expensive systems more than small, cheap ones. Why do we rarely see VM sprawl on mainframes? Because people don't mess with them.

This respect is important.  How important? Just watch any movie.

Nobody goes on a quest to seek advice from "blade rack"

Nobody tries to defeat the enemy by knocking out their "midrange"

Nobody crawls through air ducts to gain access to "file and print"

They go to Mainframe

They go to the biggest, most vertically scaled thing they can find.

In fact, the only prominent example of a horizontally scaled technology in a movie that I can think of is the Borg. But it was evil – more like a grid with attitude. And that may have been a TV show.

So next time you are sizing servers for virtualization be sure to pay some attention to the ultimate non-technical consideration: reverence for the platform.  And who knows - if a bunch of actors come walking through the server room they may just try wreck your stuff instead…