ScriptLogic Windows 7 Study Is A Crock; Media Swallows It Anyway

The marketing tactic of spreading FUD — fear, uncertainty, and doubt — is nothing new in the computer industry. Many companies point to questionable studies as their tools of dissemination, like a PR agency study for Credant Technologies that I recently deconstructed. Well, there appears to be another bogus study, only this time the effective target is Microsoft Windows 7, as the study effectively suggests that enterprises aren’t planning to upgrade to the new version. The press pick-up pretty has been wide and undiscerning. That’s a pity, because dig into it and you find classic varieties of bias that make for good sound bites but egregiously bad for competitive decision making.

I first noticed the study in a reference that my colleague Michael Hickins made about Microsoft’s last stand. Michael pointed to an article in InformationWeek, usually a reputable news source. The short take is that according to a survey commissioned by tools vendor ScriptLogic, enterprise customers aren’t largely interested in moving to Windows 7:

59.3% of the 1,100 IT administrators that responded to a survey by management tools vendor ScriptLogic said they have no plans to deploy Windows 7.
42.7% of survey respondents said time and resources required to implement a new OS were the biggest barriers to deployment, while 39.1% cited application compatibility as the biggest hurdle.34% of those surveyed said they would likely deploy Windows 7 by the end of 2010, while just 5.4% said they would move to the OS right after it debuts later this year.

The article then goes on to state that the results “show tepid corporate interest in Windows 7, raises questions about Microsoft’s ability to innovate in the enterprise market.”

But while reading this, I started remembering a very different story that I had read and quoted in a BNET Technology post just a few months ago. According to a story that Ed Bott had done on ZDNet, two different studies suggested a potentially rapid uptake of Windows 7 by enterprise IT departments within a one to three year time frame:

• One study by Dimension Research suggests that 80 percent of the IT professionals it surveyed planned to move their companies to Windows 7 within 36 months. That would be an amazingly quick shift, compared to upgrade patterns of previous versions of Windows.
• Investment bank Pacific Crest Securities polled 80 IT decision makers at companies with more than 1,000 employees. Yes, a smaller sample than you’d like, but significant companies. Half planned to upgrade to Windows 7 immediately, 46 percent were going to upgrade before the end of 2009, and 55 percent of respondents were going to do company-wide upgrades rather than rely on new PC purchases to make the change.

The Dimension Research study in particular fueled a lot of “Microsoft is dead” press from people who presumed that adoption rates should be faster. But as Bott showed, if the projections were close to accurate, the success of Windows 7 (blue line) could cleanly blow Windows XP (orange line) out of the water.

A three year time frame may seem forever, but it has historically taken longer for new desktop operating system versions to take hold. In fact, when Windows 95 came about, I was in the industry and remember on being surprised at how long older versions of Windows (and even MS-DOS, at that point) hung around. That’s because corporations are measured in the ways they roll out new technology. First they want to test, and then they are generally careful and slow in roll-outs.

Of course, projections can be wildly off. But so can surveys, which claim to show what is, not present one entity’s estimation of what might be. Although the ScriptLogic study boasted 1,100 responses, it seemed to be from a different planet, so I began checking online and came across a fairly scathing critique of the study, with a link to the PDF version of the results.

Author Sean Kalinich honed in on the questions, and rightly so. Question construction one of the biggest sources of survey error in market research, with another being how you construct the sample. Here’s my take on the questionable aspects of the study and of the reporting in InformationWeek and other outlets:

• The question of how a company plans to deploy Windows 7 gave respondents the choice of whether they had already deployed, would do so by the end of 2009, would do so in 2010, or had “no current plans” to deploy. The fact that some of the people claimed to have deployed an operating system that is only in beta and that won’t be commercially available at all until August at the very earliest for site license customers is immediately questionable. And then the question effectively forces anyone not deploying by 2010 to indicate no “current” plans — which was then taken up by the media as no plans at all. This is classic question bias.
• The reporting failed to note that many responses showed a huge impact of the economy on IT spending, with over a third of respondents indicating that their companies had saved money by skipping upgrades or delaying purchases. That puts the question of upgrading to Windows 7 into a context that is potentially about the economy, not about the product, suggesting that any software would have the same problem.
• There are no questions about other software upgrades or replacements, so there is no way to know if enterprises have a particular animus toward Windows 7 or to spending money on software in general if they can get along with what they have.
• The response about upgrading can also be taken as 40 percent upgrading their companies in the next 18 months. Given that the OS won’t be out until August to September, it’s more like a 15 month upgrade for 40 percent of a customer base. That is pretty quick.
• It’s important to remember that the study was sponsored by a vendor, much of whose focus is selling product to manage Windows and migrate systems to the new version. In other words, the vendor sponsor has a huge axe to grind, which shows up in the question about what is keeping companies from upgrading — 42.7 percent answering “time and resources.” This is a pitch to sell its own products.
• Although the sample size is 1,100, that’s not the only question you would need to ask. How was it chosen? What were the demographics of the people answering? Was the company getting a representative sample of enterprise decision makers? According to Kalinich, the survey got the 1,100 out of 20,650 survey invitations sent. That’s about a 5.3 percent response rate, which means it’s likely in the realm of so-called self-selected samples and not something that you could trust.
• There is no mention of margin of error, which brings to question whether this was really a random sampling, which you’d need to draw the broad conclusions that so many media outlets have found appealing link bait, or a stacked deck.

In short, this study reads like a mathematical crock put together by a company so it can try to offer itself as a solution to an enterprise IT problem. Unfortunately, too many “news” organizations are either overly anxious for sensational stories, desirous of information that will confirm a pre-existing hypothesis, or ignorant enough about the statistics of surveys that they assume anything with numbers put before them must be real.

The real irony here is that Microsoft itself has been for years a master of FUD. In fact, I had thought for years that the company had actually started the term, though according to Wikipedia, it was the formulation of Gene Amdahl when he left IBM. It’s amazing what you can learn when you check what is “known.”

Of course, it might be that the results of a poor exercise in survey work could happen to be right anyway. But to take take such a gross mathematical error as fact for decision making is to firmly place your head onto a sand dune and push downward.

Image via stock.xchng user hotblack, site standard license.