Beware of Vendor-Sponsored “Analysis” Reports

Mark Twain allegedly came up with the famous line: "Figures don’t lie, but liars figure." That’s a good thing to keep in mind any time you’re looking through a report that was sponsored ("sponsored" = "paid for") by a vendor that concludes that their product is better than the other guy’s.

Maybe it is better than the other guy’s. But you might want to look closely at what was tested, how it was tested, and whether they were, shall we say, selective in the facts they present.

Case in point: The Tolly Group’s report, released May 27, comparing VMware View 4.6 Premier Edition to Citrix XenDesktop 5 Platinum edition. There are several interesting aspects to this report, which are dealt with in detail in Tal Klein’s blog over on the Citrix Community blog site. Here are a few of the more egregious items:

  • VMware View 4.6 Premier licensing costs less than XenDesktop 5 Platinum. Absolutely true, and absolutely irrelevant. That’s like pointing out that if you load every possible dealer option onto your new car, it’s going to cost more than the basic model. Thank you, Captain Obvious. If you want an "apples-to-apples" comparison, you need to compare VMware View to the XenDesktop VDI Edition. But wait, if you do that, XenDesktop is actually less expensive, and that would be an awkward point to publish in a paper that’s being paid for by VMware.
  • VMware’s PCoIP provides a more consistent multi-media experience than XenDesktop 5. (Over a LAN. Using a single thin client device that did not support any of the Citrix HDX media acceleration features.) Sorry, guys, but once again this is not an apples-to-apples comparison. And did they publish any results of testing across a WAN link? Nope…and for the same reason they didn’t use XenDesktop VDI Edition for their price comparison.
  • It’s easier to upgrade View 4.5 to View 4.6 than it is to upgrade XenDesktop 4 to XenDesktop 5. Once again, both true and irrelevant. It’s easier to give your kitchen a new coat of paint than it is to rip out the cabinets and completely remodel it. Anybody surprised by that? There are significant architectural changes from XenDesktop 4 to XenDesktop 5. It shouldn’t be surprising to anyone that this will involve more effort than a "dot release" upgrade.

I’ve always been skeptical of vendor-sponsored "analysis" reports, and, to be fair, Citrix has used the Tolly Group in the past for its own sponsored reports – but it seems to me that this one is just over the top. Apparently, former Gartner analyst Simon Bramfitt agrees. His pithy assessment of the report: "There are undiscovered tribes lost in the darkest parts of the Amazon jungle that would know exactly what to do if a vendor airdropped a pile of competitive marketing literature authored by the Tolly Group; send it back, and asked [sic] that it be re-printed on more absorbent paper."

What do you think?

2 replies
  1. Sid Herron
    Sid Herron says:

    @Simon – Frankly, I seldom use these kinds of reports in my selling activities, and if I do, it’s with full disclosure that it’s a sponsored report and that people need to look carefully at the methodology and draw their own conclusions on validity. I prefer to assume that my customers are at least as smart as I am, and will figure that out anyway. Therefore, any approach short of full disclosure risks insulting their intelligence and hurting my own credibility.

    Perhaps there’s a place for sponsored reports. I can understand going down that road if you’re new to a market and simply don’t have a sufficiently high profile to expect that objective comparisons will emerge on their own. But I do applaud Citrix for moving toward more objective testing that can easily be validated by anyone who cares enough to do so.

    Reply
  2. Simon Bramfitt
    Simon Bramfitt says:

    I didn’t really write “asked” did I?

    OK – fixed that. A combination of Dragon Dictate, inadequate proofreading and late nights, sorry.

    Citrix seem to have sworn off the sponsored report at least as far as the desktop division is concerned, but the NetScaler team still seems to think that it is an acceptable approach to take. I wonder if it is possible to read anything into that?

    I’d be interested to hear if you are ever asked about the content of these reports when speaking to your customers?

    Reply

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply to Sid Herron Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.