OSDL, Say It Ain't So

We all know how silly some of the “Get the facts” benchmarks/reports were. When I saw this commentary on the Linux TCO study that was recently co-sponsored by OSDL, I figured it was just the anti-Linux Rob Enderle we've grown accustomed to. Then I read the report. Let me start out by saying there are some very accurate depictions in the report and I think a report like this was something we needed. But, some of the claims and numbers in this thing are so unbelievable that I'm almost surprised OSDL put their name on it. Have we really stooped to this level…even after calling Microsoft out on it like we did? While the commentary by Rob is also full of a few holes and is greatly exaggerated, the fact that we give him any fodder at all is a bit disappointing. Hopefully I'm just missing something, but let me make a few comments:
* The report starts off comparing itself to older Microsoft studies. This study should have merit on it's own, so the older studies really aren't pertinent IMHO. This is tangential at best and makes the report seem biased. To be fair this is more of a personal gripe than a legitimate compliant.
* A quote from within the report: A large hosting site reported 100% availability across 230 managed servers since October 2001 Wow, 100% over almost 5 years? Not a single reboot, hardware failure, power issue, system upgrade or user error in all that time? I guess it's possible, but 100% seems either highly anecdotal (and includes oh, we didn't count foo, bar or zed when considering availability numbers) or the number is a bit exaggerated (which is something you don't want in a study like this).
* They seem to knock Windows for “requiring twice the memory” of the Linux distributions covered, but if you look at the number they give it's 256M. If you are really purchasing production servers with that kind of RAM at the current prices, please see your purchasing department immediately. The cheapest Penguin box I can find won't even let you go below 1G.
* When comparing pricing, they use Apache/JBoss on the Linux side, which is a J2EE-compliant solution. They for some reason then choose Microsoft ISA Enterprise as the “web server” for the Microsoft side. The problem there is that ISA is not a web server! That is a $23,996 mistake but more importantly shows you that there is a fundamental misunderstanding of the platform. This is the biggest blow to the report IMHO.
* Also on the pricing costs department, the compare RHEL 4AS which is a one year subscription to a product that includes top level 24/7 support with a purchased version of Windows 2003 that would need per incident support. That's apples vs. oranges at best. To be fair there is a little disclaimer about the pricing, but it doesn't give someone without an understanding for the two products an indication of what the numbers really mean. They should have really justed used numbers that are fair to both sides, which is certainly possible.
I'll stop there, but you get the idea. So, while the report contains a ton of highly relevant and highly accurate data, the fact that it is marred by such inaccuracies means that the validity of the whole thing can now be called into question. This report could have been so much more – it could have been proof that we do things right and could have been used as a shining example of why Linux really is better.
–jeremy
OSDL, Levanta, TCO, Microsoft, Linux, Open Source