eWEEK Labs Bakeoff: Open Source Versus .Net Stacks

You guessed it, yet another “interesting” benchmark has been released. This time it's an eWEEK Labs Bakeoff. The benchmark seems to ascertain that .NET and Windows in general ended up outperforming LAMP. The test?
We used portals we consider popular—Microsoft SharePoint Portal Server 2003 (built on ASP), XOOPS (PHP), Plone (Python), and Liferay and JBoss Portal (JSP).
The problems with the report start already. They are comparing products that are in no way similar.
SharePoint is a “Office collaboration and information sharing portal”
XOOPS is a “extensible, OO (Object Oriented), easy to use dynamic web content management system”
Plone is a Zope based “content management system with strong multilingual support”
As you can see, SharePoint is not a product you'd ever use as a portal replacement for these other products. Aside from that huge differences, there are myriad other problems. While XOOPS may be the “most popular on SourceForge” I'd not say it's an enterprise ready product that out of the box focuses on performance. Additionally, SharePoint uses MSSQL, XOOPS MySQL and Plone ZODB. To be fair, despite the title of the article they do start referring to it as a stack test, which is seeming about what it's turning into…except you can't test a stack and then base the results on completely different products meant to do completely different things. It gets worse though. “But the point was to test the stacks, not their ideal performance points, which is also why we didn't tune or optimize any of the systems but ran them as close to default as possible.” In no circumstance would any company just deploy any sized portal without doing any optimization. For the case of plone, not only did they not perform any optimizations, but they installed it directly contradictory to how the manual says to install it. XOOPS is also going to get heavily penalized here, since out of the box MySQL comes with a horrific setup from a performance perspective (if you're wondering why, it's because MySQL is installed by the majority of Linux installs with the realization that the end result is not going to be a high end MySQL server. Because of this, the default setup is a resource friendly one, not a performance friendly one). The end result is they somehow managed to get 2 (yes, 2) transactions per section out of the LAMP install. That number should have raised an immediate red flag IMHO. The end result to me, is that the study is near useless.
On the topic though, the study could have been an interesting one and still could be. A tuned setup comparing a Windows portal and a Linux portal would be interesting. Not just from a performance perspective either. Let's get one of each setup tuned to the max and configured properly. Then, test it repeatedly over a long period of time so that the stability can be entered into the equation. Lightning fast but completely unstable is of no use in the real world after all. Also, let's apply official patches over this long period and see the impact they have on both performance and stability. Now that would be a useful article. It would also be extremely time consuming and costly, which I assume is why no one ever does it. The reality though is that until someone puts that amount of effort into a study like this, they aren't going to be useful to anyone…and they'll be misleading to almost everyone.
–jeremy
Microsoft, MSFT, Open Source, .net, LAMP, WAMP, SharePoint, Zope, XOOPS, plone, ASP, python, eWeek, Jboss