The Sin Of Omission
More interesting than the numbers they published were the numbers they didn't publish:

• a direct comparison of two OS platforms on the same hardware
• a direct comparison of core kernel OS functions between Mac OS X and Linux
• a direct comparison of properly optimized software across the two platforms
• a direct comparison of different database software packages.

For example, they could have also tested the free evaluation versions of Oracle's database, which is optimized for both Linux and Mac OS X. Were they testing server performance, or their own ability to configure MySQL properly?

Their analysis of the results indicated they didn't really understand as much about Mac OS X as their impressive sounding conclusions suggested. Mac OS X actually does use kernel threads, and they are mapped efficiently to Mach threads. What was all that talk about layers and wrappers? It seems Anandtech had been bitten by the Mac OS X Microkernel Myth.

Unfortunately, the inaccuracy and irrelevance of their numbers was largely ignored. The only thing readers took away from the article was that "Mac OS X is apparently up to 5 times slower than Linux, and the problem lies conclusively with the design of its kernel."

In reality, Anandtech's article, along with later revisions and corrections, really said little about the Mac OS X kernel. Rather, all they conclusively demonstrated was that Linux experts know more about optimizing MySQL on Linux than they do about optimizing for Mac OS X. We knew that already.

Apple sells the Xserve and Mac OS X Server products as an easy to use solution that leverages the power of open source software, not as a product that that can successfully withstand the sabotage of a Linux admin intent on installing software compiled against the wrong version of GCC and lacking the proper patches required.

More "No More Mysteries" = More Mysteries
Last fall, Anandtech published a follow-up article where they admitted some of the original errors they made. Had Anandtech set up a test of a custom install of MySQL on Linux versus the MySQL included with Mac OS X Server, and applied their results as a ding against Apple for not having better optimizations in Mac OS X Server, their benchmarks would have made some sense. Instead, they left the impression that Mac OS X has an inherently flawed kernel that, if replaced by Linux, could garner 2-5 times the performance; that's certainly what people citing the article are thinking.

The idea of a 5x performance gap between Mac OS X and Linux, and the speculation about threading which Anandtech published as proof that the fault lies within the design of the "asthmatic" kernel, flies in the face of results reported by PC Mag. In an article published a few weeks before Anandtech's second article, PC Mag wrote:

"Performance-wise, the dual-processor Xserve G5 compares well with Linux-based x86 servers. This is not surprising, considering that Mac OS 10.x is based on FreeBSD UNIX. Using the included Apache server, we ran the Xserve G5 through our standard WebBench tests. It did quite well on the static WebBench test, outperforming a competitor's dual processor x86-64 server running Apache server on SuSe Linux-64."

PC Mag is hardly an advocacy site for Apple Computer. Since their tests found that Mac OS X running on a Xserve G5 compared favorably with 64-bit x86 servers running SuSE Linux, the same distro Anandtech used, it blows away any remains of credibility in Anandtech's report that Mac OS X is a pile of crap that can only handle a few concurrent users before falling apart.

While it's not hard to imagine that a highly optimized Linux server could be tuned to outperform Mac OS X's stock configuration, Anandtech's shoddy benchmarks certainly don't prove that Mac OS X is architecturally unsound and desperately needing a retrofitting. Rather, it indicates they are either very biased against Apple, or are just grossly incompetent.

Another recent example of benchmarking Mac OS X vs. Linux comes from someone who tested the R Project's statistical software on Mac OS X. The findings were not flattering.

• iPods and Mobile Devices
• Open Source and Development
• Wishlists and Product Concepts
• Product Reviews and Comparisons
• Unraveling Tech Myths
• Deconstructing Tech Analysts and FUD
• All Articles
• Hide Article List