Open Source Code Contains Security Holes
January 10, 2008 4 Comments
That’s the title of a recent InformationWeek article. I hope this doesn’t come as a surprise to anyone. All code contains bugs, and some of those bugs may turn into security issues.
Open source code, much like its commercial counterpart, tends to contain one security exposure for every 1,000 lines of code, according to a program launched by the Department of Homeland Security to review and tighten up open source code’s security.
Popular open source projects, such as Samba, the PHP, Perl, and Tcl dynamic languages used to bind together elements of Web sites, and Amanda, the popular open source backup and recovery software running on half a million servers, were all found to have dozens or hundreds of security exposures and quality defects.
A total of 7,826 open source project defects have been fixed through the Homeland Security review, or one every two hours since it was launched in 2006, according to David Maxwell, open source strategist for Coverity, maker of the source code checking system, the Prevent Software Quality System, that’s being used in the review.
A couple comments. Looking at the defect rate for some popular Open Source projects, they are consistently way below the average:
* Samba was found to have a total of 236 defects, a far lower rate than average for 450,000 lines of code. Of the 236 defects, 228 have been corrected, said Maxwell in an interview.
* Linux came in with far fewer defects than average as did a number of other open source projects. The version 2.6 of the Linux kernel had a security bug rate of .127 per thousand lines of code.
* The Apache Web server includes 135,916 lines of code, which yielded a security defect rate of .14 bugs per thousand lines of code.
* The PostgreSQL database system contains 909,148 lines of code, with a .041 defect rate.
I think those numbers speak volumes about the Open Source methodology. Beyond that though, is the fact that a study like this can be done in the first place. That’s the real power of Open Source. Security defects aren’t hidden and denied. They are out in the open and actively being fixed. Security by obscurity is a myth.
–jeremy
Jeremy's Blog 
jeremy,
I’d like to ask you to introduce (if you have them) the numbers of defect rate in non-Open Source software for the sake of healthy comparison. Obviously to the reader numbers such as .127 per thousand lines sound bold: they sound to me as no defective at all. I’ll starve to know how much the defect rate sounds for a – for instance – Microsoft software. Do you have this? Great article, by the way, thanks a lot for sharing.
Well, that’s part of the issue. Numbers like that are not usually available for proprietary software, since the code to run the tests is not openly available. Even if a company gave the source to Coverity to test for internal knowledge, they would almost never publicly release the numbers.
–jeremy
Neat article, however defects per line of code is somewhat of an irrelevant benchmark. For one, lines of code is only a measure of a program’s complexity in the broadest sense, and one remotely accessible vulnerability is enough to leave you in serious trouble.
For another, the number of defects entirely leaves out the severity of each defect, as by this metric a remotely exploitable root code execution defect gets the same weight as the use of an uninitialized variable.
I wouldn’t say that defects per line of code is *completely* irrelevant, but you certainly need to take it for what it is… and nothing more. Generally speaking, less defects should mean less vulnerabilities and also less critical vulnerabilities. Of course, specific cases may very, but I think the track record of the projects mentioned bares out this general case.
–jeremy