"Finding and fixing a software problem after delivery is often 100 times more expensive than finding and fixing it during the requirements and design phase."
Software Defect Reduction Top-10 List
Open Source Software (OSS) has been embraced by individuals for decades, but only recently have organizations around the globe looked upon open source as an attractive and practical alternative to proprietary software. In addition to its appealing price tag, usually free, OSS can be inspected, modified, and freely redistributed according to the terms of its license.
In spite of this, open source developers still find their code quality and security challenged by a question that has followed them from the beginning: "If it's free, how good can it be?"
Scanning for Defects
The short answer to this question is "Quite good", thanks to the determination of open source developers and the success of the Scan project a collaborative venture between Coverity, Inc. and Stanford University.
Launched in early 2006, Scan is a key component in the "Vulnerability Discovery and Remediation Open Source Hardening Project", a three-year initiative funded by the U.S. Department of Homeland Security to identify and correct vulnerabilities in widely used open source projects.
The goal of Coverity's Scan is to reinforce the security and improve the overall quality of OSS. By applying the latest innovations in automated defect detection, the Scan site has uncovered some of the most critical bugs in OSS. The technological underpinning of Scan is Coverity's Prevent SQS (Software Quality System) solution, which automatically identifies and helps developers eliminate defects in source code. Using static source code analysis, Prevent allows developers to find and fix defects at the earliest stage in the software development lifecycle. In static analysis, the code being analyzed is not executed; therefore, test cases and specially designed input datasets are not required and examining code for defects is not limited to lines of code that are run during some number of executions of the program.
Static analysis provides a comprehensive examination of all lines of code in a given codebase, and all of the different paths through that code that can be triggered by varying application input. Coverity Prevent SQS is a robust static analysis solution that pinpoints buffer overflows, memory allocation bugs, and other vulnerabilities that become a target for malicious hacking attacks. It also reveals quality defects that may become larger issues over time, such as insufficient checking of error codes.
Hunting for these types of defects in software code is a meticulous and time-consuming process that most software developers would prefer not to do manually. Prevent automates this process, providing workflow that allows developers to assign and monitor software defects and overall code quality. Despite these obvious benefits, initially we weren't sure what reaction we could expect from the open source developer community.
The initial plan was to perform daily security audits of approximately 40 of the most popular open source software packages, including Linux, Apache, MySQL, Sendmail, FreeBSD, Mozilla, and Samba. Stanford University would perform a detailed analysis of the results and maintain a database of the findings. From the start, we intended to make the results available to each project's developers; how they chose to respond was up to them.
At Coverity, we had performed earlier, non-government funded scans of the Linux kernel and MySQL. The developer response was promising. For instance, the original 2004 study of the Linux kernel revealed five file system buffer overrun conditions and one network buffer overrun condition, both of which were considered serious defects. The 2005 study, performed six months later, showed zero defects of the same type: all had been resolved. Even though the size of the Linux kernel had increased significantly during those six months, there was a significant decrease in the number of potentially serious defects, thanks to the response of Linux developers.
An Auspicious Beginning
In the initial analysis, we scanned more than 17.5 million lines of code from 32 open source projects. On average, we found 0.434 bugs per 1,000 lines of code which prompted a response from developers of other open source projects. More than 200 developers registered for access to Scan's online database the week after we published our initial analysis results.
Over the next seven days, more than 900 defects were resolved, or an average of more than 5 bug fixes per hour. After one week, defect density for the same 32 projects dropped from 0.434 defects per thousand lines of code to 0.371. Samba, the popular program that allows end users to access and use files, printers, and other resources on a company's network, showed the fastest developer response, reducing their defects from 216 to 18 in the first seven days. "Coverity found bugs in parts of Samba that we had previously considered completely robust and tested," said Jeremy Allison, head of the Samba development team. "Coverity is making a major contribution to the code quality of the Samba project."
The Amanda project was another clear-cut success story. Amanda is the Advanced Maryland Automatic Network Disk Archiver, a backup system that allows the administrator to set up a single master backup server to back up multiple hosts over the network to tape or optical media. The initial scan of Amanda's code revealed 108 defects. Within the first week, Amanda's developers resolved all issues, a fact that was verified by a scan revealing zero defects.
Also within the first week, Scan revealed a major security vulnerability in X Windows software, a graphical windowing system used in most distributions of Linux and Unix systems. The vulnerability would allow any user with a login to execute arbitrary code with root privileges or cause a denial of services with root privileges. X.org, the developers of X Windows, responded immediately by issuing a security advisory which included a patch to fix the problem.
Scan Celebrates its First Birthday
By the end of the first year, Coverity had witnessed some remarkable successes in the field of OSS development: Developers fixed 6,132 software defects across 53 open source projects, including 13 projects that remedied all outstanding defects. Hundreds of developers were using Scan's analysis to improve their projects and many others had contacted us asking to be included in the Scan. Based on demand from the open source community, we decided to expand the program. On the first-year anniversary, March 6, 2007, we unveiled the expansion of the Scan project. In addition to a site redesign, 100 new projects were added. More information was made available for developers and others interested in understanding what Scan is and how developers use it.
We also put a new framework in place to help open source developers learn how to use Scan results by gradually introducing them to more advanced features of Coverity's Prevent SQS solution. Projects that actively use the Scan results became eligible to move up a ladder of "rungs" and receive access to additional functionality. Finally, within the new framework of the Scan Ladder, additional analysis results that were not enabled during Scan's first year were made available to the developers. The response has been overwhelmingly positive.
The Scan Ladder Grows
Because of the extraordinary successes we've seen through Scan, Coverity has invested in dedicated resources to the project beyond the requirements of its contract with the U.S. Department of Homeland Security. The volume of requests for access to results and for the inclusion of additional projects has shown us that the open source community recognizes the benefits of static analysis.
In addition to adding many new projects, we've also included projects outside the scope of critical infrastructure originally defined by the Department of Homeland Security, since preventing crashes and data loss are obviously worthwhile contributions in other code bases.
These new projects come from a wide array of OSS categories, from developer utilities to graphic tools. The projects are displayed on the Scan Ladder, organized by the degree of experience the developer has built with the project, their communication, and their progress in addressing the issues found by the analysis. Currently the Scan Ladder consists of two rungs:
- Rung 0: the first rung includes projects that have been built and analyzed, but representatives of the project are yet to register for access to the results
- Rung 1: once a project provides a set of official contacts to represent the project to Coverity, developers have access to a mailing list designed to facilitate the discussion of results and questions surrounding Scan and Prevent SQS functionality
In the near future, a number of projects are poised to progress beyond the first rung, as 14 of the original Scan member projects successfully reached zero defects within their first year.
There are currently 265 projects on the Scan ladder. We have found that most of the open source developers don't need much encouragement to participate in correcting security and quality defects in their code. Open source developers take a lot of pride in their code, which-being open-is already subject to public scrutiny. They tend to be quick to fix issues that have obvious consequences, and many of them want to fix defects that may have potential future consequences.
It should be noted that Scan is not the only way these projects check for bugs and defects. Each project has its own methods, and several have some form of regression tests, as well as development and release branches and a formal release engineering process.
Looking to the Future
The Scan site currently analyzes over 32 million lines of code daily. There have been more than 7500 defects fixed in open source projects since the Scan project started, which equates to more than one bug fixed every two hours.
Since Prevent points directly to the root cause of a problem, it's difficult to say how much developer time would have been required to identify and fix these problems if they had been manually tracked down by examining reports of the bug's effects. End-user reports take time to process because of the difficulties in getting clear explanations of the problem, its context, and then having a developer duplicate it. For example, notoriously subtle heisenbugs which appear in the binary but not under debug mode, can take weeks to track down.
According to some people, OSS shouldn't have any bugs because the source code is public, and so many people can look at it. That theory doesn't take into account that many of the "eyes" don't belong to programmers with domain-specific knowledge and the interest to spend time working on that code.
Open source developers regularly give me feedback about how useful the Scan service is, so I have no doubt it is contributing value to the projects. Individuals have reported that their coding methods have changed as they know that certain sloppy programming habits will be called out by the analysis, and they feel that they've become better programmers by correcting these habits. Scan developers have seen what Prevent can identify through static analysis on their code. As a result, when the defects identified at the Scan site have all been fixed, they have a higher degree of confidence regarding the security and quality of their code. The goal of the Scan project is simple: to further improve the quality and security of OSS. If, in doing so, we can better understand how the combination of our technology and the open source development model leads to defects being fixed at such a tremendous pace, we believe that we'll be able to apply what we've learned to improve the development process and security of all software, open source and proprietary. So far, Scan has taken us a long way towards that goal.