BogoSec Source Code Security Quality Metric

Credits

Legal Statement

Overview

Many software security vulnerabilities occur because of poor programming practices. Vulnerabilities are often algorithmically detectable by static source-code scanners designed for identifying potential security issues. As the number and severity of potential security holes per line of code increase, we believe that the overall quality of the source code in terms of security decreases. BogoSec metrics are computed values that attempt to reflect relative ratings of source code security quality for comparative purposes.

The motivation behind BogoSec is to influence developers to produce more secure source code over time. The various scanners that exist point developers to potentially insecure sections of code. Developers are often reluctant to use such scanners because of a seemingly high degree of "false positive" output as well as the difficulties associated with use. BogoSec attempts to reduce the penalty of false positives while broadening the scope of the source scan by utilizing multiple independent scanners. This produces high level metrics allowing developers and users alike to comparatively judge the quality of the source code in terms of security.

Methodology

Several source-code scanners exist that identify numerous vulnerabilities with varying accuracy and success. BogoSec parses the output of any number of source-code scanners and computes its metric based on the number, severity, and frequency of potential bugs found as per number of lines scanned. BogoSec currently supports the following scanners:

Support for additional scanners is easily extended by creating plugins that parse the output of the new scanners in the same way that existing plugins parse the currently supported scanners. This is useful for incorporating support for proprietary or internal scanning tools.

BogoSec requires that at least one of the above scanners is installed on the system and can be found in the path. These scanners are not distributed as part of BogoSec. However, BogoSec does include plugins that interface with each scanner.

The basic methodology of BogoSec is as follows:

  1. Execute each scanner present on target source code or tree
  2. Parse output of each scanner, determining the {filename, line number, severity, description} of each possible vulnerability
  3. Interpret the severity indicator and adjust to a common scale (by default, 10 being "most severe", 1 being least severe)
  4. Report the total number of vulnerability severity points, as well as the total number of lines analyzed by each scanner
  5. Calculate and report the BogoSec final score
    BogoSec Final Score    =    total vulnerability severity points from all scanners

    total lines of code analyzed by all scanners

The algorithm above considers weighted vulnerabilities. The total number of vulnerability severity points accumulates as scanners identify potential vulnerabilities. The ratio of points per line is the indicator by which users of BogoSec are able to compare source code in terms of security quality. BogoSec operates under the assumption that as the number of weighted severity points per line increases, the overall security of the compiled code decreases.

Implementation

BogoSec is implemented as a Perl script and accompanying Perl modules. One can add support for additional scanners by creating a module that executes the scanner against the source code, interprets the output, and normalizes severity ratings to BogoSec's scale.

Initially, BogoSec sets up its execution environment by parsing configuration files and reading command line parameters. (See the manpage for an extensive description of command line options.) The final parameter specifies the target, which may be a single source file, an entire source tree, an archive {*.tar.gz, *.tgz}, or a source RPM {*.src.rpm}. BogoSec handles the last two formats by creating a temporary directory and expanding the code accordingly.

The script builds a list of files in the target tree, consisting of any case insensitive file matching {*.c, *.h, *.cpp, *.c++}. Currently, scanners are executed on one file at a time. Although some scanners have the ability to recursively scan an entire tree, the stability and consistency of BogoSec favors compiling the results of individually scanned files. BogoSec is multithreaded such that all three scanners execute simultaneously on the list of files, waiting for the slowest scanner to complete execution before analyzing the results.

Each scanner's plugin has a routine defined for analyzing the particular output of that scanner. This consists of scanning each line of output to determine if the line specifies an identified potential vulnerability. If it does, the filename, line number, severity, and description are parsed out according to rules defining each scanner's output conventions. The severity is scaled according to BogoSec's scale (by default, a 10-point scale). The plugin obtains the number of lines of source code scanned from the output whenever possible. This data is stored in a structure accessible by the main program.

Finally, the main program tallies the number of points accumulated by all scanners on all files, as well as the number of lines of code scanned by all scanners in all files. The BogoSec final score is this quotient, which the script reports and then exits.

Interface

Files

BogoSec is installed through the bundled makefile, using the "make install" target. The installation creates the following:

Example

BogoSec is executed as any other command-line utility. Arguments are paths to the source code to be evaulated.

Example:

# bogosec tests/apache/apache_1.3.31.tar.gz
Running flawfinder...
Running rats...
Running its4...
flawfinder
  5212 points
  75682 lines
its4
  9502 points
  110022 lines
rats
  8573 points
  99234 lines
>>> Using scanners: (flawfinder its4 rats )
>>> 23287 total severity points
>>> 284938 total lines of code scanned
>>> final score = 0.0817277208843072

In the above example, BogoSec executed against the Apache source tree archived in tests/apache/apache_1.3.31.tar.gz. BogoSec can also evaluate individual files or uncompressed source trees. For a discussion of the various options associated with BogoSec, please see the manpage.

Testing

To test the operation of BogoSec and the reliability of its results, test cases have been executed, documented, and studied. Tests were executed against several popular packages against all released versions available for download. These tests demonstrate BogoSec's use against a given package to indicate the general trend of the quality of source code. Also, several equivalent open source software packages (when available) were tested to compare in an absolute sense which of the packages have better BogoSec quality scores.

Web Servers

Apache is the world's most popular Web server, accounting for over 60% of all internet Web sites. The Apache 1.3 tree has been under constant development since 1998 and up to the present. In that time, the Apache team has added many features and fixed many bugs. Apache 2.0 is the next generation Web server from the Apache development team.

Results

These test results show consecutive runs of BogoSec against all Apache 1.3 (in maroon) and 2.0 (in blue) released verisions available.
Absolute Points: The overall scores of 1.3 are very slowly but smoothly increasing. On the other hand, the tremendous spike between 2.0.18 and 2.0.28 (both beta releases) should cause some concern. Also, it looks like the 2.0.44 release fixed some security problems present in previous releases.
Points / Line: Both versions exhibit BogoSec ratios that are generally improving over time, though the 2.0 releases have better scores than the 1.3 branch.

Absolute PointsPoints / Line
						(continued)
Package	Version	Score	LOC	Points		Package	Version	Score	LOC	Points
apache	1.3.0	0.0896	186578	16711		httpd	2_0_15	0.0537	409238	21965 
apache	1.3.11	0.0811	256983	20841		httpd	2_0_16	0.0573	413056	23673 
apache	1.3.12	0.0812	257446	20895		httpd	2_0_18	0.0555	468691	26035 
apache	1.3.14	0.0809	267005	21595		httpd	2_0_28	0.0686	570302	39112 
apache	1.3.17	0.0809	270501	21889		httpd	2.0.32	0.0676	594210	40162 
apache	1.3.19	0.0799	271083	21654		httpd	2.0.35	0.0672	617261	41503 
apache	1.3.1	0.0879	190542	16753		httpd	2.0.36	0.0665	624265	41540 
apache	1.3.20	0.0797	272119	21681		httpd	2.0.39	0.0651	639129	41620 
apache	1.3.22	0.0814	276845	22546		httpd	2.0.40	0.0659	648812	42737 
apache	1.3.23	0.0808	280312	22660		httpd	2.0.42	0.0649	661539	42933 
apache	1.3.24	0.0798	285126	22761		httpd	2.0.43	0.0646	664596	42925 
apache	1.3.27	0.0796	287594	22900		httpd	2.0.44	0.0571	678224	38701 
apache	1.3.28	0.0790	290025	22923		httpd	2.0.45	0.0568	685602	38918 
apache	1.3.29	0.0791	290240	22965		httpd	2.0.46	0.0569	688417	39160 
apache	1.3.2	0.0888	195124	17329		httpd	2.0.47	0.0566	689951	39044 
apache	1.3.31	0.0817	284938	23287		httpd	2.0.48	0.0565	693187	39191 
apache	1.3.32	0.0816	285458	23305		httpd	2.0.49	0.0603	655544	39517 
apache	1.3.33	0.0816	285501	23305		httpd	2.0.50	0.0597	658322	39305 
apache	1.3.3	0.0889	196542	17479		httpd	2.0.51	0.0594	661497	39268 
apache	1.3.4	0.0871	202112	17596		httpd	2.0.52	0.0594	661740	39308 
apache	1.3.6	0.0880	206900	18209                                                 
apache	1.3.9	0.0806	247724	19963                                                 

Secure Shell Servers

OpenSSH provides an encrypted command shell, usually for remote network access to systems. OpenSSH is primarily developed by members of the OpenBSD project, a group of developers known for security-conscious code. OpenBSD conducts extensive manual audits of source code in order to identify and fix security vulnerabilities.

Results

These graphs demonstrate admirable models for secure software development. Both sets of data seem to approach asymptotes, with Absolute Points gradually increasing, and Points/Line scores gradually decreasing, and neither have significant spikes. The false positives reported by the tools probably form the asymptote base of these graphs.

Absolute PointsPoints / Line
						(continued)
Package	Version	Score	LOC	Points		Package	Version	Score	LOC	Points 
openssh	2.1.1p4	0.1329	92619	12307		openssh	3.2.2p1	0.0961	159646	15342  
openssh	2.2.0p1	0.1305	93999	12267		openssh	3.2.3p1	0.0961	159660	15342  
openssh	2.3.0p1	0.1252	108205	13549		openssh	3.3p1	0.0954	161720	15430  
openssh	2.5.1p1	0.1128	122159	13778		openssh	3.4p1	0.0952	161964	15416  
openssh	2.5.1p2	0.1126	122387	13778		openssh	3.5p1	0.0953	166756	15899  
openssh	2.5.2p1	0.1106	128989	14266		openssh	3.6.1p1	0.0945	170301	16101  
openssh	2.5.2p2	0.1106	129019	14266		openssh	3.6.1p2	0.0945	170316	16101  
openssh	2.9.9p1	0.1040	143107	14880		openssh	3.6p1	0.0946	170270	16101  
openssh	2.9.9p2	0.1040	143107	14880		openssh	3.7.1p1	0.0895	178101	15943  
openssh	2.9p1	0.1048	133867	14029		openssh	3.7.1p2	0.0894	178288	15943  
openssh	2.9p2	0.1048	133972	14040		openssh	3.7p1	0.0895	178084	15943  
openssh	3.0.1p1	0.1039	143342	14899		openssh	3.8.1p1	0.0864	181919	15722  
openssh	3.0.2p1	0.1039	143351	14899		openssh	3.8p1	0.0895	181852	16269  
openssh	3.0p1	0.1039	143319	14897		openssh	3.9p1	0.0869	186467	16203  
openssh	3.1p1	0.1018	144189	14683                                                  

FTP Servers

Vsftpd and wu-ftpd are two major open source FTP servers. The first, vsftpd, was written with security as a primary objective, with the "vs" standing for "very secure". The second, wu-ftpd, is Washington University's FTP server, which pre-dates vsftpd and has had a long and colorful history of some high profile security vulnerabilities.

Results

These charts show a drastic difference in source-code security between vsftpd and wu-ftpd. Both BogoSec scores of vsftpd appear orders of magnitude better than wu-ftpd. This is consistent with the popular opinion regarding the security of these two FTP servers.

Absolute PointsPoints / Line
						(continued)
Package	Version	Score	LOC	Points		Package	Version	Score	LOC	Points
vsftpd	0.9.2	0.0201	25656	515		wu-ftpd	2.0	0.2912	24345	7089  
vsftpd	1.0.1	0.0197	26149	515		wu-ftpd	2.1	0.3282	24334	7986  
vsftpd	1.1.0	0.0196	27784	544		wu-ftpd	2.2	0.3403	26388	8980  
vsftpd	1.1.1	0.0194	28029	544		wu-ftpd	2.3	0.3403	26385	8980  
vsftpd	1.1.2	0.0188	28868	544		wu-ftpd	2.4.2	0.2743	35509	9741  
vsftpd	1.1.3	0.0194	29228	568		wu-ftpd	2.4	0.3384	26495	8966  
vsftpd	1.2.0	0.0194	32849	637		wu-ftpd	2.5.0	0.2754	54752	15081 
vsftpd	1.2.1	0.0185	34409	637		wu-ftpd	2.6.0	0.3093	63603	19670 
vsftpd	1.2.2	0.0185	34540	637		wu-ftpd	2.6.1	0.3047	65322	19903 
vsftpd	2.0.0	0.0195	37607	734		wu-ftpd	2.6.2	0.3051	65212	19898 
vsftpd	2.0.1	0.0195	37657	734                                                   
vsftpd	2.0.2	0.0194	37883	734                                                   

Mail Transfer Agents

Postfix, qmail, and sendmail are major open source mail transfer agents (MTAs). Sendmail has long served as the primary mailer for UNIX® environments. But sendmail's history has been riddled with exploitable security vulnerabilities. Recently, several alternative MTA's have emerged, such as postfix and qmail. Postfix was written and is maintained by an expert member of IBM Research in secure computing. Qmail is another option with a notably small code base designed in the interest of security.

Results

These results are interesting in that the package that has the lowest absolute scores (qmail) does not have the lowest points/line ratios. This is because of the significantly smaller code base of qmail, and perhaps identifies an unfair BogoSec bias toward larger projects. (However, this is exactly why both metrics must be considered. The Postfix scores are very good, as expected.) A concerted effort by the sendmail development team between releases 8.11 and 8.12 demonstrates a marked BogoSec score improvement. Since that time, sendmail has maintained approximately the same status.

Absolute PointsPoints / Line
						(continued)
Package	 Version  Score	  LOC	  Points	Package	 Version  Score	  LOC	  Points
postfix	 1.0.8	  0.0241  199281  4800		sendmail 8.11.7	  0.0894  232285  20775 
postfix	 1.1.13	  0.0222  237491  5273		sendmail 8.12.10  0.0501  324450  16251 
postfix	 2.1.5	  0.0193  293908  5683		sendmail 8.12.11  0.0500  325152  16265 
qmail	 0.92	  0.0669  44957	  3008		sendmail 8.13.0   0.0494  330588  16330 
qmail	 1.00	  0.0660  44913	  2966		sendmail 8.13.1   0.0493  331157  16341	
qmail	 1.01	  0.0679  44918	  3050		sendmail 8.13.2   0.0493  331607  16335 
qmail	 1.02	  0.0693  45339	  3143          sendmail 8.13.3   0.0493  331647  16335 
qmail	 1.03	  0.0722  46737	  3377                                                  

Scripting Languages

Open source scripting languages include perl, php, python, and ruby. Each of these provide a higher level programming language easily used by developers to very easily and quickly accomplish software tasks. A binary interpreter is needed by each of these languages to executed the scripted source code.

Results

Ruby and Python have the lowest absolute and ratio scores. PHP, by far, has the largest code base and the highest absolute points. In all cases, the latest release shows marked improvement over the earliest release.*

Absolute PointsPoints / Line
					(continued)
Package	Version	Score	LOC	Points		Package	Version	Score	LOC	Points
perl	5.005	0.0502	278627	13982		Python	2.1.3	0.0315	628654	19797 
perl	5.6.1	0.0456	386938	17649		Python	2.2.3	0.0247	746199	18461 
perl	5.8.6	0.0457	523662	23908		Python	2.3.4	0.0263	895668	23539 
perl	5.9.0	0.0475	500034	23765		Python	2.4	0.0214	980119	20976 
perl	5.9.1	0.0484	498403	24145		ruby	1.3.5	0.0580	203286	11792 
php	3.0.18	0.0753	406914	30655		ruby	1.3.7	0.0578	203739	11781 
php	4.2.0	0.0512	876430	44854		ruby	1.3	0.0539	183631	9893  
php	4.2.3	0.0516	893882	46141		ruby	1.4.0	0.0578	204141	11799 
php	4.3.10	0.0441	1227476	54180		ruby	1.4.6	0.0572	207843	11887 
php	4.3.8	0.0446	1216570	54224		ruby	1.6.0	0.0561	219104	12299 
php	4.3.9	0.0444	1221208	54194		ruby	1.6.8	0.0510	240446	12267 
php	5.0.0	0.0361	1433945	51758		ruby	1.8.0	0.0421	396570	16687 
php	5.0.1	0.0358	1431329	51212		ruby	1.8.1	0.0394	426031	16778 
php	5.0.3	0.0357	1441051	51379		ruby	1.8.2	0.0353	492828	17405 

*More comprensive results for these tests available upon request. In order to create a readable chart, limited numbers of scores were chosen for each interpreter.

All Packages Tested

The following data shows an absolute comparison of the latest release of all packages tested above. One must be somewhat cautious of this comparison, because it's very difficult to compare vastly different packages especially when one package consists of a few hundred lines of code and another comprises millions of lines of code. Still, this chart teaches some valuable lessons about the advantages of BogoSec as well as perhaps some of its shortcomings.

Results

Absolute Points: It seems that vsftpd, qmail, and postfix are clear winners--these being packages designed and implemented by experts to be secure software. On the other end of the spectrum, httpd and php totaled the highest number of points--probably because of the fact that these are two of the largest packages that were tested.
Points / Line: Again, postfix and vsftpd exhibited the best scores, while wu-ftpd scored several orders of magnitude worse than the rest of the field--consistent with the popular opinion of its state of security.
All four of the scripting languages (python, php, ruby, perl) appear to have relatively similar BogoSec scores. It is somewhat surprising to see apache, httpd, and openssh near the upper end of the spectrum. This also deserves further investigation as it is possible that these packages are yielding an abnormally high number of false positives, thereby driving their scores disproportionately higher.

Absolute PointsPoints / Line
Package	 Version Score	LOC	Points
postfix	 2.1.5	 0.0193	293908	5683  
vsftpd	 2.0.1	 0.0195	37657	734   
Python	 2.4	 0.0214	980119	20976 
ruby	 1.8.2	 0.0353	492828	17405 
php	 5.0.3	 0.0357	1441051	51379 
perl	 5.9.1	 0.0484	498403	24145 
sendmail 8.13.3	 0.0493	331647	16335 
httpd	 2.0.52	 0.0594	661740	39308 
Qmail	 1.03	 0.0722	46737	3377  
apache	 1.3.9	 0.0806	247724	19963 
openssh	 3.9p1	 0.0869	186467	16203 
wu-ftpd	 2.6.2	 0.3051	65212	19898 

Applications

We hope that BogoSec drives developer awareness of insecure code by providing a higher level interface to numerous scanners. One way of encouraging developers to use these existing tools is by integrating BogoSec into some common development tools and processes, such as:

To demonstrate our point, we have created wrapper scripts that integrate BogoSec into the CVS source code repository on both the client and server side. While these example scripts use statically configured thresholds, it is conceivable to extend the scripts to check the score of previous commits and only allow new code that is at least as good as what is currently in the repository. In this way, one might try to ensure that the security quality of some code base never decreases.

We have also created a wrapper script that invokes the tool on a directory which may contain any number of file formats supported by BogoSec. The script will run BogoSec on each file and output the results in a tabulated format.

CVS Client

Proactive developers can make use of the script "client_cvs_wrapper" to be used in place of the "cvs commit" command. Defined in this script are two thresholds, MAX_SCORE and MAX_SEV_PTS. The BogoSec scores of the code to be committed must meet both of these requirements, or else the developer is notified to fix some of the suspected vulnerabilities. The user can configure the script to either entirely block the commit or simply issue a warning. Developers can use this script without bothering the server administrator.

CVS Server

Heavy-handed administrators can set MAX_SCORE and MAX_SEV_PTS thresholds in a script "server_cvs_wrapper", which is called by "CVSROOT/commitinfo". The wrapper informs the user of the BogoSec scores and exits with a non-zero return code when the code's scores do not meet requirements, thus preventing the code commit (or optionally issuing a warning).

BogoSec wrapper

This is an automated script that administrators may use to run BogoSec on a directory of different file formats to help them identify potential weak points of the system. The script will collect the results of the multiple BogoSec executions and arrange them in a concise and easy to read tabulated format { Package, Sev Points, Lines of Code, Final Score}.

Conclusions

References

Manual

BOGOSEC(1)                    BogoSec User Manual                   BOGOSEC(1)

NAME
       bogosec  -  source  code quality metric using established static source
       code scanners

SYNOPSIS
       bogosec [-l] [--log-dir directory ] [--min-sev 0-10 ] [--nhf] [-p  plu-
       gin_name  [args]  ]  [--plugin-dir  directory  ] [--sev-range-max num ]
       [--temp-log-dir directory ] [-v 0|1 ] [--xp plugin_name ] TARGET

       bogosec_wrapper TARGET-DIRECTORY

DESCRIPTION
       BogoSec attempts to influence developers to produce more secure  source
       code  over  time.  Various existing scanners point developers to poten-
       tially insecure sections of code. BogoSec broadens the scope of  source
       code scans by utilizing multiple independent scanners and compiling the
       results into high-level calculated metrics.   These  metrics  can  help
       developers  and users alike to comparatively judge the security quality
       of source code.

       bogosec_wrapper automates the process by running bogosec on a direcotry
       containing  different  file formats supported by bogosec and collecting
       the results in : /tmp/bogosec.results  and  /tmp/bogosec.detail_results
       (overwritten  by  each  invocation). Keep in mind, bogosec_wrapper does
       not accept any options.

OPTIONS
       -l     Turn on scanner  output  logging.  Log  will  be  called  <scan-
              ner_name>.log  and  created in current working directory, unless
              --log-dir is used to specify a different location.

       --log-dir directory
              Specify a directory for scanner output logs (only makes sense if
              -l is also used).  Default is current working directory.

       --min-sev minimum_severity_level
              Specify  a  minimum severity level. Any vulnerabilities reported
              by the scanners whose score falls  below  this  number  will  be
              ignored.  The argument must be a number 0-10.  Default is 0.

       --nhf, --no-header-files
              Do not scan header files.  Useful if the scanners being used do
              not support scanning header files.

       -p, --plugin plugin_name [args]
              Specify a plugin to use. If no plugins are  defined  on
              the  command line, all of the plugins in the plugins_dir will be
              used. This option can be passed more than once to specify a set
              of scanners to use. Each scanner requires a separate instance of
              the --plugin flag (please see examples). Optionally,  a  set  of
              command  line arguments can be passed to the scanner - this fea-
              ture must be used with care.   Keep  in  mind  that  the  plugin
              requires  a  certain formatting of the scanner output (for exam-
              ple, '-SQ' is always passed to flawfinder, and '-w 3' is  always
              passed to rats).  You can pass additional command line arguments
              using this option, but be aware of the effect it might  have  on
              the  formatting  of the scanner output, and the effect that will
              have on the plugin's ability to parse it correctly.  If you must
              change the defaults ('-SQ', '-w 3', etc.) you must edit the plu-
              gin directly.

       --plugin-dir directory
              Specify the directory where the plugins are stored.  Default  is
              /usr/local/bogosec/plugins.

       --sev-range-max number
              Specify the maximum severity value to be used in calculating the
              severity value range.  The default is 10.  For example,  setting
              --sev-range-max to 50 would mean that the severity results would
              now be on a scale of 0-50 instead of on a scale of 0-10.  This  can
              be  used  to  scale  the result if more granularity is required.
              NOTE: -v 1 will not work if this option is used.

       --temp-log-dir directory
              Specify a directory where you want the temporary files  used  by
              BogoSec  to be stored (scanner output logs, etc.)  The default is
              /tmp/.

       -v, --verbosity 0|1
              Specify verbosity level (default is 0).  If 1, then  a  graph  of
              the  severity  points is shown, which breaks the results down by
              severity levels. This option does not work if  the  --sev-range-
              max is changed from 10.

       --xp, --exclude-plugin plugin_name
              Do not run plugin defined by plugin_name.

FILES
       /etc/bogosec.conf
              Global  configuration file. The settings here are overwritten by
              any settings in user's ~/.bogosecrc file.
       ~/.bogosecrc
              Default user  configuration  file  (overrides  the  settings  in
              /etc/bogosec.conf).   This file is not created during an instal-
              lation, you must create it yourself.
       /usr/local/bogosec/plugins/
              Default plugins directory.  Can  be  changed  with  --plugin-dir
              option.   Plugins must be executable, and must end in .pm as per
              convention.
       /usr/local/bogosec/documents/
              Directory of BogoSec documentation and other germane  documents.

BUGS
       Not  all  input  validated.   Not  all environmental variables checked.
       This program expects to be run by trusted users.

AUTHORS
       Developed by Dustin Kirkland, Agoston Petz, and Loulwa Salem at the IBM
       Linux Technology Center.

Linux                             Jan 25 2004                       BOGOSEC(1)