AN EMPIRICAL ANALYSIS OF VULNERABILITY DISCLOSURE POLICIES. Research in Progress Submission to WISE 2010 Total Word Count: 3409

Transcription

1 AN EMPIRICAL ANALYSIS OF VULNERABILITY DISCLOSURE POLICIES Research in Progress Submission to WISE 2010 Total Word Count: 3409 Sabyasachi Mitra College of Management Georgia Institute of Technology Atlanta, Georgia Sam Ransbotham Carroll School of Management Boston College Chestnut Hill, Massachusetts September 2010

2 INTRODUCTION The importance of information security in the current business environment can hardly be overstated. Recent research indicates that security concerns are major impediments to the widespread adoption of electronic commerce and that equity markets penalize the announcement of software vulnerabilities (Telang and Wattal 2007). Furthermore, recent legislation such as Sarbanes-Oxley mandate penalties for inadequate security. Clearly, information security is no longer a purely technical issue and effective economic frameworks and incentives are becoming as important as technical design in protecting systems (Anderson and Moore 2006, p. 610). Attackers often compromise computer systems by exploiting vulnerabilities present in the software running on these systems (Cavusoglu et al. 2007, Cavusoglu et al. 2008). The impact of a software vulnerability depends on whether the software vendor and security professionals have the opportunity to eliminate the vulnerability or otherwise protect systems before they are attacked. Consequently, the discovery and disclosure process for vulnerabilities plays a vital role in information security. There is considerable debate on the design of effective disclosure processes that advantage security professionals and disadvantage attackers, but there is no consensus on the optimal design. When a vulnerability is discovered by attackers, they either exploit the vulnerability or sell the vulnerability to other attackers on the black market. In both cases, the vulnerability is first exploited before it is observed in the wild by security professionals. On the other hand, there are two primary methods that security professionals use to disclose vulnerabilities. First, security professionals may choose Immediate Disclosure they publicly disclose the vulnerability immediately through security mailing lists such as Bugtraq. When disclosed through immediate disclosure, the vulnerability information is immediately disseminated to a wide audience of security professionals who can install countermeasures, to vendors who can develop patches, as well as to potential attackers who can exploit the information to their advantage. Second, security professionals may choose Non-public Disclosure they report the vulnerability to organizations like CERT (Computer Emergency Response Team). CERT, for example, immediately notifies the software vendor and discloses the vulnerability to the public when a patch is available from the vendor, or after a specific period (typically 45 days after notifying the vendor). In non-public disclosure, security service providers and potential attackers receive notification at the time of public disclosure, while vendors are notified in advance. A current and significant debate in the security industry revolves around the benefits and drawbacks of immediate disclosure. The dominant viewpoint, termed as Responsible Disclosure, encourages disclosure through CERT and other similar mechanisms that provide a reasonable time for the vendor to develop patches. The basic motivation behind responsible disclosure, which is supported by many software companies, security vendors and security organizations such as CERT, is that the alternative immediate disclosure creates an unsafe period when the vulnerability may be exploited before the patch is developed and deployed. Proponents of responsible disclosure therefore argue that responsible disclosure will lead to lower attack volume, more protected systems, and a safer security environment. On the other hand, immediate disclosure is often motivated by the need to force unresponsive vendors to address a vulnerability and to create incentives for developing secure software (Arora et al. 2006, Arora et al. 2008). Proponents argue that immediate disclosure will lead to more responsive software vendors and more alert security service providers, and consequently a safer information security environment. In this paper, we shed light on this overall debate through a large-scale empirical study that compares vulnerabilities disclosed through the immediate disclosure and non-public disclosure mechanisms. Specifically, we gauge the impact of immediate disclosure by analyzing over 2.4 billion information security alerts for 960 clients of an US based security service provider. We examine two measures of impact: (a) attack diffusion does immediate disclosure accelerate the diffusion of attacks corresponding to the vulnerability through the population of target systems and increase the number of affected systems, and (b) attack volume does immediate disclosure increase the volume of attacks that are based on the 1

3 vulnerability? Diffusion speed is important because it allows vendors to release a patch and for security service providers to protect systems before they are attacked, while attack volume measures of the amount of malicious activity (Park et al. 2007). There are two primary contributions of our research to the literature on optimal policies and methods to ensure the security of information systems. First, while several analytical models in the literature examine optimal vulnerability disclosure and patching policies (Arora et al. 2006, Arora et al. 2008, August and Tunca 2006, August and Tunca 2008, Cavusoglu, et al. 2007), this research is one of a few that empirically evaluates a contemporary vulnerability disclosure phenomenon through the examination of intrusion detection system (IDS) data, providing needed diversity in research methods. Second, while economic models based on rational choice form the basis of the published research in this area (Arora et al. 2008, Cavusoglu et al. 2007, Kannan and Telang 2005), we develop our hypotheses through a review of the innovation diffusion literature (Rogers 2003), providing additional diversity in the theoretical lenses used to study the phenomenon. Finally, we empirically evaluate a research question that is of significant practical importance whether immediate disclosure has a detrimental effect on security. We believe that our findings are of significant practical interest to policy makers and vendors. MODELING THE DIFFUSION OF ATTACKS We model the diffusion of attacks through the population of target systems through the familiar s- curve that has been extensively used to model the diffusion of innovations in the literature (Rogers 2003). Let N(t) be the cumulative number of target systems affected at time t where t is measured from the time the vulnerability is disclosed. Let P be the height of the s-curve, or the maximum number of target systems in the population affected by the vulnerability (referred to as penetration of the diffusion process). D is the time when P/2 systems are affected by the vulnerability (the s-curve reaches half of its ultimate penetration level) and captures the delay associated with the diffusion process. R is the slope of the s-curve which is dependent on factors such as the type of vulnerability, complexity of developing exploits, and the impact of the vulnerability on systems. N(t) is modeled using the following familiar form of the s-curve. (1) IMMEDIATE DISCLOSURE: A CONTRARIAN VIEW The dominant view in the information security community is that immediate disclosure will lead to a less secure environment because public disclosure of the vulnerability can lead to systems being attacked before the vendor provides a patch. We provide a contrarian view of immediate disclosure that focuses on the role of security service providers and the race between attackers who exploit vulnerabilities and security service providers who install countermeasures. When a patch is not available or installed, specific countermeasures can provide partial protection against attacks. For example, Ransbotham and Mitra (2009) describe three types of countermeasures in systems that limit the impact of a vulnerability: (a) access control methods that limit access to the affected software to specific groups, (b) feature control methods that disable functionality and features in the affected software and devices, and (c) traffic control methods that filter suspicious traffic based on signature based attack detection. Countermeasures are easier to implement than patches, but they provide temporary and imperfect protection until the core vulnerability is removed through patching or a software upgrade. Our basic argument is that immediate disclosure induces a race between attackers who attack systems and security service providers who develop and install countermeasures to protect systems. This race, which is similar in concept to a patent race in the economics literature (Denicolo 2000), accelerates the diffusion process of attacks because attackers are aware of the vulnerability at the time of disclosure. However, like in a patent race, this race also raises urgency among security service providers and 2

4 accelerates the development and deployment of countermeasures. Consequently, the time window for successful exploitation by attackers is small until countermeasures are installed, and the vulnerability has a short life span. This leads to a lower penetration level of attacks among the population of target systems since many target systems have countermeasures installed and the population of vulnerable systems rapidly decreases. The short life span of the vulnerability and its lower penetration levels among target systems reduce the overall volume of attacks as attackers divert their attention to more profitable opportunities. This forms the basis of the following three primary hypotheses: H1: The diffusion of attacks through the population of target systems will have less delay for vulnerabilities reported through immediate disclosure than through non-public disclosure. H2: The diffusion of attacks through the population of target systems will have reduced penetration for vulnerabilities reported through immediate disclosure than through non-public disclosure. H3: The volume of attacks will be lower for vulnerabilities reported through immediate disclosure than through non-public disclosure. DATA Our primary data source is a proprietary database of alerts generated from intrusion detection systems (IDS) installed in client firms of a security service provider. Each time the IDS detects a signature in an incoming data stream, it generates an alert for further analysis. The dataset provides a unique research opportunity because it contains real alert data (as opposed to data from a research setting) from a large number of clients with varied infrastructure across many industries. The alert database contained over four hundred million alerts generated during 2006 and Our analysis is based on a panel dataset of the number of alerts generated every day during the two-year period of our analysis, summarized by target firm and specific vulnerability. Our second main data source is the National Vulnerabilities Database (NVD 2008) that combines several other public vulnerability data sources such as CERT, Bugtraq, XForce and Secunia. We believe that ours is the first study that combines the NVD data with actual intrusion detection data from a large number of firms to empirically evaluate a contemporary information security issue. We match the signatures for each unique vulnerability in our intrusion alert database with detailed information available through the NVD. The matching is done through a CERT assigned unique ID that links our two databases together. It is important for our analysis that we insure that the effects we see are due to immediate disclosure and not due to characteristics of the vulnerability itself. Thus, we use the following variables from the NVD data as controls in our empirical analysis. Once the attacker has access, vulnerabilities require varying degrees of complexity to exploit and are categorized by experts as Low or High Complexity. We also include an indicator variable (Sig) that is set to 1 if a signature was available at the time that the vulnerability was disclosed, 0 otherwise. Because disclosure through Bugtraq is immediate, we include an additional variable (Immediate) to capture the effects of immediate disclosure. The impact of a vulnerability is categorized by experts into one or more categories, and we use an indicator variable for each impact category that is set to 1 if the potential for the specific impact is present, 0 otherwise. The NVD classifies vulnerabilities into seven different types based on the specific software flaw that the vulnerability represents, and we used indicator variables to control for vulnerability type. We also include an indicator variable (Patch) that is set to 1 if a patch was available on the focal day of analysis, 0 otherwise. We also include the Age of the vulnerability (log transformed) at the time of our analysis, measured as the number of days since the vulnerability was reported. RESULTS To evaluate H1 and H2, we estimate equation (1) through non-linear least squares estimation of parameters. In (1), the variables P, R and D are allowed to vary as a function of focal (Immediate) and other control variables. The results are reported in Table 1. Based on the estimated parameters, we find 3

5 that immediate disclosure reduces delay (D) of diffusion (accelerates the diffusion process) and decreases penetration (P) of attacks based on the vulnerability. Thus, we find support for H1 and H2. The results from our evaluation of H3 are reported in Table 2. The dependent variable is the number of attacks (log transformed) on a specific date for a specific client and for a specific vulnerability. Table 2 reports results from a two-stage Heckman selection model that incorporates selection bias in the data since many vulnerabilities in the NVD data are never exploited in our sample. The coefficient of the Immediate variable is negative and significant, indicating that immediate disclosure reduces the volume of attacks. Thus, we find support for H3. Interestingly, we also find that public availability of an attack signature increases penetration of attacks and increases the number of attacks, indicating that the signature contains information that the attacker can utilize to build tools and exploit the vulnerability. Contrary to expectations, the public availability of a signature also increases the delay associated with the attack diffusion process and more research is needed to understand the reasons behind this coefficient estimate. Some of the other variables in the models also provide interesting insights. For example, vulnerabilities that require complex execution methods (e.g. social engineering) have delayed diffusion processes and lower attack volumes. SUMMARY AND IMPLICATIONS Contrary to the dominant view in the security industry and the practitioner literature, we find that immediate disclosure of vulnerabilities reduces delay in the attack diffusion process (as expected), but also reduces penetration of attacks in the population of target systems and reduces the volume of attacks. Our results can be explained by viewing the attack process as a race between attackers who attack systems and security service providers who develop countermeasures, similar to a patent race that has been examined in the economics literature (Denicolo 2000). This race accelerates the attack diffusion process, but also increases awareness, forces security service providers to be more vigilant, accelerates the deployment of countermeasures, and reduces the window of opportunity for attackers before countermeasures are installed. Our results have two important implications for policy makers, security organizations such as CERT, and software vendors. First, limited public disclosure of vulnerability information can combine the benefits of non-public and immediate disclosure to skew the race towards securing systems. For example, organizations such as CERT can immediately disclose the vulnerability to security service providers (as well as the software vendor) so that they can develop countermeasures to protect systems until a patch is made available by the software vendor. This will provide an advantage to security service providers in the attack and countermeasures race without publicly disclosing the signature and other attack details. This limited disclosure to security service providers is particularly important since our results indicate that public disclosure of signatures increases attack penetration and attack volume. Second, while immediate disclosure causes security service providers to be more vigilant and limits the penetration level and volume of attacks based on the vulnerability, it is possible (and perhaps even likely) that the effect on those who are not protected through such services is in the opposite direction as attackers focus their attention on such targets in the absence of others. Also, a similar diversion-based argument applies to vulnerabilities not disclosed through immediate disclosure. In general, the attack and countermeasures race for immediate disclosure vulnerabilities may cause security service providers to focus less on other (perhaps more critical) vulnerabilities. Overall, our analysis and results indicate that the effects of different disclosure methods are complex and nuanced, and represent a fruitful area of further research with important practical implications. FUTURE RESEARCH We intend to provide a more complete analysis of the effect of immediate disclosure through additional empirical evaluation. We envision two immediate directions for further analysis: (a) is 4

6 immediate disclosure and the consequent race more effective for certain types of vulnerabilities, and (b) is there a diversionary and negative effect of immediate disclosure and the consequent race on other vulnerabilities? Both of the above analyses can be performed through the intrusion detection data available to us such as by interacting the Immediate variable with other focal and control variables, and evaluating changes in attack volume of other vulnerabilities subsequent to the date of immediate disclosure of the focal vulnerabilities examined here. In addition to the above, the intrusion detection data can be used to empirically evaluate the findings of various analytical models in this area. For example, Ransbotham et al. (2008) evaluates the impact of disclosure through market based mechanisms such as idefense and Tipping Point (Kannan and Telang 2005). The data set can be used to evaluate the antecedents of attacks for a firm (Ransbotham and Mitra 2009), or whether different patching policies affect the number of attacks (August and Tunca 2008). Additionally, the data set can be used to evaluate the impact of social, political and other events on attack activity. Space limitations do not allow us to describe many of our findings here. In summary, linking the intrusion detection data with the NVD database provides a rich data source to evaluate various information security related debates of significant practical importance. Anderson, R., T. Moore The Economics of Information Security. Science. 314(5799) Arora, A., J.P. Caulkins, R. Telang Sell First, Fix Later: Impact of Patching on Software Quality. Management Science. 52(3) Arora, A., R. Telang, X. Hao Optimal Policy for Software Vulnerability Disclosure. Management Science. 54(4) August, T., T.I. Tunca Network Software Security and User Incentives. Management Science. 52(11) August, T., T.I. Tunca Let the Pirates Patch? An Economic Analysis of Software Security Patch Restrictions. Information Systems Research. 19(1) Cavusoglu, H., H. Cavusoglu, S. Raghunathan Efficiency of Vulnerability Disclosure Mechanisms to Disseminate Vulnerability Knowledge. IEEE Transactions on Software Engineering. 33(3) Cavusoglu, H., H. Cavusoglu, J. Zhang Security Patch Management: Share the Burden or Share the Damage? Management Science. 54(4) Denicolo, V Two-Stage Patent Races and Patent Policy. RAND Journal of Economics. 31(3) Kannan, K., R. Telang Market for Software Vulnerabilities? Think Again. Management Science. 51(5) Nvd National Vulnerability Database. Park, I., R. Sharman, H.R. Rao, S. Upadhyaya Short Term and Total Life Impact Analysis of Worms in Computer Systems Decision Support Systems Ransbotham, S., S. Mitra Choice and Chance: A Conceptual Model of Paths to Information Security Compromise. Information Systems Research. 20(1) Ransbotham, S., S. Mitra, J. Ramsey Are Markets for Vulnerabilities Effective? Proceedings of the Twenty-ninth International Conference on Information Systems Rogers, E.M Diffusion of Innovations. The Free Press, New York, NY. Telang, R., S. Wattal An Empirical Analysis of the Impact of Software Vulnerability Announcements on Firm Stock Price. IEEE Transactions on Software Engineering. 33(8)

7 TABLE 1: Diffusion Analysis of Attacks Based on Immediate and Non-Public Disclosure P (Penetration) R (Rate) D (Delay) Constant (3.5941)*** (0.0008)*** (0.2565)*** Patch Available (0.9488) (0.0003)* (0.0915)*** High Complexity (1.2958)*** (0.0004)*** (0.0928)*** Signature Available (3.4006)*** (0.0014)*** (0.1442)*** Immediate (0.9367)*** (0.0003)*** (0.0942)*** Impact Indicators Included Included Included Type Indicators Included Included Included No. of Observations 132,768 Adjusted R % 132,768 daily observations of 333 vulnerabilities from Robust standard errors in parentheses; 2 tailed significance: * p<.05; ** p<.01; *** p<.001 Nonlinear regression on number of firms affected, where the cumulative penetration (P), the rate of diffusion (R) and delay (D) are linear functions of the variables shown in the table. TABLE 2: Volume of Alerts per Client Firm per Vulnerability Number of Alerts (log) Model 0 (Controls) Model 1 (Full Model) Constant (0.101)*** (0.101)*** Age of Vulnerability (ln) (0.002)*** (0.002)*** Patch Available (0.002)*** (0.003)*** High Complexity (0.003)*** Signature Available (0.003)*** Immediate (0.002)*** Vulnerability Impact Indicators Indicators Vulnerability Type Indicators Indicators Alert Month Indicators Indicators Selection Stage Constant (0.008)** (0.008)*** Patch Available (0.003)*** (0.003)*** High Complexity (0.004)*** Signature Available (0.004)*** Immediate (0.003)*** Vulnerability Impact Indicators Indicators Vulnerability Type Indicators Indicators Publication Month Indicators Indicators Wald X 2 (x10 6 ) 1.20*** 1.18*** Heckman two stage regression; n = 1,302,931; 709,090 uncensored; 333 vulnerabilities; standard errors in parenthesis. Two-tailed significance: * (p<0.05); ** (p<0.01); *** (p<0.001) 6