An Empirical Analysis of Vendor Response to Vulnerability Disclosure

1. An Empirical Analysis of Vendor Response to Vulnerability Disclosure Ashish Arora, Ramayya Krishnan, Rahul Telang, Yubao Yang Carnegie Mellon University

2. Motivation • Information security breaches: A significant and increasing threat • Lack of systematic policy for how vulnerability information should be disclosed Self reported security incidences

3. Motivation • While theoretical models are useful to understand the issues surrounding vulnerability disclosure, we need empirical estimates for policy making. • One of the key factors is to understand how vendors respond to disclosure and disclosure policies? • An empirical estimate on vendor response to disclosure window will be very useful in calibrating the current policies. However, data collection is non-trivial.

4. Research goals • Whether (and by how much) early disclosure induces vendors to patch faster. • What are other key factors that condition patching time?

5. Literature • Arora, Telang, and Xu (2003) outline a model for the optimal policy for software vulnerability disclosure. • Telang and Wattal (2004) show that disclosure is costly to vendors and hence provides incentives to vendors to improve the quality of their software • Market based mechanism • Camp and Wolfram (2004) describe a means for creating market for vulnerabilities in order to increase the security of systems • Kannan and Telang (2004) show that markets always perform worse that CERT because of poor disclosure rules • Schechter (2002) argues that vendors should create and exploit a market for testers • Ozment (2004), an auction based market based mechanism

6. Predictions of Analytical Model (Arora, Telang and Xu [2003]) • Vendors face cost of patching. More time they have for patching less it costs them. • Vendors’ customer incur loss when they are breached. Depending on the market structure, vendors “internalize” some of the customer loss. The more loss they internalize, more costs they incur and earlier is the patch. • Disclosure of vulnerability is potentially hurtful to customers because disclosure makes it easier for hackers to find the information too. Thus disclosure threat supposedly forces vendors to patch faster because disclosure increases their costs. • However, there is little (if any) empirical evidence that vendors indeed patch faster and by how much.

7. Model Prediction • Besides understanding the role of disclosure, we also investigate other factors that have bearing on vendor response. Some of the factors are • Severity of the vulnerability • Vendor characteristics • Open source / closed source • Disclosure source • Publicly traded firm • Effect of September 11.

8. Data • Vulnerabilities published by SecurityFocus or CERT/CC. • Information on the key time variables (Patching time = Date of patch – Date of notification). CERT provided us with information on when they notified the vendors. The date on which vendors delivered a patch to them etc. • Vendor information from Hoover’s online business information database and vendor’s website • Vulnerability information from the NIST ICAT database • Time period from 9/26/2000 to 8/11/2003 • 1280 observations, related to 255 unique vendors and 303 unique ICAT database documented vulnerabilities

9. CERT/CC Vs SecurityFocus • Two major vulnerability disclosure sources • CERT/CC (A Federal supported R&D center) • Typically 45 days of secret period after notifying vendors • No exploit code disclosed • SecurityFocus (An online open forum) • Policy of instant disclose (many time individuals may provide vendors some time before disclosure) • Disclose full information • We discard all vulnerabilities which are reported first by vendors

10. Early disclosure • Anytime vulnerability is disclosed within the disclosure window (mostly 45 days) and vendor has not patched, early disclosure happens. However, in our sample most of the time disclosure happens quite early. • Instant disclosure is a case when disclosure happens before or at the same time when vendor is notified of the vulnerability. • “Not early” case on SecurityFocus • Identifiers tend to be careful in using this powerful instant disclosure tool. They inform the vendor first and wait for the vendor patch before posting on SecurityFocus website • 30% in our sample • “Early” case on CERT/CC • Disclosure by others in CERT/CC secret time period • Already known public when CERT/CC picked it up • A vendor was missed when CERT/CC notify other vendors • Disclosure before 45 days if 80% of the vendors are ready

11. Impact of instant disclosure Impact of publication source

12. Impact of disclosure source (for Instantly disclosed vuls) • Disclosure by CERT has a significant impact on patching speed of the vendor than disclosure by Securityfocus or by other sources

13. Vendor Characteristics There are total 255 unique vendors. Above statistics is based on the 121 vendors that we have reliable information. Vulnerability Characteristics • There are total 301 unique vulnerabilities. • Average Severity Score was 16.25. • Each vulnerability affected on an average 11 vendors.

14. Analysis • Two sets of analysis • Impact of disclosure on patching time. • Conditional on not having patched until time t-1, how will disclosure at time t will affect vendor’s patching speed. • We choose different values of t. • Impact of expected “disclosure window” on patching time. • How will change in disclosure window affect vendors’ patching behavior?

16. Results • Disclosure accelerates the patch delivery significantly. For vulnerabilities that are disclosed instantly, patch comes 55% faster than otherwise. • When disclosure happens later the patch still comes significantly faster but the difference between with and without disclosure patching speed seems to reduce. • Open source vendors tend to patch faster; almost 44% faster. • Significant impact of 9/11. Patches come faster post 9/11.

17. Impact of Disclosure Source

19. Impact of Disclosure Window “T” • We now want to understand what is the impact of disclosure window on patching time. This is the information a policy maker like CERT needs. Before they decide how much time should be given vendors, they need to know what is impact of giving one additional day. • CERT provides approximately 45 days. However, it is clear the most of the time disclosure happens much earlier. This means that expected disclosure window “T” is much smaller and is unobservable to econometrician. • But we know that for all vulnerabilities that are disclosed instantly, T = 0. For all other, T>0. Thus these two samples should provide us with the directional effect of “T” on patching time.

20. Impact of disclosure window “T” • We use only CERT data to analyze this because CERT has a more well defined policy. • We test whether there is significant difference between patching times for vulns instantly disclosed and otherwise in the CERT sample.

22. Results • Vendors are 56% faster when T = 0 compared to when T > 0. • On an average the disclosure happens in our sample in 20 days. • If we believe that the effect is linear then on an average, one day decrease in the disclosure window increases the patching speed by 2.8%.

23. Conclusions • We find that disclosure has significant and expected result on vendor’s patching behavior. • There is a significant CERT effect. Involvement of CERT leads to faster patching time irrespective of disclosure. • Open source vendors patch faster; more severe vulnerabilities are patched faster and there is a significant post 9/11 effect.