Evolutions in Browser Security

Transcription

1 ANALYST BRIEF Evolutions in Browser Security TRENDS IN BROWSER SECURITY PERFORMANCE Author Randy Abrams Overview This analyst brief aggregates results from NSS Labs tests conducted between 2009 and 2013 in a comparison of phishing and socially engineered malware (SEM) protection by the leading browsers. Figure 1 reveals trends in protection levels of the four leading browsers, comparing combined test results from 2009 to the recent 2012 and 2013 scores. 100% % 80% Average Phish 2012 Malware 70% 60% 50% Chrome Firefox IE 40% 30% Average Malware 2009 Safari Average Phish 20% Average Malware 10% % 0% 20% 40% 60% 80% 100% Phishing Figure 1 Leading Browser Malware and Phishing Block Rates (2009, 2012, 2013)

2 In the NSS tests, the browsers are rated on performance in four categories: Average phishing block rate Zero- hour phishing block rate Average SEM block rate Zero- hour SEM block rate Internet Explorer (IE) shows a consistently superior ability to block SEM, while providing competitive phishing protection; it leads the tested browsers in combined protections for these categories. Google s Download Protection technology has improved significantly over time, placing it behind IE but well ahead of Firefox and Safari. Both Firefox and Safari lead the other browsers in phishing protection but provide negligible protection against SEM attacks. This analyst brief includes data from previously published NSS phishing tests conducted in 2009, 2012, and 2013, as well as SEM tests published every year from 2009 to In 2010, two SEM tests were published. Figure 2 shows the overall performance of the browsers for the 2013 browser phishing and SEM tests. IE 89% Chrome 76% Safari 53% Firefox 52% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure Combined Phishing And Malware Block Rates Figure 2 treats all protection metrics in 2013 equally. Later in this analyst brief, Figure 15 will add weighting based on the relative protection importance of the tested parameters in order to provide a more realistic ranking of the browsers in Figure 14 provides an aggregation of all of the tests from 2009 to 2013, with emphasis placed on freshness and relative importance of the test metrics. 2

3 NSS Labs Findings The browser is the first line of defense against multiple web- based threats; however, with a maximum historical protection rate of just 80 percent, the browser should not be the only line of defense. Products that do not provide the bulk of their protection in the earliest hours of an attack are not meeting the security requirements of today s threatscape. Microsoft s Internet Explorer continues to provide the best combination of malware and phishing protection. The application reputation technologies used by browsers from both Microsoft and Google provide a significantly safer browsing experience than do the browsers from Apple and Mozilla. User education is often better protection against social engineering attacks than browser technologies. NSS Labs Recommendations Invest in awareness education about social engineering for all users. Evaluate trends that may indicate the need for browser replacement. Select and use security products that augment the protective capabilities of the browser. 3

4 Table of Contents Overview... 1 NSS Labs Findings... 3 NSS Labs Recommendations... 3 Analysis... 6 Phishing Trends and Threats... 6 NSS Empirical Results: Phishing Protection... 7 NSS Empirical Results: Socially Engineered Malware Protection... 9 Combined Protection Effectiveness Aggregate Values Weight A Minute Evaluating the Data The Great Equalizer Appendix A: Raw Data Appendix B: Weighting Formulas Protection Multipliers: Year Multipliers: Reading List Contact Information

5 Table of Figures Figure 1 Leading Browser Malware and Phishing Block Rates (2009, 2012, 2013)... 1 Figure Combined Phishing And Malware Block Rates... 2 Figure 3 Unique Phishing Attacks... 6 Figure 4 APWG Phishing Uptime Statistics... 7 Figure 5 Mean Block Rate for Phishing... 7 Figure 6 Zero- Hour Phishing Block Rate... 8 Figure 7 Time to Block Phishing Attacks Relative to Uptime Trends (Hours)... 9 Figure 8 Mean Block Rate for Socially Engineered Malware... 9 Figure 9 Zero- Hour Socially Engineered Malware Block Rate Figure 10 Content Agnostic Malware Protection Breakout Figure 11 Combined Test Results (Not Weighted) Figure 12 Time Weighted Figure 13 Protection Weighted Figure 14 Time and Protection Weighted Scores Figure Weighted Scores Figure 16 Mean Block Rate for Phishing Figure 17 Zero- Hour Block Rate Figure 18 Mean Block Rate for SEM Figure 19 Zero- Hour Block Rate for SEM Figure 20 Time Weighting Figure 21 Protection Type Weighting Figure 22 Type and Year Weighting

6 Analysis Socially engineered malware and phishing attacks are two of the most significant threats against which web browsers must defend. NSS has for several years tested the leading browsers for their ability to protect against these attacks; however, each test has been presented as a stand- alone snapshot in time. While these real- world snapshot tests yield useful information, a correlated report is equally valuable in order to assess trends and establish vendor track records. This analyst brief examines the historical performance of browsers against phishing and against socially engineered malware attacks. The browsers are evaluated against each other and against the phishing threatscape. If the best performing product affords little protection, then the worst performing product is not significantly different. Fortunately, there are browsers that are addressing the challenges and that are able to provide significant protection for users. Phishing Trends and Threats The Anti- Phishing Working Group (APWG) has collected and published statistics about phishing attacks for several years. The APWG Phishing Attack Trends 1 and Global Phishing Survey 2 reports provide important insight into the phishing problem. From Figure 3 it can be inferred that although the number of unique phishing s and web sites has varied from 2009 to 2012, the scope of the problem remains significant. The uptick in unique phishing sites discovered in 2012 is not accompanied by a significant uptick in reports of unique phishing s. While this may be due to under- reporting of phishing , it is likely indicative of cyber criminals increasingly using redirects in an attempt to compensate for the declining lifetimes of their attacks. 700, , , , , , , Unique Phishing s Reported Unique Phishing Web Sites Discovered Figure 3 Unique Phishing Attacks 1 reports/ 2 reports/whitepapers 6

7 One of the critical metrics surrounding a browser s effectiveness in combatting phishing attacks is how quickly it adds protection once an attack is live. Figure 4 illustrates the general decline in the lifetimes of phishing sites. In 2012, the average phishing site was live for just under 25 hours, and the median lifetime was approximately 12 hours. Products that do not provide the bulk of their protection in the earliest hours of an attack are not meeting the security requirements of today s threatscape H2009 2H2009 1h2010 2h2010 1h2011 2h2011 1h2012 2h2012 Average Uphme (Hours) Median Uphme (Hours) Figure 4 APWG Phishing Uptime Statistics NSS Empirical Results: Phishing Protection NSS tested the leading browsers for phishing protection in 2009, 2012, and 2013, with the results presented in Figure 5 and Figure % 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Figure 5 Mean Block Rate for Phishing Chrome Firefox IE Safari Average In 2009, only IE and Firefox provided competitive block rates for phishing, with results in 2012 narrowing the differences to a four- point spread between browsers. In 2013, Firefox and Safari posted modest improvements in their scores, while Chrome dropped by 2 percent. IE has a trend of solid performance, but the browser s phishing block rate declined from 92 percent in the 2012 Browser Security Comparative Analysis Report (CAR) on Phishing Protection to 83 percent in the 2013 CAR on Phishing Protection. 7

8 The lower results in the 2013 test should be evaluated against future tests to determine if the decline in IE s mean block rate in the 2013 test indicates a problematic trend. The time required to add new phishing sites is an important metric when determining the relevance of the mean block rate to consumer protection. For example, a browser that blocks more phishing sites in the first 12 hours will provide better protection than a less responsive browser that achieves a better block rate in the long run. Historically, NSS testing has found that the browsers with the best early detection continue to lead until the end of the test; however, this may not always be the case. 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Figure 6 Zero- Hour Phishing Block Rate Chrome Firefox IE Safari Average During the 2013 test, all browsers showed improvement over their historical zero- hour block rates, as depicted in Figure 6. A metric that has a high correlation to the zero- hour block rate is the average time required to add protection for new phishing sites. Figure 7 shows the APWG average phishing site uptime statistics and the mean phishing site uptime statistics overlaid with the results of the tested browser performance for average time to add protection for new phishing sites. (The APWG statistics for the first half of 2013 were not available at the time of writing.) Note that the browser phishing protection tests were performed only in 2009, 2012, and 2013, so performance in 2010 and 2012 is graphed in a linear fashion and may not reflect actual performance in those two years. Safari is the only browser to have had a worse response time to phishing attacks than either the mean or the average uptime for phishing sites since NSS began testing browsers. But Apple has dramatically improved its performance, and Safari posted the fastest response times in the most recent test report. The median uptime for phishing attacks is significantly lower than the average response time and is the more important metric. All of the browsers are adding protection very quickly, with IE requiring 2.6 hours and Safari averaging 30 minutes. Firefox, however, has demonstrated the most consistent protection for phishing over time. 8

9 Chrome Firefox IE Safari APWG Average Uphme APWG Median Uphme Figure 7 Time to Block Phishing Attacks Relative to Uptime Trends (Hours) NSS Empirical Results: Socially Engineered Malware Protection Socially engineered malware (SEM) refers to an attack that deceives users into downloading and installing malicious software. In recent years, rogue antivirus programs have been at the forefront of SEM; however, there are many types of malicious programs that criminals use in conjunction with social engineering for financial gain. Figure 8 demonstrates that there are, and historically have been, dramatic differences in browser protection against SEM. 100% 80% 60% 40% u Google Buys VirusTotal 20% 0% 2009 Q Q Chrome Firefox IE Safari Figure 8 Mean Block Rate for Socially Engineered Malware NSS conducted six browser malware protection tests between 2009 and 2013, and IE significantly outperformed the competition in all six texts. Only recently has Chrome become a viable option that provides significant malware protection for users. While Chrome, Firefox, and Safari all use Google s Safe Browsing API, Chrome alone incorporates Google s Content Agnostic Malware Protection technology (CAMP). Prior to 2012, all three of the browsers using Google s Safe Browsing API performed comparably. 9

10 When Google acquired VirusTotal, it was widely assumed that the service would be used to improve Chrome s malware blocking abilities. The 2013 results do show an improvement in Chrome s mean block rate, but also a significant drop in zero- hour protection. If Google continues to improve Chrome s SEM protection, it may be difficult to differentiate the contribution of the VirusTotal acquisition from ongoing investment in Google s Download Protection technology. As with phishing attacks, response time is critical when providing protection against malware attacks. Figure 9 shows browser performance graphed at zero hour against SEM attacks. IE significantly outperforms the competition in all six tests. Both Firefox and Safari have declined in SEM protection since % 80% 60% 40% 20% 0% u Google buys VirusTotal 2009 Q Q Chrome Firefox IE Safari Figure 9 Zero- Hour Socially Engineered Malware Block Rate Google s Download Protection technology and Microsoft s App Rep technologies are the reason that Chrome and IE are able to block such high percentages of SEM. Neither Chrome nor IE relies on the certain knowledge that a file is bad; rather, they block files that do not meet reputational criteria IE 88.5% 10.6% 2012 Chrome 4.5% 65.8% 2013 IE 83.2% 16.8% 2013 Chrome 10.0% 73.2% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% URL Reputahon Applicahon Reputahon Download Protechon Figure 10 Content Agnostic Malware Protection Breakout Figure 10 displays the combination of SEM- blocking technologies used by Chrome and IE. Both URL and CAMP protection methodologies can suffer from false positives; however, the more important consideration is the protection that URL reputation adds over CAMP. 10

11 When a web page contains both exploits and SEM, URL reputation will protect the user from exploits, in addition to SEM. Where CAMP is the only protection mechanism, the user can still fall victim to exploits. Consequently, IE s strong use of URL reputation compared to Chrome s use renders IE s SEM protection significantly more valuable. Combined Protection Effectiveness Aggregate Values The purpose of combining scores to arrive at a single value is to allow for the reuse of this value in conjunction with other metrics in order to select the browser that best balances selection criteria. Phishing and SEM protection, the metrics used in this brief, are a part of overall browser security. The number, severity, and longevity of exploits against a browser constitutes a metric, and security of stored passwords is yet another metric. Privacy protection capabilities, such as those discussed in the analyst brief 2013 Browser Security Comparative Analysis: Privacy, can be combined with vulnerability metrics as well as other performance metrics in order to make educated product selection decisions based on the combined performance across all aspects of interest. Weight A Minute Not all protection metrics are equal. The majority of phishing attacks will fail because the intended victim is not a customer of the targeted brand. A customer of Barclays is not going to fall victim to a phish against Wells Fargo customers. A user without a Gmail account will not surrender credentials if they receive a phish targeting Gmail. SEM attacks are brand agnostic and can even customize the payload for specific operating systems. A larger percentage of SEM attacks will succeed compared to phishing attacks. As such, protection against SEM is of greater importance than phishing protection. For both phishing and SEM, time is of the essence. Zero- hour SEM blocking is a more important metric than overall SEM blocking. Overall SEM blocking is of more significance than zero- hour phishing protection, and the mean block rate for phishing ranks lowest in the protection hierarchy. The data regarding the various block rates is empirical; however, the weighting of relative security values is subjective and the importance of protection metrics may vary based on other layers of protection in different environments. With the exception of clearly unrealistic weighting values, significant fluctuations in weights assigned to protection categories and test dates will not materially alter the rankings of the browsers. A scientific approach to weighting, if possible, would not alter significantly the results of the graphs. The raw data for the various NSS tests are provided in Appendix A. The formulas used to weight the scores and create the weighted figures are provided in Appendix B. Evaluating the Data Figure 11 depicts the rankings of the browsers without ranking the importance of different protection metrics: that SEM protection is significantly more important than phishing protection, or that newer tests are more relevant than older tests, and so on. The relative importance of protection types is only one of the variables that require weighting. The freshness of tests is critical. Old tests help assess a track record; however, browsers that have added new technologies are improperly impacted when older tests are weighted too highly. Equal value for older tests also paints an unreliable picture of browsers that are declining in protection ability. 11

12 Figure 11 is not weighted, and it shows Firefox as out performing Chrome in the combined tests. In 2009, Firefox outperformed Chrome by 26 percent across the four metrics. In 2012 and 2013, Chrome eclipsed Firefox by 26 percent and 25 percent respectively. Clearly, an accurate ranking of the browser requires more recent scores to carry more weight than older results. IE 79% Firefox Chrome 46% 49% Safari 37% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 11 Combined Test Results (Not Weighted) Leaving protection scores unmodified and assigning older tests progressively less weight, Figure 12 shows Chrome in second place with a significant lead over Firefox and Safari. IE 83% Chrome 66% Firefox Safari 56% 54% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 12 Time Weighted Figure 13 does not weight for time; however, the different types of protection are weighted based on relative importance. The resulting graph does not adequately reflect performance improvements in Chrome with respect to SEM protection in 2012 and Weighting values can be found in Appendix B. 12

13 IE 80% Chrome Firefox 36% 39% Safari 27% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 13 Protection Weighted IE 85% Chrome 58% Firefox Safari 40% 39% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 14 Time and Protection Weighted Scores Figure 14 incorporates weighting that places a higher value on newer tests and a higher value on more important protection categories. IE 92% Chrome 71% Safari Firefox 37% 35% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure Weighted Scores Both Figure 14 and Figure 15 display a wider performance margin between IE and Chrome, as well as between Chrome and the other browsers than does Figure 2. Figure 15 does not consider trends or track records. If current protection value, track records, and trends are considerations, then Figure 14 provides a more comprehensive picture than does Figure

14 The Great Equalizer Both phishing and SEM are social engineering attacks. By definition, these are social problems, and technology has rarely solved a social problem. Technology can help to mitigate problems, but education is paramount. For users who are adept at identifying social engineering attacks, the browser adds little additional security; however, most users are not aware of the dynamics of social engineering and will fall prey to SEM even when they are able to identify many types of phishing attacks. Proper education provides the best protection against most social engineering attacks. 14

15 Appendix A: Raw Data The figures in Appendix A provide the raw data from NSS testing used to create all of the figures in this analyst brief, with the exception of Figure 3, Figure 4, Figure 10, and the APWG metrics in Figure 7. The APWG metrics in Figure 3, Figure 4, and Figure 7 are derived directly from APWG published reports. The percentages used in Figure 10 are published in the 2012 and 2013 Browser Phishing Protection CARs. Browser Chrome 26% 94% 92% Firefox 80% 90% 96% IE 83% 92% 83% Safari 2% 91% 95% Figure 16 Mean Block Rate for Phishing The mean block rate for phishing reflects overall phishing detection for the duration of each test. Browser Chrome 16.00% 53.20% 81.50% Firefox 48.00% 79.20% 93.30% IE 52.00% 55.90% 73.30% Safari 2.00% 76.90% 93.40% Figure 17 Zero- Hour Block Rate The zero- hour block rate is a critical metric. The value of a higher mean block rate can be marginalized by ineffective zero- hour performance. Due to the diminishing uptimes of phishing sites, a browser with a lower mean block rate and better zero- hour response times may provide more effective protection for most users than will a browser with a better overall block rate but poorer zero- hour performance. Browser 2009 Q Q Chrome 16% 17% 3% 13% 70% 83% Firefox 30% 29% 19% 8% 4% 10% IE 69% 85% 99% 99% 99% 100% Safari 24% 29% 11% 8% 4% 10% Figure 18 Mean Block Rate for SEM The mean block rate for SEM reflects the SEM performance for the duration of each test. Throughout a test, various browsers can fluctuate significantly in their instantaneous block rate. Histograms in NSS CARs provide additional detail. 15

16 Browser 2009 Q Q Chrome 25% 15% 4% 10% 67% 49% Firefox 28% 28% 18% 7% 6% 8% IE 41% 58% 89% 99% 86% 98% Safari 13% 27% 10% 6% 5% 12% Figure 19 Zero- Hour Block Rate for SEM The zero- hour block- rate is the percentage of malware each browser was already blocking when the hosting site was first discovered. Browsers with higher zero- hour protection generally provide better protection than browsers with delayed protection times. 16

17 Appendix B: Weighting Formulas The following multipliers were used in calculating weighted scores. Protection Multipliers: Mean Block Rate for Phishing = Score *.3 Zero Hour Phishing Block Rate = Score *.5 Mean SEM Block Rate = Score *.8 Zero Hour SEM Block Rate = Score * 1 Year Multipliers: 2009 = Score * = Score * = Score * = Score * = Score * 1 Figure 20 displays scores weighted for date and not protection type. Figure 21 displays scores weighted for protection type but not for the dates of the test. Figure 22 displays the combined protection type and date weighting. Protection Mean Block Rate - Phish 10% X X 80% 100% Zero- Hour Block Rate - Phish 10% X X 80% 100% Mean Block Rate - SEM 10% 20% 40% 80% 100% Zero- Hour Block Rate - SEM 10% 20% 40% 80% 100% Figure 20 Time Weighting Protection Mean Block Rate - Phish 30% X X 30% 30% Zero- Hour Block Rate - Phish 50% X X 50% 50% Mean Block Rate - SEM 80% 80% 80% 80% 80% Zero- Hour Block Rate - SEM 100% 100% 100% 100% 100% Figure 21 Protection Type Weighting Protection Mean Block Rate - Phish 3% X X 24% 30% Zero- Hour Block Rate - Phish 5% X X 40% 50% Mean Block Rate - SEM 8% 16% 32% 64% 80% Zero- Hour Block Rate - SEM 10% 20% 40% 80% 100% Figure 22 Type and Year Weighting 17

18 The maximum time and protection type weighted scores attainable for four protection types are as follows: Mean Block Rate - Phish: ((100*.03)+(100*.24)+(100*.3))/3=.19 or 19%. Zero Hour Block Rate - Phish: ((100*.05)+(100*.4)+(100*.5)/3=.3167 or 31.67%. Mean Block Rate - SEM: ((100*.08)+(100*.16)+(100*.16)+(100*.32)+(100*.64)+(100*.8))/6=.36 or 36%. Zero Hour Block Rate - SEM: ((100*.1)+(100*.2)+(100*.2)+(100*.4)+(100*.8)+(100*.1))/6=.45 or 45%. The maximum combined total score is therefore: ( )/4=.3292 or 32.92%. To normalize to a 100 percent scale, the total weighted scores are divided by For Chrome, the total weighted performance for all of the tests would be calculated as follows: (((((((0.26*0.03)+(0.94*0.24)+(0.92*0.3))/3)+(((0.16*0.05)+(0.53*0.4)+(0.82*0.5))/3)+(((0.16*0.08)+(0.17*0.16)+(0. 03*0.16)+(0.13*0.32)+(0.7*0.64)+(0.83*0.8))/6)+(((0.25*0.1)+(0.15*0.2)+(0.04*0.2)+(0.1*0.4)+(0.67*0.8)+(0.49))/ 6))/4)))/.3292 Chrome - Mean Block Rate - Phish: ((.26*.03)+(.94*.24)+(.92*.3))/3=17% Chrome - Zero Hour Block Rate - Phish: ((.16*.05)+(.53*.4)+(.82*.5)/3=21% Chrome - Mean Block Rate - SEM: ((.16*.08)+(.17*.16)+(.3*.16)+(.13*.32)+(.7*.64)+(.83*.8))/6=20% Chrome - Zero Hour Block Rate - SEM: ((.25*.1)+(.15*.2) +(.04*.2)+(.1*.4)+(.67*.8)+(.49*1))/6=.19% Chrome - ((17%+21%+20%+19%)/4)/.3292=58% 18

19 Reading List 2013 Browser Security Comparative Analysis Report: Phishing Protection. NSS Labs browser- security- comparative- analysis- phishing- protection 2012 Browser Security Comparative Analysis Report: Phishing Protection. NSS Labs browser- security- comparative- analysis- phishing- protection User Education Effectiveness Can Be Measured. NSS Labs education- effectiveness- can- be- measured 2013 Browser Security Comparative Analysis: Privacy. NSS Labs 19

20 Contact Information NSS Labs, Inc. 206 Wild Basin Rd Building A, Suite 200 Austin, TX USA +1 (512) This analyst brief was produced as part of NSS Labs independent testing information services. Leading products were tested at no cost to the vendor, and NSS Labs received no vendor funding to produce this analyst brief NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors. Please note that access to or use of this report is conditioned on the following: 1. The information in this report is subject to change by NSS Labs without notice. 2. The information in this report is believed by NSS Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this report are at the reader s sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this report. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON- INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader s expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners. 20