Microsoft-prevalence-based analysis of the File Detection Test

Commissioned by Microsoft Microsoft-prevalence-based analysis of the File Detection Test Supplementary Report to the File Detection Test of March 2014 Language: English March 2014 Last Revision: 7 th August 2014 Commissioned by Microsoft Commissioned by Microsoft - 1 -

Introduction This report has been commissioned by Microsoft and is based on telemetry data provided by Microsoft. This is a first prototype customer-impact report; more enhanced versions might be provided for future File-Detection Test reports. The essence of this report is as follows. Of the prevalent samples used in the malware-detection test, some will pose a greater threat to the average user than others as they are more widespread than others. Some may target e.g. a specific company or user base, but present less of risk to other users. Other malware samples may only be found on specific websites, affect specific countries/regions, or only be relevant to particular operating system versions or interface languages. Microsoft s initiative uses its global telemetry data (malware prevalence) to consider the customer impact posed by missed detections. That is, the malware files that antimalware products failed to detect are weighted based on malware-family prevalence, and each vendor s prevalence-weighted results are reported along with the file-detection results in this report. These results are designed to give greater insight into the customer impact of the missed detections during testing. In addition to global prevalence weighting impact, geo-location prevalence is also used to determine the customer impact of missed detections in specific countries for products tested. This is used to present the filedetection efficacy of antimalware products in the test against prevalent malware samples. This report is supplementary to AV-Comparatives main report 1, already published, of the March 2014 File-Detection Test. No additional testing has been performed; rather, the existing test results have been re-analysed from a different perspective, to consider what impact the missed samples are likely to have on customers. It is conceivable that a product with a lower score in the test may actually protect the average user better than one with a higher score, under specific circumstances. Let us imagine that Product A detects 99% of malware samples in the test, but that the 1% of samples not detected are very widespread, and that the average user is quite likely to encounter them. Product B, on the other hand, only detects 98% of samples, but the samples missed are either not as prevalent, or only run on a specific operating system. In this case, users would probably be more at risk using Product A, as it misses more of the malware that is likely to present a threat to them. AV-Comparatives has for many years focused on using prevalent samples in its tests, as mentioned in our reports and also in a Microsoft blog 2. Furthermore, same sample variants (e.g. polymorphic malware) are clustered to avoid a disproportional test-set 3. AV-Comparatives makes uses of telemetry data from various sources, not just Microsoft, as the test-set must remain independent and not based solely on data provided by one specific vendor or organisation. Therefore, minor discrepancies between one vendor s data and our independently sorted combination are possible. The original File- Detection Test in March 2014 used a malware set sorted using various telemetry sources; however, the analysis in this supplementary report is based solely on Microsoft s data. 1 http:///wp-content/uploads/2014/04/avc_fdt_201403_en.pdf 2 http://blogs.technet.com/b/mmpc/archive/2010/06/15/update-on-telemetry-usage-in-tests-part-1.aspx 3 http://blogs.technet.com/b/mmpc/archive/2009/07/16/let-telemetry-be-your-guide-a-proposal-for-securitytests.aspx Commissioned by Microsoft - 2 -

Tested products: The following products tested in March 2014 are included in this report: 1. AhnLab V3 Internet Security 8.0 2. avast! Free Antivirus 2014 3. AVG Internet Security 2014 4. AVIRA Internet Security 14.0 5. Baidu Antivirus 2014 (en) 6. Bitdefender Internet Security 2014 7. BullGuard Internet Security 14.0 8. Emsisoft Anti-Malware 8.1 9. escan Internet Security 14.0 10. ESET Smart Security 7.0 11. F-Secure Internet Security 2014 12. Fortinet FortiClient 5.0 13. Kaspersky Internet Security 2014 14. Kingsoft Internet Security 2013.SP6 15. Lavasoft Ad-Aware Free Antivirus+ 11.1 16. McAfee Internet Security 2014 17. Microsoft Security Essentials 4.4 18. Panda Cloud Free Antivirus 2.3 19. Qihoo 360 Internet Security 4.9 (en) 20. Sophos Endpoint Security 10.3 21. Tencent QQ PC Manager 8.9 22. ThreatTrack Vipre Internet Security 7.0 23. Trend Micro Titanium Internet Security 2014 The test-set used was built consulting telemetry data from various sources (not only Microsoft), with the aim of including mainly prevalent malicious samples from the last weeks/months prior the test which posed a threat to users in the field. Detection vs. Protection Although very important, the file-detection rate of a product is only one aspect of a complete antivirus product. Almost all antivirus products contain features such as URL-blockers and behavioural protection that protect the user s computer without necessarily identifying every malicious file. AV-Comparatives also provides a whole-product dynamic real-world protection test 4, as well as other test reports that cover these aspects/features of the products. We invite users to look at our other tests and not only the File-Detection Test, even though a good file-detection rate is still one of the most important, deterministic and reliable basic features of an anti-virus product. 4 http:///dynamic-tests/ Commissioned by Microsoft - 3 -

Methodology This analysis was carried out using AV-Comparatives file-detection test data from March 2014. Telemetry data was gathered for the files in the test over the period between December and April 2014. This telemetry came from Microsoft real-time protection (RTP) products and included not only threat telemetry but also behaviour-based early warning telemetry. Only computers whose users have agreed to provide data to Microsoft are considered when calculating encounter rates. This data was used to generate a weighting to represent customer impact for the files used in the test. This weighting was generated by first counting the total number of distinct computers identified through a unique product GUID (not IP address) that encountered the malware files in the test. Furthermore, a weighting for the malware family associated with the specific files in the test was generated from computers running Microsoft RTP products to assess total customer impact for that family in March 2014. Family Weight Calculation The family weighting was calculated by dividing the number of Microsoft RTP computers that encountered each family by the entire number of RTP computers that encounter any high or severe malware during the period. The family mapping used is based on the naming 5 given by Microsoft at a specific point in time (time of testing). This weight was then applied to misses for that family in the test set (see calculation details below). Finally, these weightings were summed to give the final weighted score of each vendor. Descriptions and Information about malware families can be found in the Microsoft Malware Protection Center s Malware Encyclopedia http://www.microsoft.com/security/portal/threat/threats.aspx Vendor impact calculation The total vendor impact calculation is generated by first measuring the machines encountering the missed files per vendor per family, which we call the MissedFileFamilyMachineCount. The FamilyWeight for each family is then applied to the MissedFileFamilyMachineCount and the results are summed over all families, which produces the NonNormalizedVendorImpact. 5 http://www.microsoft.com/security/portal/mmpc/shared/malwarenaming.aspx Commissioned by Microsoft - 4 -

The WholeTestImpact is then calculated to normalize the NonNormalizedVendorImpact. The WholeTestImpact is calculated by first measuring the machines encountering the all files per family in the test set, which we call the WholeTestFileFamilyMachineCount. The FamilyWeight for each family is then applied to the WholeTestFileFamilyMachineCount and the results are summed over all families, which produces the WholeTestImpact. Finally, the VendorImpact is calculated by dividing the NonNormalizedVendorImpact by the WholeTestImpact. Country Vendor Impact calculation Country-based vendor impacts were calculated in a similar way to the total vendor impact calculation outlined above: The MissedFileFamilyMachineCount per vendor per family from above was weighted using a the FamilyWeightByCountry, which is a geo-specific family weighting for each country and malware family. This weighted result produces the NonNormalizedVendorImpact ByCountry. Finally, the WholeTestImpactPerCountry was calculated by applying the FamilyWeightByCountry to the WholeTestFileMachineCount PerFamily described above. This was then used to normalize the NonNormalizedVendorImpact ByCountry to produce the final VendorImpactByCountry. Commissioned by Microsoft - 5 -

Microsoft generated telemetry information on many of the files that they did not have detection for (10% of the test-set was not detected by Microsoft), so for those files they had customer impact numbers as well. For the files for which Microsoft did not have customer impact numbers, Microsoft used the average customer impact of the low-prevalence files in the test. Therefore, a generic family has been created and an average family weighting has been used to calculate the impact of malware samples not detected by Microsoft. This report should be regarded as a prototype, the purpose of which is spark debate on the significance of prevalence data, and promote ideas for improving the method. For upcoming prevalence analysis reports, the goal would be to translate third-party detections into Microsoft families to increase the accuracy. This would involve other vendors working closely with Microsoft to understand how various vendors count prevalence and to investigate how to roll in telemetry from other vendors, in order to produce a better picture. AV-Comparatives encourage vendors to share their telemetry data with Microsoft, in order to get a more significant and impartial customer-impact analysis. Microsoft regularly reviews and refines its data collection methodology to improve its scope and accuracy. For this reason, the statistics presented in this report may differ from comparable statistics in future reports. Microsoft uses relativly few family names, of which about 1300 have been seen in-the-field, and about 600 with currently at least 1 machine reporting the family in question. The test-set of AV- Comparatives consisted of about 1250 different families according to Microsoft s family mapping. When Microsoft gains new information on malware samples and families, Microsoft s family-based detection may change. About 2% of Microsoft detected samples were later mapped to different families from those to which they had originally been assigned. Commissioned by Microsoft - 6 -

Test-Set description The test-set used in March 2014 for the File-Detection Test contained 125977 malware samples. The number of encounters caused by the malware samples used in the test was according to Microsoft s telemetry data around 3,361,802. Over 3 million unique machines were affected. The world map below shows the countries in which the malicious files had the biggest impact. There are over 150 countries of the world for which Microsoft have data for less than 100,000 computers. These are considered to be too small to be statistically relevant the margin of error is too high to accurately represent the population of Internet users in the country. These appear as white on the map. The impact on the remaining 91 countries (~94% of machine population) are shown as blue in the map. Top 30 impacted countries: 1. India 10.1% 2. Mexico 6.6% 3. Turkey 6.6% 4. United States 5.4% 5. Philippines 4.9% 6. Russian Federation 4.8% 7. Indonesia 4.8% 8. Brazil 3.7% 9. Pakistan 3.1% 10. Vietnam 2.6% 11. Ukraine 2.5% 12. Iran 1.8% 13. Colombia 1.8% 14. Argentinia 1.8% 15. Thailand 1.8% 16. Malaysia 1.6% 17. Peru 1.5% 18. United Kingdom 1.4% 19. Italy 1.3% 20. Egypt 1.3% 21. France 1.2% 22. Kazakhstan 1.1% 23. Venezuela 1.0% 24. Germany 0.9% 25. Algeria 0.8% 26. Ecuador 0.8% 27. Poland 0.8% 28. Canada 0.8% 29. Chile 0.8% 30. Saudi Arabia 0.8% Commissioned by Microsoft - 7 -

Distribution of Malware Types in the test-set Top 20 Malware Families in the test-set 1. Vobfus 6.5% 2. Zbot 5.5% 3. Bladabindi 4.2% 4. VBInject 4.1% 5. Urausy 4.0% 6. Injector 3.3% 7. CeeInject 3.3% 8. Dorkbot 2.9% 9. Sality 2.8% 10. Winwebsec 2.6% 11. Dynamer 2.5% 12. Expiro 2.3% 13. Siresef 2.1% 14. Ramnit 2.1% 15. Beebone 2.1% 16. Virut 1.6% 17. Malagent 1.5% 18. Gamarue 1.5% 19. Autorun 1.3% 20. Obfuscator 1.2% Top 20 Malware Families in the test-set with highest Customer Impact according to Microsoft Malware Family 1. Iethic 2. Dorkbot* 3. Gamarue* 4. Zbot* 5. Sirefef 6. Necurs* 7. Kilim* 8. Cutwail 9. [Undefined]* 10. Kuluoz Machine Count 991747 661301 529802 226961 186885 183147 154041 149391 147179 118035 % of all machines 30% 20% 16% 7% 6% 5% 5% 4% 4% 4% Malware Family 11. Upatre 12. CeeInject 13. Fareit 14. Malagent* 15. Meredrop 16. VBInject 17. Bladabindi* 18. Caphaw 19. Dynamer* 20. Miuref Machine Count 115015 114558 112026 107182 102037 92370 91838 88503 83282 82841 % of all machines 3% 3% 3% 3% 3% 3% 3% 3% 2% 2% The Top Missed Families in the Top20 Malware Families in the test-set with highest Customer Impact are marked with an asterisk. Commissioned by Microsoft - 8 -

Detection Rates and Customer Impact Based on the missed samples and the detection rate over the whole test-set, Microsoft have calculated the normalized Customer Impact. This can be seen in the table below. The different colors in the table illustrate products scoring better than the baseline (Microsoft). Customer Impact (normalized) Missed machines out of 1000 Missed Samples 6 100% -Missed Samples 1. Kaspersky Lab 0.001209 1.209 0.2% 99.8% 2. McAfee 0.001220 1.220 0.7% 99.3% 3. AVIRA 0.001980 1.980 0.8% 99.2% 4. Tencent 0.001996 1.996 0.8% 99.2% 5. Panda 0.002253 2.253 0.7% 99.3% 6. ThreatTrack Vipre 0.005283 5.283 1.1% 98.9% 7. Fortinet 0.005724 5.724 0.4% 99.6% 8. Sophos 0.005938 5.938 1.7% 98.3% 9. ESET 0.006411 6.411 1.2% 98.8% 10. AVG 0.007327 7.327 2.5% 97.5% 11. F-Secure 0.008212 8.212 0.4% 99.6% 12. Trend Micro 0.011808 11.808 1.0% 99.0% 13. escan 0.011861 11.861 0.4% 99.6% 14. Bitdefender 0.012304 12.304 0.5% 99.5% 15. Lavasoft 0.012934 12.934 0.5% 99.5% 16. Kingsoft 0.012936 12.936 0.5% 99.5% 17. Emsisoft 0.013018 13.018 0.5% 99.5% 18. Qihoo 0.013025 13.025 0.5% 99.5% 19. BullGuard 0.013027 13.027 0.5% 99.5% 20. Avast 0.014447 14.447 2.3% 97.7% 21. Baidu 7 0.018903 18.903 0.4% 99.6% 22. AhnLab 0.055918 55.918 11.0% 89.0% * Microsoft is represented as the baseline with the following results: customer impact (normalized) of 0.003369, missed 3.369 machines out of 1000, missed 10% of samples, or 100% - missed sample percentage of 90%. For some few vendors there is a significant difference between the number of missed samples and the Customer Impact. In some cases, a high number of missed files did not translate into a high Customer Impact. This indicates that the samples detected in the test were those with higher prevalence, while those that were not detected had lower prevalence. Conversely, in some cases vendors with a low number of missed samples had a relatively high Customer Impact. In these cases, the samples that were missed were higher prevalence samples, which had a greater impact on the customer population. This supplementary report suggests that using prevalence to weight malicious file detection tests can produce useful insights into the efficacy of antimalware products against the most common risks. That is, when file-detection test results are weighted by malware family prevalence it is possible to get a better idea of the risk that customers face when using antimalware products. Furthermore, it shows that detection of highly prevalent malware is valuable in reducing the risk to customers, and is a useful factor for detection prioritization. 6 About 77% of the test-set was detected by all products in the test, as the samples are well-known/common. 7 According to the data and analysis from Microsoft, Baidu has a high impact score, because in the few misses it had, it missed high-prevalence files that belonged to prevalent families this is why it looks like an outlier. Commissioned by Microsoft - 9 -

Customer Impact Report March 2014 Heat-Maps Overview The interactive heat maps can be found on http://impact.av-comparatives.org TEST-SET AhnLab Avast AVG AVIRA Baidu Bitdefender BullGuard Emsisoft escan ESET Fortinet F-Secure Kaspersky Lab Kingsoft Lavasoft McAfee Microsoft Panda Qihoo Sophos Tencent ThreatTrack Vipre Trend Micro The heat maps for each vendor, i.e. the coloured maps of the world show data that is normalised by the relative size of the country. Thus the maps represent the countries with the highest risk relative to the prevalence of files that were missed in the test set. This normalisation differs from the heat map displayed in the Test-Set Description; that map is normalised based on the prevalence of the entire test set to show the prevalence of the files that were used in the test set. As a consequence, the scale on the vendor-specific heat maps and the colours shows are not directly comparable to the test-set description heat map. Commissioned by Microsoft - 10 -

AhnLab V3 Internet Security 8.0.8.2 1. Czech Republic 13293 in 100000 2. Slovak Republic 12027 in 100000 3. China 11653 in 100000 4. Finland 10958 in 100000 5. Netherlands 10402 in 100000 6. Belgium 10111 in 100000 7. South Korea 10026 in 100000 8. Estonia 9975 in 100000 9. Sweden 9814 in 100000 10. Denmark 9781 in 100000 11. France 9651 in 100000 12. Portugal 9595 in 100000 13. Hungary 9560 in 100000 14. Norway 9524 in 100000 15. Brazil 9129 in 100000 16. Slovenia 9089 in 100000 17. Taiwan 8893 in 100000 18. Spain 8871 in 100000 19. Greece 8866 in 100000 20. Iceland 8767 in 100000 Global Non-Detection Risk: 5592 in 100000 1. Kishop 2. Bladabindi 3. Dorkbot 4. Lamrek 5. Tesch Commissioned by Microsoft - 11 -

avast! Free Antivirus 2014.9.0.2013 1. Ireland 6169 in 100000 2. Estonia 5779 in 100000 3. Uruguay 5749 in 100000 4. Iceland 5713 in 100000 5. France 5658 in 100000 6. Belgium 5026 in 100000 7. Brazil 4834 in 100000 8. Norway 4612 in 100000 9. Germany 4568 in 100000 10. New Zealand 4477 in 100000 11. Denmark 4400 in 100000 12. Finland 4035 in 100000 13. Hong Kong 3778 in 100000 14. Japan 3708 in 100000 15. Bulgaria 3609 in 100000 16. Italy 3588 in 100000 17. South Korea 3453 in 100000 18. Switzerland 3431 in 100000 19. Canada 3419 in 100000 20. Austria 3414 in 100000 Global Non-Detection Risk: 1445 in 100000 1. Brantall 2. Necurs 3. Malagent 4. Kuluoz 5. Tesch Commissioned by Microsoft - 12 -

AVG Internet Security 2014.0.4335 1. Czech Republic 1998 in 100000 2. Slovak Republic 1672 in 100000 3. China 1522 in 100000 4. Iraq 1386 in 100000 5. Ghana 1256 in 100000 6. South Korea 1246 in 100000 7. Portugal 1236 in 100000 8. Israel 1233 in 100000 9. Saudi Arabia 1209 in 100000 10. Tunisia 1198 in 100000 11. Estonia 1163 in 100000 12. Finland 1107 in 100000 13. South Africa 1099 in 100000 14. Taiwan 1083 in 100000 15. Turkey 1081 in 100000 16. Qatar 1079 in 100000 17. Egypt 1050 in 100000 18. Morocco 1033 in 100000 19. Hungary 1032 in 100000 20. Venezuela 1022 in 100000 Global Non-Detection Risk: 733 in 100000 1. Balamid 2. Meredrop 3. Bladabindi 4. Dorkbot 5. Necurs Commissioned by Microsoft - 13 -

AVIRA Internet Security 14.0.3.350 1. Finland 721 in 100000 2. France 638 in 100000 3. Netherlands 624 in 100000 4. Sweden 618 in 100000 5. Czech Republic 596 in 100000 6. Belgium 585 in 100000 7. Denmark 580 in 100000 8. Spain 575 in 100000 9. Norway 530 in 100000 10. Germany 526 in 100000 11. Slovak Republic 514 in 100000 12. Austria 503 in 100000 13. Italy 470 in 100000 14. China 466 in 100000 15. Portugal 446 in 100000 16. Switzerland 440 in 100000 17. Estonia 436 in 100000 18. Greece 426 in 100000 19. Taiwan 425 in 100000 20. Ireland 420 in 100000 Global Non-Detection Risk: 198 in 100000 1. Sefnit 2. Dynamer 3. Bladabindi 4. VBInject 5. Kishop Commissioned by Microsoft - 14 -

Baidu Antivirus 4.0.9.57999 (en) 1. Ireland 7135 in 100000 2. Uruguay 7105 in 100000 3. France 6562 in 100000 4. Estonia 6119 in 100000 5. Iceland 5864 in 100000 6. Belgium 5843 in 100000 7. Norway 5738 in 100000 8. Brazil 5735 in 100000 9. New Zealand 5547 in 100000 10. Denmark 5289 in 100000 11. Germany 5218 in 100000 12. Chile 4703 in 100000 13. Finland 4531 in 100000 14. Costa Rica 4297 in 100000 15. South Korea 4218 in 100000 16. Bulgaria 4141 in 100000 17. Italy 4119 in 100000 18. Greece 3999 in 100000 19. Jamaica 3883 in 100000 20. Switzerland 3878 in 100000 Global Non-Detection Risk: 1890 in 100000 1. Brantall 2. Dorkbot 3. Vundo 4. Kilim 5. Sefnit Commissioned by Microsoft - 15 -

Bitdefender Internet Security 17.26.0.1106 1. Ireland 4959 in 100000 2. Estonia 4751 in 100000 3. Uruguay 4614 in 100000 4. France 4422 in 100000 5. Iceland 4395 in 100000 6. Belgium 4019 in 100000 7. Brazil 3697 in 100000 8. Norway 3685 in 100000 9. Germany 3639 in 100000 10. New Zealand 3559 in 100000 11. Denmark 3392 in 100000 12. Bulgaria 3330 in 100000 13. Finland 3111 in 100000 14. Serbia 2992 in 100000 15. Cyprus 2946 in 100000 16. Greece 2889 in 100000 17. Hungary 2830 in 100000 18. Italy 2806 in 100000 19. Jamaica 2790 in 100000 20. Switzerland 2735 in 100000 Global Non-Detection Risk: 1230 in 100000 1. Brantall 2. Kilim 3. Necurs 4. Gamarue 5. Kishop Commissioned by Microsoft - 16 -

BullGuard Internet Security 14.0.278.3 1. Ireland 5006 in 100000 2. Estonia 4778 in 100000 3. Uruguay 4742 in 100000 4. France 4461 in 100000 5. Iceland 4412 in 100000 6. Belgium 4063 in 100000 7. Brail 3812 in 100000 8. Norway 3729 in 100000 9. Germany 3665 in 100000 10. New Zealand 3620 in 100000 11. Denmark 3440 in 100000 12. Bulgaria 3377 in 100000 13. Finland 3135 in 100000 14. Serbia 3033 in 100000 15. Cyprus 2990 in 100000 16. Greece 2927 in 100000 17. Hungary 2894 in 100000 18. Jamaica 2850 in 100000 19. Italy 2848 in 100000 20. Tunisia 2822 in 100000 Global Non-Detection Risk: 1303 in 100000 1. Brantall 2. Kilim 3. Necurs 4. Gamarue 5. Kishop Commissioned by Microsoft - 17 -

Emsisoft Anti-Malware 8.1.0.40 1. Ireland 5005 in 100000 2. Estonia 4778 in 100000 3. Uruguay 4742 in 100000 4. France 4460 in 100000 5. Iceland 4412 in 100000 6. Belgium 4062 in 100000 7. Brazil 3802 in 100000 8. Norway 3728 in 100000 9. Germany 3665 in 100000 10. New Zealand 3619 in 100000 11. Denmark 3439 in 100000 12. Bulgaria 3377 in 100000 13. Finland 3134 in 100000 14. Serbia 3033 in 100000 15. Cyprus 2989 in 100000 16. Greece 2926 in 100000 17. Hungary 2893 in 100000 18. Jamaica 2850 in 100000 19. Italy 2847 in 100000 20. Tunisia 2822 in 100000 Global Non-Detection Risk: 1302 in 100000 1. Brantall 2. Kilim 3. Necurs 4. Gamarue 5. Kishop Commissioned by Microsoft - 18 -

escan Internet Security 14.0.1400.1572 1. Ireland 4818 in 100000 2. Uruguay 4620 in 100000 3. France 4319 in 100000 4. Estonia 4268 in 100000 5. Iceland 4206 in 100000 6. Belgium 3839 in 100000 7. Brazil 3701 in 100000 8. New Zealand 3514 in 100000 9. Norway 3512 in 100000 10. Germany 3493 in 100000 11. Denmark 3304 in 100000 12. Finland 2923 in 100000 13. Bulgaria 2760 in 100000 14. Jamaica 2744 in 100000 15. Italy 2718 in 100000 16. Serbia 2698 in 100000 17. Japan 2644 in 100000 18. Switzerland 2628 in 100000 19. Hong Kong 2626 in 100000 20. Cyprus 2625 in 100000 Global Non-Detection Risk: 1186 in 100000 1. Brantall 2. Necurs 3. Kishop 4. Gamarue 5. Dorkbot Commissioned by Microsoft - 19 -

ESET Smart Security 7.0.302.26 1. Hungary 5628 in 100000 2. Slovenia 3615 in 100000 3. Bulgaria 3273 in 100000 4. Estonia 2851 in 100000 5. Ghana 2663 in 100000 6. Turkey 2559 in 100000 7. Tunisia 2293 in 100000 8. Slovak Republic 2244 in 100000 9. South Korea 2120 in 100000 10. Greece 2096 in 100000 11. China 2004 in 100000 12. Lithuania 1899 in 100000 13. Netherlands 1853 in 100000 14. Cyprus 1703 in 100000 15. Portugal 1681 in 100000 16. Serbia 1640 in 100000 17. Romania 1568 in 100000 18. Belgium 1544 in 100000 19. Poland 1539 in 100000 20. Finland 1480 in 100000 Global Non-Detection Risk: 641 in 100000 1. Kilim 2. Malagent 3. Sisron 4. Dynamer 5. Bladabindi Commissioned by Microsoft - 20 -

F-Secure Internet Security 14.99.103 1. Ireland 3524 in 100000 2. Uruguay 3388 in 100000 3. France 3168 in 100000 4. Estonia 3158 in 100000 5. Iceland 3099 in 100000 6. Belgium 2799 in 100000 7. Brazil 2726 in 100000 8. Norway 2560 in 100000 9. New Zealand 2555 in 100000 10. Germany 2528 in 100000 11. Denmark 2412 in 100000 12. Finland 2142 in 100000 13. Bulgaria 2025 in 100000 14. Italy 1976 in 100000 15. Japan 1973 in 100000 16. Chile 1942 in 100000 17. Hong Kong 1936 in 100000 18. Jamaica 1928 in 100000 19. Serbia 1921 in 100000 20. Switzerland 1883 in 100000 Global Non-Detection Risk: 821 in 100000 1. Brantall 2. Necurs 3. Kishop 4. Dorkbot 5. Bladabindi Commissioned by Microsoft - 21 -

Fortinet FortiClient 5.0.8.344 1. France 3025 in 100000 2. Ireland 2783 in 100000 3. Belgium 2581 in 100000 4. Finland 2408 in 100000 5. Uruguay 2370 in 100000 6. Denmark 2364 in 100000 7. Germany 2350 in 100000 8. Norway 2296 in 100000 9. Estonia 2286 in 100000 10. Iceland 2273 in 100000 11. Brazil 2095 in 100000 12. New Zealand 2051 in 100000 13. Spain 1921 in 100000 14. Sweden 1893 in 100000 15. Austria 1858 in 100000 16. Italy 1806 in 100000 17. Switzerland 1772 in 100000 18. Netherlands 1754 in 100000 19. Japan 1738 in 100000 20. Greece 1679 in 100000 Global Non-Detection Risk: 572 in 100000 1. Brantall 2. Sefnit 3. Notorgatro 4. Necurs 5. Bancos Commissioned by Microsoft - 22 -

Kaspersky Internet Security 14.0.0.4651 (e) 1. China 681 in 100000 2. South Korea 602 in 100000 3. United States 504 in 100000 4. Canada 407 in 100000 5. Taiwan 334 in 100000 6. Estonia 329 in 100000 7. Czech Republic 314 in 100000 8. Hong Kong 297 in 100000 9. Brazil 294 in 100000 10. Latvia 288 in 100000 11. Slovak Republic 284 in 100000 12. Finland 271 in 100000 13. Sweden 255 in 100000 14. Netherlands 254 in 100000 15. Iceland 244 in 100000 16. Slovenia 243 in 100000 17. Spain 241 in 100000 18. Hungary 237 in 100000 19. Portugal 233 in 100000 20. Lithuania 226 in 100000 Global Non-Detection Risk: 121 in 100000 1. Tesch 2. Dynamer 3. Necurs 4. Sodebral 5. Ramnit Commissioned by Microsoft - 23 -

Kingsoft Internet Security 2013.SP6.0.030511 1. Ireland 5005 in 100000 2. Estonia 4777 in 100000 3. Uruguay 4737 in 100000 4. France 4457 in 100000 5. Iceland 4411 in 100000 6. Belgium 4059 in 100000 7. Brazil 3804 in 100000 8. Norway 3727 in 100000 9. Germany 3660 in 100000 10. New Zealand 3618 in 100000 11. Denmark 3438 in 100000 12. Bulgaria 3375 in 100000 13. Finland 3134 in 100000 14. Serbia 3032 in 100000 15. Cyprus 2986 in 100000 16. Greece 2925 in 100000 17. Hungary 2892 in 100000 18. Jamaica 2849 in 100000 19. Italy 2843 in 100000 20. Tunisia 2789 in 100000 Global Non-Detection Risk: 1294 in 100000 1. Brantall 2. Kilim 3. Necurs 4. Gamarue 5. Kishop Commissioned by Microsoft - 24 -

Lavasoft Ad-Aware Free Antivirus+ 11.1.5354.0 1. Ireland 5005 in 100000 2. Estonia 4776 in 100000 3. Uruguay 4737 in 100000 4. France 4457 in 100000 5. Iceland 4411 in 100000 6. Belgium 4059 in 100000 7. Brazil 3804 in 100000 8. Norway 3727 in 100000 9. Germany 3660 in 100000 10. New Zealand 3618 in 100000 11. Denmark 3437 in 100000 12. Bulgaria 3375 in 100000 13. Finland 3133 in 100000 14. Serbia 3031 in 100000 15. Cyprus 2986 in 100000 16. Greece 2925 in 100000 17. Hungary 2892 in 100000 18. Jamaica 2849 in 100000 19. Italy 2843 in 100000 20. Tunisia 2789 in 100000 Global Non-Detection Risk: 1293 in 100000 1. Brantall 2. Kilim 3. Necurs 4. Gamarue 5. Kishop Commissioned by Microsoft - 25 -

McAfee Internet Security 16.8.708 1. China 928 in 100000 2. Brazil 823 in 100000 3. South Korea 552 in 100000 4. Taiwan 452 in 100000 5. Portugal 373 in 100000 6. Czech Republic 285 in 100000 7. Italy 277 in 100000 8. Netherlands 275 in 100000 9. Estonia 273 in 100000 10. Slovak Republic 269 in 100000 11. Hong Kong 254 in 100000 12. Belgium 249 in 100000 13. Finland 245 in 100000 14. Denmark 236 in 100000 15. Latvia 230 in 100000 16. Sweden 229 in 100000 17. Norway 228 in 100000 18. Egypt 226 in 100000 19. France 223 in 100000 20. Germany 216 in 100000 Global Non-Detection Risk: 122 in 100000 1. Sulunch 2. Banker 3. Dynamer 4. Sisron 5. Bancos Commissioned by Microsoft - 26 -

Microsoft Security Essentials 4.4.304.0 1. Czech Republic 1026 in 100000 2. Slovak Republic 865 in 100000 3. South Korea 857 in 100000 4. China 776 in 100000 5. Netherlands 723 in 100000 6. Sweden 674 in 100000 7. Finland 654 in 100000 8. Denmark 645 in 100000 9. Croatia 629 in 100000 10. Portugal 624 in 100000 11. Brazil 624 in 100000 12. Estonia 620 in 100000 13. Iraq 620 in 100000 14. Norway 617 in 100000 15. Belgium 617 in 100000 16. Saudi Arabia 603 in 100000 17. Latvia 600 in 100000 18. Iceland 597 in 100000 19. Austria 595 in 100000 20. Taiwan 590 in 100000 Global Non-Detection Risk: 337 in 100000 1. CeeInject 2. Stekct 3. Bladabindi 4. Dynamer 5. Ainslot Commissioned by Microsoft - 27 -

Panda Cloud Free Antivirus 2.3.0 1. Ireland 1037 in 100000 2. Uruguay 970 in 100000 3. Estonia 951 in 100000 4. France 945 in 100000 5. Iceland 907 in 100000 6. Brazil 879 in 100000 7. Belgium 851 in 100000 8. Norway 788 in 100000 9. New Zealand 759 in 100000 10. Germany 754 in 100000 11. Denmark 745 in 100000 12. Finland 687 in 100000 13. South Korea 659 in 100000 14. Italy 598 in 100000 15. Hong Kong 594 in 100000 16. Bulgaria 585 in 100000 17. Japan 574 in 100000 18. Serbia 559 in 100000 19. Switzerland 559 in 100000 20. Greece 557 in 100000 Global Non-Detection Risk: 225 in 100000 1. Brantall 2. SystemHijack 3. Dynamer 4. Necurs 5. Dofoil Commissioned by Microsoft - 28 -

Qihoo 360 Internet Security 4.9.0.4109 (en) 1. Ireland 5006 in 100000 2. Estonia 4778 in 100000 3. Uruguay 4742 in 100000 4. France 4461 in 100000 5. Iceland 4412 in 100000 6. Belgium 4063 in 100000 7. Brazil 3808 in 100000 8. Norway 3729 in 100000 9. Germany 3665 in 100000 10. New Zealand 3620 in 100000 11. Denmark 3440 in 100000 12. Bulgaria 3377 in 100000 13. Finland 3135 in 100000 14. Serbia 3033 in 100000 15. Cyprus 2990 in 100000 16. Greece 2927 in 100000 17. Hungary 2894 in 100000 18. Jamaica 2850 in 100000 19. Italy 2848 in 100000 20. Tunisia 2822 in 100000 Global Non-Detection Risk: 1303 in 100000 1. Brantall 2. Kilim 3. Necurs 4. Gamarue 5. Kishop Commissioned by Microsoft - 29 -

Sophos Endpoint Security and Control 10.3.1 1. China 1897 in 100000 2. South Korea 1507 in 100000 3. Hungary 1495 in 100000 4. Estonia 1396 in 100000 5. Slovenia 1260 in 100000 6. Czech Republic 1210 in 100000 7. Greece 1205 in 100000 8. Slovak Republic 1200 in 100000 9. Bulgaria 1065 in 100000 10. Hong Kong 1054 in 100000 11. Lithuania 1049 in 100000 12. Finland 1016 in 100000 13. Latvia 999 in 100000 14. Taiwan 999 in 100000 15. Netherlands 954 in 100000 16. Brazil 954 in 100000 17. Iceland 938 in 100000 18. Sweden 910 in 100000 19. Turkey 884 in 100000 20. Portugal 884 in 100000 Global Non-Detection Risk: 594 in 100000 1. Kilim 2. Dynamer 3. Vundo 4. Necurs 5. Tesch Commissioned by Microsoft - 30 -

Tencent QQ PC Manager 8.9.25000. 501 1. Finland 809 in 100000 2. Sweden 704 in 100000 3. Netherlands 701 in 100000 4. Czech Republic 693 in 100000 5. France 690 in 100000 6. China 675 in 100000 7. Belgium 651 in 100000 8. Denmark 649 in 100000 9. Spain 632 in 100000 10. Norway 600 in 100000 11. Slovak Republic 590 in 100000 12. Germany 574 in 100000 13. Austria 550 in 100000 14. Estonia 522 in 100000 15. Italy 514 in 100000 16. Taiwan 510 in 100000 17. South Korea 502 in 100000 18. Portugal 499 in 100000 19. Hungary 465 in 100000 20. Ireland 464 in 100000 Global Non-Detection Risk: 200 in 100000 1. Sefnit 2. Dynamer 3. VBInject 4. Bladabindi 5. Kishop Commissioned by Microsoft - 31 -

ThreatTrack Vipre Internet Security 7.0.6.2 1. Japan 981 in 100000 2. Ghana 771 in 100000 3. Canada 761 in 100000 4. Hong Kong 741 in 100000 5. United Kingdom 738 in 100000 6. Austria 733 in 100000 7. Croatia 720 in 100000 8. Australia 707 in 100000 9. Italy 698 in 100000 10. Switzerland 694 in 100000 11. Pakistan 679 in 100000 12. Romania 658 in 100000 13. Germany 655 in 100000 14. Albania 641 in 100000 15. Czech Republic 640 in 100000 16. Egypt 633 in 100000 17. Netherlands 627 in 100000 18. Iceland 623 in 100000 19. Finland 621 in 100000 20. Latvia 619 in 100000 Global Non-Detection Risk: 528 in 100000 1. Zbot 2. Gamarue 3. VBInject 4. Necurs 5. Malagent Commissioned by Microsoft - 32 -

Trend Micro Titanium Internet Security 7.0.1206 1. Ireland 5719 in 100000 2. Uruguay 5441 in 100000 3. France 5054 in 100000 4. Estonia 4843 in 100000 5. Iceland 4731 in 100000 6. Belgium 4511 in 100000 7. Brazil 4343 in 100000 8. New Zealand 4202 in 100000 9. Norway 4161 in 100000 10. Germany 4101 in 100000 11. Denmark 3941 in 100000 12. Finland 3428 in 100000 13. Italy 3144 in 100000 14. Bulgaria 3084 in 100000 15. Chile 3080 in 100000 16. Japan 3025 in 100000 17. Jamaica 3024 in 100000 18. Switzerland 2998 in 100000 19. Serbia 2975 in 100000 20. Canada 2941 in 100000 Global Non-Detection Risk: 1181 in 100000 1. Brantall 2. Dorkbot 3. Notorgatro 4. Zbot 5. Sirefef Commissioned by Microsoft - 33 -

Copyright and Disclaimer This publication is Copyright 2014 by AV-Comparatives. Any use of the results, etc. in whole or in part, is ONLY permitted after the explicit written agreement of the management board of AV- Comparatives, prior to any publication. AV-Comparatives and its testers cannot be held liable for any damage or loss, which might occur as result of, or in connection with, the use of the information provided in this paper. We take every possible care to ensure the correctness of the basic data, but a liability for the correctness of the test results cannot be taken by any representative of AV- Comparatives. We do not give any guarantee of the correctness, completeness, or suitability for a specific purpose of any of the information/content provided at any given time. No one else involved in creating, producing or delivering test results shall be liable for any indirect, special or consequential damage, or loss of profits, arising out of, or related to, the use or inability to use, the services provided by the website, test documents or any related data. For more information about AV-Comparatives and the testing methodologies, please visit our website. AV-Comparatives (August 2014) Commissioned by Microsoft - 34 -