Endurance Test: Does antivirus software slow

Transcription

1 23rd April 2015 created by Markus Selinger Endurance Test: Does antivirus software slow down PCs? Critics maintain that protection software for Windows really puts the brakes on PCs. In a 14-month, extremely comprehensive performance endurance test, AV-TEST examined the trade-off of performance versus protection, and came up with some conclusive answers. For the reader of an endurance test, the provided rating for speed or performance of a product is only expressed in one single aggregate number. For the tester in the laboratory, it's a long journey to arrive at this number. To get there, he had to accompany a solution through 7 test rounds over 14 months. Of the 19,000 individual ratings per product, 35 test area findings were aggregated. It requires this time and effort to arrive at the rating for a single product. In the current endurance test from January 2014 to the end of February 2015, a total of 23 products were tested for speed in the labs at AV-TEST. In this, the tests alternated in the use of the operating systems Windows XP, 7 and products in the Performance Test Included in the test were products from Ahnlab, Avast, AVG (freeware and purchase product), Avira, Bitdefender, BullGuard, Comodo, ESET, F-Secure, G Data, Kaspersky, McAfee, Microworld, Norman, Norton, Panda, Qihoo 360, Quick Heal, Tencent, Threat Track and Trend Micro. In addition, a Windows system with the freeware Microsoft Security Essentials or Defender (for Windows 8/8.1) was put through all the tests as an additional comparative figure. The reference system without any protection software was used for later comparison of the measured performance ratings. A more in-depth explanation on the topic of reference systems can be found later in the "Test Configuration" section. For all products in the tests, it was measured how long they required for various test sets, in order to - download files from the Internet; - launch websites; - install applications; - open applications, including a file; - copy files. For all subsections, load points of 1 to 5 were assigned after each test, whereby a score of 1 represents good and a score of 5 poor. If an antivirus application slowed down a system between 0 and 20 percent, it received 1 load point. For 21 to 40 percent there were 2 points, for 41 to 60 percent drop in performance, 3 points, and so on up to 5 points. Thus for all 5 test sections, this resulted in a perfect score of 5 and a worst score of 25 load points. All partial results of the tests were aggregated and at the end divided by the number of 7 test rounds. The Test Result The lowest influence and thus the best speed-performance rating was achieved by the product from Kaspersky with only 5.1 load points. As previously mentioned, the maximum perfect score was 5.0. Following close behind are the security packages from Bitdefender and Qihoo 360 with 5.3 and 5.7 points respectively. The additional 6 products from McAfee, Bullguard, Trend Micro, Norton, Avira and F-Secure also achieve good rankings at 6.1 to 7.9 load points in their endurance test results. All other 14 products are above that threshold. If the system is protected by the Windows built-in Security Essentials or Microsoft Defender, the load is 8.7 points. The solutions from Norman, Quick Heal and Threat Track already slow down a PC significantly with 12, 12.4 and 13.9 load points. No product attained the worst score of 25 slow-down points.

2 In order to more clearly understand the figures, here is an example with two individual scores: Every user quickly becomes impatient, especially when copying files under Windows. In the test, the reference system without any protection requires just over 141 seconds to copy the test set from AV-TEST containing 3.3 GB. A Windows system with an installed Kaspersky security application examines the files and requires 165 seconds for copying. The application finishing last, Threat Track, requires over 300 seconds on average for the same procedure. The individual test sections AV-TEST refers to the 5 test sections as the "Real-World Test", as all the tested operations are typically performed daily multiple times by users. 1. Downloading files from the Internet In order to ensure that all the products have exactly the same line for downloading, a server is used as a download server in a separate test network. All the programs offered on the server are available unmodified in the same version during the entire test period. This guarantees the identical download scenario during the endurance test. Most security packages scan the data or the data stream during the download and may thus slow down the procedure. Nearly all the products handle this task so well, however, that the scores fall below the 20 percent mark, thus 21 out of 23 packages consistently receive only an average of one load point. Only the products from Avira and G Data are slightly above this level on average and for that reason are penalized with somewhat more load points. 2. Launching websites For this test, a few dozen premium websites are launched, such as Amazon, Yahoo, Apple or Google. The Internet Explorer of the test operating system Windows XP, Windows 7 or Windows 8.1 is used for this. The selected websites are always highly available in the web and therefore perfect for the comparison. The test sounds simple, but it is tremendously revealing. Because in final analysis, only three products achieve a low load level under 20 percent and therefore only sustain 1 point: Bitdefender, Kaspersky and McAfee. An additional 20 products break through the barrier of 20 percent more load and require up to 40 percent. For this reason, those products also receive 2 load points. The only anomaly here is the product Threat Track, which slows down the website launch tremendously and therefore scores 4.6 load points on average. 3. Installing applications In the test, applications are installed per command line (without clicks), and the time is clocked for this operation. Included in this test section are popular programs such as the Flash Player or the Adobe Reader. The applications and their versions are naturally identical throughout the entire endurance test. This test is also part of a user's typical daily procedure. The findings show that many security programs delay the installation procedure to a certain extent. Bitdefender, Bullguard, G Data and Trend Micro remain within normal bounds, thus only scoring 1 load point. An additional 9 solutions are between 1 and 2 points on average, 9 products score between 2 and 3 points, and Microsoft Security Essentials or Microsoft Defender even above 3 load points. 4. Opening applications, including a file In this test, a DOC file, a pdf file and a presentation file all having a large size are opened repeatedly and directly with LibreOffice. The security packages monitor the access and the opening of files. Some solutions thus delay the procedure. In this area of the endurance test, the products from Bitdefender, ESET and Kaspersky were hardly noticeable and thus received only 1 load point. An additional 16 products were above 1 point on average, 4 solutions scored even 2 or more load points. 5. Copying files Especially when copying data under Windows, most security solutions cause frustration among users. That is why the lab team examined how heavily the products delayed the copying of files. The test featured a 3.3 GB set with a wide variety of file types such as films, images, graphics, documents, pdfs and programs. Up to this test phase, most of the products were still virtually neck and neck. Due to the findings in this test category, however, many security packages fell behind the top finishers. The test demonstrates that Qihoo 360 and Kaspersky delay the copying procedure only ever so slightly. 8 products sustained between 1 and 2 points for their load. 5 security packages were between 2 and 3 points on average, 3 products between 3 and 4 points, and 5 products even attained an average above 4 points.

3 The most conspicuous products in the endurance test are from Ahnlab, ESET, Norman, Quickheal and Threat Track. They considerably slow down the copying procedure under routine conditions. This fact is also reflected in the overall findings. Test Configuration For the test, several PCs with absolute identical hardware were used in the laboratory: Intel Xeon 2.83GHz, 4 GB RAM and a 500 GB hard drive. Because these computers might perform differently, however, they were selected from a pool of 60 test PCs. All the PCs were tested multiple times with unprotected windows XP, windows 7 and Windows 8.1 systems. The computers of identical speed were then defined as test systems. Their speed ratings serve as a reference for the test. Before and after each product test, it was evaluated whether all the computers still delivered the same reference ratings. Each product goes through a cycle of all the tests multiple times. Afterwards, a status quo ante of the PC is re-established with the help of a disk image, including the installed application. Then the test is performed again, the system is rebooted, and the battery of tests to be conducted is repeated again. The PC is rebooted and tested several times, then the system is restored with the help of the disk image, and the procedure starts all over again. At the end, there are a minimum of 20 and a maximum of 40 individual scores for each individual small test step. For some 90 test steps per product, from 1,800 up to 3,600 individual scores are recorded per product. In an endurance test lasting 7 test rounds, an actual 12,600 up to 25,200 scores are analyzed; on average around 19,000 per product. In the test with 23 products, some 430,000 test scores were recorded. 430,000 individual scores all add up The analysis of 20 to 40 individual scores from each test step is performed according to a special methodology. The individual scores are calculated in groups according to the mean value and standard deviation. The group with the lowest standard deviation therefore has the lowest possible error ratio. This is then used for additional calculation. The procedure of analysis used is standard statistics and eliminates the skewing of results due to outliers. The aggregate calculation is subsequently compared to the performance rating of the unprotected reference system. Does antivirus software slow down PCs? The result indicates that good security software does not heavily slow down a Windows PC. A few systems resources are required, which slows down the PC, just like with any other software application launched. The products from Kaspersky, Bitdefender and Qihoo 360 exhibited the lowest load on the systems, although they provide good protection, as proven by the previous security function tests ("The best antivirus software for Windows Home User"). The "subjective" slowing down of a PC frequently reported by users is actually a proven objective fact. This is clearly demonstrated by the products from Norman, Quickheal and Threat Track. These security packages cause the copying routine to run 2.5 to 3 times more slowly than with Kaspersky software, for instance. The AV-TEST team occasionally receives inquiries from users arriving at totally different results in their own tests. This may indeed occur for special configurations and particular hardware and software combinations. Those who are uncertain whether the fastest software is right for them ought to try the following test: First, create a system backup, then load the test software and try everything out yourself. If something doesn't quite perform the way it should, then you simply revert to the backup. Test Methodology and Critics

4 Typical objections by critics against the way results were obtained are the influences by the hardware used, by the software such as Windows or by access to the Internet. The point of criticism concerning hardware is rendered moot based on the abovedescribed procedure of the test PC selection under "Test Configuration". The reference scores are re-validated after each test on each PC. With respect to Windows, background tasks could influence system behavior. That is why the testers deactivated all the tasks that can be shut down. In the event that an outlier does occur in a test step, this is statistically balanced out based on 20- to 40-fold repetition of the step. The third point of criticism pertains to the Internet access in the test when visiting websites. On the one hand, the speed may depend on the requested server and the routing, and on the other, ad banners or images may modify the websites. The test is completed with the help of a 500 megabit line, and it runs over several days and at different times. This minimizes any possible negative effects. The findings also demonstrate that the slowing-down effect caused by security software does not occur sporadically, but is rather constant for all websites tested. Maik Morgenstern, CTO AV-TEST GmbH Antivirus products in the "Performance" endurance test.

5 Performance endurance test: Some protection solutions cause quite a heavy burden on a Windows PC for everyday routines.

6 Category "Copying under Windows": While the Kaspersky product hardly slows down the system, Threat Track and Quick Heal are a different story entirely. Performance endurance test: The products from Kaspersky, Bitdefender and Qihoo 360 cause the lowest load on a Windows PC.

7 5 categories in the performance test: Most of the products in the endurance test slow down the PC in the category of "copying data". Copyright 2015 by AV-TEST GmbH, Klewitzstr. 7, Magdeburg, Germany Phone +49 (0) , Fax +49 (0) ,