Real Performance? Ján Vrabec David Harley
|
|
|
- Dylan Moore
- 10 years ago
- Views:
Transcription
1 Real Performance? Ján Vrabec David Harley
2 Agenda Introduction Detection vs. Whole Product Test Performance Tests Black box testing suites Irrelevant Testing Types of users
3 Introduction Detection vs. performance testing Evaluation Testing Buying decisions Missing guidelines Own testing procedures AMTSO
4 Detection performance isn t enough in itself Usability, ergonomics and configurability Functional adaptation Responsiveness to the needs of and changes in the organizational environment or infrastructure Responsiveness or adaptability to business needs
5 Detection Testing Versus Whole Product Testing More functions Impact on performance Interactions Out of the box settings Different view
6 Detection Testing Versus Whole Product Testing
7 Balanced Product Protection False Positive Performance Usability
8 Scanning Throughput Static scanning Samples - Clean Set Representative set Skip over files Multiple measurements First and following scans Hashing Faster Secure?
9 Memory Usage System is in idle state Windows Task Manager Hidden processes, 10 MB antivirus Source: Performance Benchmarking, PassMark January 2010
10 Memory Usage System in idle Several readings Reserved memory CC of system with installed solution -CC of clean system Memory Consumption Source: Performance Benchmarking, PassMark January 2010
11 System Boot Speed Active on a system at an early stage When to stop the measurement? Antivirus presence detector Idle state of system
12 Black Box Testing Suites Instant Testing Tools World Bench or Passmark Focused on hardware White listed Error bigger then difference Interpretation of results
13 Irrelevant Testing False tests Registry Key Count Process Count No effect on computer performance Improved product rating? Source: Performance Benchmarking, January 2010
14 Magazines Results German computer magazine reviews Varied results Confused reader
15 Malware Performance Testing by User Type Consumer Surfer Gamer Worker Corporate Users Administrators
16 Consumer All Boot time Memory consumption Installing common software applications Copying files to the system or to and from a local network resource Surfer Browsing of web pages from proxy server Browser start-up time Viewing video files streamed from a Web server
17 Consumer Gamer Latency on the network Degradation of frame per seconds Worker Downloading s from server clients start-up Time of opening, closing, saving and copying documents Editing video and audio files Converting from one format to another Start-up times of specific applications
18 Corporate Users Simulation of work with common business software Time taken to open, process and close single or multiple documents and applications Network performance Accessing or messaging services Web browsing Designing internal applications, procedures and implementations in-house. Administrators Performance on File and mail servers, gateways
19 Conclusion Testing with pitfalls Valid and objective techniques Guidelines More focus on Whole Product Testing
20
21 The Nine Principles 1/2 1. Testing must not endanger the public. 2. Testing must be unbiased. 3. Testing should be reasonably open and transparent. 4. The effectiveness and performance of anti malware products must be measured in a balanced way. 5. Testers must take reasonable care to validate whether test samples or test cases have been accurately classified as malicious, innocent or invalid.
22 The Nine Principles 2/2 6. Testing methodology must be consistent with the testing purpose. 7. The conclusions of a test must be based on the test results. 8. Test results should be statistically valid. 9. Vendors, testers and publishers must have an active contact point for testing related correspondence.
23 AMTSO Compliance Technically, there s no such thing (yet) The Review Analysis Board can assess whether a test report is compliant with the nine principles: Two analyses (near-)completed to date: NSS Labs Dennis Publishing
24 AMTSO Documentation Documents and Principles These documents are either fully published or work in progress specifying AMTSO Principles and Guidelines related to testing. Virus Bulletin Spotlight article on AMTSO What AMTSO has achieved so far, and what might lie ahead. David Harley, AMTSO lutely Fabulous, January 2010, Virus Bulletin. Copyright is held by Virus Bulletin Ltd, but is made available on this site for personal use free of charge by permission of Virus Bulletin. AMTSO Fundamental Principles of Testing AMTSO Fundamental Principles of Testing as approved by the AMTSO meeting held in Oxford 31st October AMTSO Best Practices for Dynamic Testing AMTSO Best Practices for Dynamic Testing as approved by the AMTSO meeting held in Oxford 31st October 2008 AMTSO Best Practices for Validation of Samples AMTSO Best Practices for validation of samples as approved by the AMTSO meeting held in Budapest 7th May 2009
25 AMTSO Documentation AMTSO Best Practices for Testing In-the-Cloud Security Products AMTSO Best Practices for Testing In-the-Cloud Security Products as approved by the AMTSO meeting held in Budapest 7th May 2009 AMTSO Analysis of Reviews Process AMTSO Analysis of Reviews Process as approved by the AMTSO meeting held in Budapest 7th May 2009 AMTSO Guidelines for testing Network Based Security Products AMTSO Guidelines for testing Network Based Security Products as approved by the AMTSO meeting held in Prague 13th October 2009 AMTSO Issues involved in the "creation" of samples for testing AMTSO Issues involved in the "creation" of samples for testing as approved by the AMTSO meeting held in Prague 13th October 2009
26 Thank you for your attention Questions? Ján Vrabec David Harley
27 References AMTSO (2010a). AMTSO Whole Product Testing Guidelines (in preparation) AMTSO (2010b). AMTSO Performance Testing Guidelines (in preparation) ESET Research (2010). Retrieved 10th March 2010 from AV Comparatives (2009) Retrieved 10th March 2010 from Harley, D. (2009a). Making Sense of Anti-Malware Comparative Testing. Information Security Technical Report. Retrieved 10th March, 2010 from Elsevier. Harley, D. (2009b). Execution Context in Anti-Malware Testing. Conference Proceedings for 18th EICAR Annual Conference. Retrieved 10th March 2010 from Lee, A.J. & Harley, D. (2007). Antimalware Evaluation and Testing. In D. Harley (Ed.) AVIEN Malware Defense Guide for the Enterprise (pp ): Syngress Vrabec, J. (2010). Generalist Anti-Malware Testing (In preparation)
VESZPROG ANTI-MALWARE TEST BATTERY
VESZPROG ANTI-MALWARE TEST BATTERY 2012 The number of threats increased in large measure in the last few years. A set of unique anti-malware testing procedures have been developed under the aegis of CheckVir
Virtual Desktops Security Test Report
Virtual Desktops Security Test Report A test commissioned by Kaspersky Lab and performed by AV-TEST GmbH Date of the report: May 19 th, 214 Executive Summary AV-TEST performed a comparative review (January
Sophos Computer Security Scan startup guide
Sophos Computer Security Scan startup guide Product version: 1.0 Document date: February 2010 Contents 1 About the software...3 2 What do I need to do?...3 3 Prepare for scanning...3 4 Install the software...4
Deep Security Vulnerability Protection Summary
Deep Security Vulnerability Protection Summary Trend Micro, Incorporated This documents outlines the process behind rules creation and answers common questions about vulnerability coverage for Deep Security
What to Look for When Evaluating Next-Generation Firewalls
What to Look for When Evaluating Next-Generation Firewalls Using independent tests to compare performance, cost and functionality Table of Contents Why Use Independent Tests in Evaluations?... 3 What to
ESET Security Solutions for Your Business
ESET Security Solutions for Your Business It Is Our Business Protecting Yours For over 20 years, companies large and small have relied on ESET to safeguard their mission-critical infrastructure and keep
Executive Brief for Sharing Sites & Digital Content Providers. Leveraging Hybrid P2P Technology to Enhance the Customer Experience and Grow Profits
Executive Brief for Sharing Sites & Digital Content Providers Leveraging Hybrid P2P Technology to Enhance the Customer Experience and Grow Profits Executive Summary The Opportunity/Challenge The revenue
Security Industry Market Share Analysis
Security Industry Market Share Analysis December Introduction The Research OPSWAT releases quarterly market share reports for several sectors of the security industry. This report includes both worldwide
DETERMINATION OF THE PERFORMANCE
DETERMINATION OF THE PERFORMANCE OF ANDROID ANTI-MALWARE SCANNERS AV-TEST GmbH Klewitzstr. 7 39112 Magdeburg Germany www.av-test.org 1 CONTENT Determination of the Performance of Android Anti-Malware Scanners...
McAfee MOVE / VMware Collaboration Best Practices
McAfee MOVE / VMware Collaboration Best Practices Christie J. Karrels Sales Engineer Federal DoD January 11, 2013 1 P a g e Contents Introduction... 3 Traditional Anti-Malware vs. Optimized Anti-Malware...
Content Security Gateway Series Real-time Gateway Web Security Against Spyware and Viruses
Content Security Gateway Series Real-time Gateway Web Security Against Spyware and Viruses 1. Why do I need a Web security or gateway anti-spyware solution? Malware attack vector is rapidly shifting from
Anti-Virus Protection and Performance
Anti-Virus Protection and Performance ANNUAL REPORT 2015 Dennis Technology Labs www.dennistechnologylabs.com Follow @DennisTechLabs on Twitter.com CONTENTS Annual Report 2015... 1 Contents... 2 Introduction...
Sophos Anti-Virus for NetApp Storage Systems startup guide. Runs on Windows 2000 and later
Sophos Anti-Virus for NetApp Storage Systems startup guide Runs on Windows 2000 and later Document date: July 2007 Contents About this guide...4 About Sophos Anti-Virus for NetApp Storage Systems...5
SaaS Endpoint Security Solutions Performance Test
Panda Managed Office Protection SaaS Endpoint Security Solutions Performance Test April 2009 Contents 1. INTRODUCTION... 3 2. PRODUCTS ANALYZED... 3 3. METRICS... 5 Benchmark 1 CPU usage... 5 Benchmark
F-Secure Internet Gatekeeper Virtual Appliance
F-Secure Internet Gatekeeper Virtual Appliance F-Secure Internet Gatekeeper Virtual Appliance TOC 2 Contents Chapter 1: Welcome to F-Secure Internet Gatekeeper Virtual Appliance.3 Chapter 2: Deployment...4
Are free Android virus scanners any good?
Authors: Hendrik Pilz, Steffen Schindler Published: 10. November 2011 Version: 1.1 Copyright 2011 AV-TEST GmbH. All rights reserved. Postal address: Klewitzstr. 7, 39112 Magdeburg, Germany Phone +49 (0)
How To Test For Performance On A 64 Bit Computer (64 Bit)
Anti-Virus Comparative Performance Test Impact of Security Software on System Performance Language: English May 2015 Last Revision: 30 th June 2015 Table of Contents 1. Introduction 3 2. Tested products
Anti-Virus Comparative - Performance Test (AV Products) May 2014
Anti-Virus Comparative Performance Test (AV Products) Impact of Anti-Virus Software on System Performance Language: English May 2014 Last Revision: 10 th June 2014 Table of Contents 1. Introduction 3 2.
Getting Ahead of Malware
IT@Intel White Paper Intel Information Technology Security December 2009 Getting Ahead of Malware Executive Overview Since implementing our security event monitor and detection processes two years ago,
Application Visibility and Monitoring >
White Paper Application Visibility and Monitoring > An integrated approach to application delivery Application performance drives business performance Every business today depends on secure, reliable information
Simphony v2 Antivirus Recommendations
DECLARATIONS WARRANTIES Although the best efforts are made to ensure that the information in this document is complete and correct, MICROS Systems, Inc. makes no warranty of any kind with regard to this
Endpoint Business Products Testing Report. Performed by AV-Test GmbH
Business Products Testing Report Performed by AV-Test GmbH January 2011 1 Business Products Testing Report - Performed by AV-Test GmbH Executive Summary Overview During November 2010, AV-Test performed
Trend Micro OfficeScan 11.0. Best Practice Guide for Malware
Trend Micro OfficeScan 11.0 Best Practice Guide for Malware Information in this document is subject to change without notice. The names of companies, products, people, characters, and/or data mentioned
How McAfee Endpoint Security Intelligently Collaborates to Protect and Perform
How McAfee Endpoint Security Intelligently Collaborates to Protect and Perform McAfee Endpoint Security 10 provides customers with an intelligent, collaborative framework, enabling endpoint defenses to
The Dirty Secret Behind the UTM: What Security Vendors Don t Want You to Know
The Dirty Secret Behind the UTM: What Security Vendors Don t Want You to Know I n t r o d u c t i o n Until the late 1990s, network security threats were predominantly written by programmers seeking notoriety,
Tracking Anti-Malware Protection 2015
Tracking Anti-Malware Protection 2015 A TIME-TO-PROTECT ANTI-MALWARE COMPARISON TEST Dennis Technology Labs www.dennistechnologylabs.com Follow @DennisTechLabs on Twitter.com This report aims to measure
ESET Endpoint Security 6 ESET Endpoint Antivirus 6 for Windows
ESET Endpoint Security 6 ESET Endpoint Antivirus 6 for Windows Products Details ESET Endpoint Security 6 protects company devices against most current threats. It proactively looks for suspicious activity
the barricademx end user interface documentation for barricademx users
the barricademx end user interface documentation for barricademx users BarricadeMX Plus The End User Interface This short document will show you how to use the end user web interface for the BarricadeMX
Internet Explorer Exploit Protection ENTERPRISE BRIEFING REPORT
Internet Explorer Exploit Protection ENTERPRISE BRIEFING REPORT TESTED PRODUCTS: AVG Internet Security Network Edition v8.0 Kaspersky Total Space Security v6.0 McAfee Total Protection for Endpoint Sophos
Anti-Virus Comparative
Anti-Virus Comparative Performance Test (Suite Products) Impact of Internet Security Software on System Performance Language: English October 2013 Last Revision: 19 th November 2013 Table of Contents 1.
Unified Threat Management, Managed Security, and the Cloud Services Model
Unified Threat Management, Managed Security, and the Cloud Services Model Kurtis E. Minder CISSP Global Account Manager - Service Provider Group Fortinet, Inc. Introduction Kurtis E. Minder, Technical
Problematic, Unloved and Argumentative: What is a potentially unwanted application (PUA)?
Problematic, Unloved and Argumentative: What is a potentially unwanted application (PUA)? Revised 11-29-2011 Aryeh Goretsky, MVP, ZCSE Table of Contents Introduction 3 The formal definition 3 Here s a
User Manual. HitmanPro.Kickstart User Manual Page 1
User Manual HitmanPro.Kickstart User Manual Page 1 Table of Contents 1 Introduction to HitmanPro.Kickstart... 3 2 What is ransomware?... 4 3 Why do I need HitmanPro.Kickstart?... 6 4 Creating a HitmanPro.Kickstart
Security Industry Market Share Analysis
Security Industry Market Share Analysis September 2011 Introduction The Research OPSWAT releases quarterly market share reports for several sectors of the security industry. This quarter s report includes
The impact of virtualization security on your VDI environment
ENTERPRISE TE The impact of virtualization security on your VDI environment ST ED WITH LO G I N VS I Contents Introduction... 3 What is VDI?... 3 Virtualization security challenges... 3 Choosing the right
Upgrade to Webtrends Analytics 8.7: Best Practices
Technical Best Practices 8.7 Software Upgrade Upgrade to Webtrends Analytics 8.7: Best Practices Version 3 Webtrends Analytics is a powerful application that must be installed on a dedicated computer.
What is PC Matic?...4. System Requirements...4. Launching PC Matic.5. How to Purchase a PC Matic Subscription..6. Additional Installations.
USER Manual Table of Contents Getting Started What is PC Matic?...4 System Requirements....4 Launching PC Matic.5 How to Purchase a PC Matic Subscription..6 Additional Installations. 6 Registration...6
Trend Micro Cloud App Security for Office 365. October 27, 2015 Trevor Richmond
Trend Micro Cloud App Security for Office 365 October 27, 2015 Trevor Richmond Too many malware incidents >90% Targeted Attacks Start with Email Attackers: Target specific companies or individuals Research
Real World and Vulnerability Protection, Performance and Remediation Report
Real World and Vulnerability Protection, Performance and Remediation Report A test commissioned by Symantec Corporation and performed by AV-Test GmbH Date of the report: September 17 th, 2014, last update:
Lab Testing Summary Report
Lab Testing Summary Report February 14 Report 132B Product Category: Web Security Gateway Vendor Tested: Key findings and conclusions: security appliance exhibits best rate to date, 91.3%, for classifying
ANDRA ZAHARIA MARCOM MANAGER
10 Warning Signs that Your Computer is Malware Infected [Updated] ANDRA ZAHARIA MARCOM MANAGER MAY 16TH, 2016 6:05 Malware affects us all The increasing number of Internet users worldwide creates an equal
Check Point: Sandblast Zero-Day protection
Check Point: Sandblast Zero-Day protection Federico Orlandi Itway Support Engineer 2015 Check Point Software Technologies Ltd. 1 Check Point Threat Prevention SandBlast IPS Antivirus SandBlast stops zero-day
Bandwidth consumption: Adaptive Defense and Adaptive Defense 360
Contents 1. 2. 3. 4. How Adaptive Defense communicates with the Internet... 3 Bandwidth consumption summary table... 4 Estimating bandwidth usage... 5 URLs required by Adaptive Defense... 6 1. How Adaptive
SonicWALL Email Security 7.2.1 Appliance
Email Security SonicWALL Email Security 7.2.1 Appliance System Compatibility SonicWALL Email Security 7.2.1 is supported on the following SonicWALL Email Security appliances: SonicWALL Email Security 200
When your users take devices outside the corporate environment, these web security policies and defenses within your network no longer work.
Deployment Guide Revision C McAfee Web Protection Hybrid Introduction Web Protection provides the licenses and software for you to deploy Web Gateway, SaaS Web Protection, or a hybrid deployment using
Secure Your Mobile Workplace
Secure Your Mobile Workplace Sunny Leung Senior System Engineer Symantec 3th Dec, 2013 1 Agenda 1. The Threats 2. The Protection 3. Q&A 2 The Mobile Workplaces The Threats 4 Targeted Attacks up 42% in
How To Test Security Products
Virtual Desktop Anti-malware Protection A COMPARATIVE TEST BETWEEN SYMANTEC ENDPOINT PROTECTION AND TREND MICRO DEEP SECURITY Dennis Technology Labs, 05/04/2012 www.dennistechnologylabs.com This report
The enemy within: Stop students from bypassing your defenses
The enemy within: Stop students from bypassing your defenses Computer literate K-12 students regularly use anonymizing proxies to bypass their school s web filters to access pornography, social networking,
Get Started Guide - PC Tools Internet Security
Get Started Guide - PC Tools Internet Security Table of Contents PC Tools Internet Security... 1 Getting Started with PC Tools Internet Security... 1 Installing... 1 Getting Started... 2 iii PC Tools
Tested product: Auslogics BoostSpeed
Software Tested Tested product: Auslogics BoostSpeed www.softwaretested.com CONTENTS 1 Contents Background... 3 Purpose of the Tests... 5 Testing Environment... 6 Test Results... 10 Windows Startup Time...
Install Guide VirusBlokAda
Antivirus / Anti-spam Vba32 for Microsoft Exchange VirusBlokAda Copyright 1993-2011 ОДО VirusBlokAda Ltd. Documentation version: 0.5 (March 2011) All rights reserved. All contents, graphics and texts,
Home Anti-Virus Protection
Home Anti-Virus Protection APRIL - JUNE 2013 Dennis Technology Labs www.dennistechnologylabs.com This report aims to compare the effectiveness of anti-malware products provided by well-known security companies.
Symantec Endpoint Protection 12.1.6
Data Sheet: Endpoint Security Overview Last year, we saw 317 million new malware variants, while targeted attacks and zero-day threats were at an all-time high 1. The threat environment is evolving quickly
Data Sheet: Endpoint Security Symantec Protection Suite Enterprise Edition Trusted protection for endpoints and messaging environments
Trusted protection for endpoints and messaging environments Overview Symantec Protection Suite Enterprise Edition creates a protected endpoint and messaging environment that is secure against today s complex
Small Business Endpoint Protection Performance Benchmarks
Small Business Endpoint Protection Performance Benchmarks Windows 7 February 2011 Document: Small Business Protection Performance Benchmarks Authors: K. Lai, D. Wren Company: PassMark Software Date: 9
Full Drive Encryption with Samsung Solid State Drives
Full Drive with Solid State Drives A performance and general review of s new selfencrypting solid state drives. Trusted Strategies LLC Author: Bill Bosen November 2010 Sponsored by Electronics Full Drive
Application Whitelisting - Extend your Security Arsenal? Mike Baldi Cyber Security Architect Honeywell Process Solutions
Application Whitelisting - Extend your Security Arsenal? Mike Baldi Cyber Security Architect Honeywell Process Solutions 1 Agenda What is Application Whitelisting (AWL) Protection provided by Application
Next-Generation Firewalls: Critical to SMB Network Security
Next-Generation Firewalls: Critical to SMB Network Security Next-Generation Firewalls provide dramatic improvements in protection versus traditional firewalls, particularly in dealing with today s more
The Advantages of Security as a Service versus On-Premise Security
The Advantages of Security as a Service versus On-Premise Security ABSTRACT: This document explores the growing trend of hosted/managed security as a service and why the cloud is quickly becoming the preferred
Integrated Approach to Network Security. Lee Klarich Senior Vice President, Product Management March 2013
Integrated Approach to Network Security Lee Klarich Senior Vice President, Product Management March 2013 Real data from actual networks 2 2012, Palo Alto Networks. Confidential and Proprietary. 2008: HTTP,
Anti-Virus Comparative
Anti-Virus Comparative File Detection Test of Malicious Software including false alarm test Language: English March 2015 Last Revision: 30 th April 2015 Table of Contents Tested Products 3 Introduction
STPIC/Admin/002/2009-2010/ Date: 18.06.09. Sub: Quotation for purchase/renewal of Anti Virus Software Reg.
STPIC/Admin/002/2009-2010/ Date: 18.06.09 Sub: Quotation for purchase/renewal of Anti Virus Software Reg. Software Technology Parks of India (STPI), an autonomous Society under Ministry of Communication
Email security Cloud vs. On-premise solutions
GFI White Paper Email security Cloud vs. On-premise solutions Choosing whether to put your email security in the cloud or host it on premise is a major decision. Hopefully this white paper will help. Contents
RESCUE CD. User s Guide
RESCUE CD User s Guide 2 CONTENTS 1. When to use the Rescue CD 2 1.1 System Requirements 2 1.2 Using the Rescue CD 3 1.2.1 Using a USB drive 3 Recommended method 3 Alternative method 4 1.2.2 Automate the
Norton Mobile Privacy Notice
Effective: April 12, 2016 Symantec and the Norton brand have been entrusted by consumers around the world to protect their computing devices and most important digital assets. This Norton Mobile Privacy
Office 365 Migration Performance & Server Requirements
Solutions for the Modern Workforce Office 365 Migration Performance & Server Requirements Understanding Migration Performance There are many factors that can affect performance when migrating to Office
F-Secure Internet Security 2014 Data Transfer Declaration
F-Secure Internet Security 2014 Data Transfer Declaration The product s impact on privacy and bandwidth usage F-Secure Corporation April 15 th 2014 Table of Contents Version history... 3 Abstract... 3
Maintaining, Updating, and Protecting Windows 7
Lesson 7 Maintaining, Updating, and Protecting Windows 7 Learning Objectives Students will learn to: Understand Disk Defragmenter Understand Disk Cleanup Understand Task Scheduler Understand Action Center
