Commercial in Confidence. Test Report December Kaspersky Whitelist Database

Transcription

1 Cmmercial in Cnfidence Test Reprt December 2011 Kaspersky Whitelist Database

2 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt WCL Crprate Offices and Test Facilities USA Headquarters and Test Facility West Cast Labs, Vn Karman Avenue, Suite 125, Irvine, CA 92606, U.S.A. Tel: +1 (949) , Fax: +1 (949) Eurpean Headquarters and Test Facility West Cast Labs, Unit 9 Oak Tree Curt, Mulberry Drive, Cardiff Gate Business Park, Cardiff, CF23 8RS, UK. Tel: +44 (0) , Fax: +44 (0) Asia Headquarters and Test Facility West Cast Labs, A2/9 Lwer Grund Flr, Safdarjung Enclave, Main Africa Avenue Rad, New Delhi , India. Tel: +91 (0) , Fax: +91 (0) Date: 14 th December 2011 Versin: 1.0 Authrs: Richard Thmas, Mark Thmas, Matt Garrad, Chris Thmas Page 2 f 54 West Cast Labs 2011

3 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Cntents Intrductin 4 Cmpetitive vendrs 5 Test Objectives 6 Test Netwrk 7 Test Cllectin 8 Test Methdlgy 10 Test Results 17 Test Result Database Cverage 18 Test Result Database Fullness 22 Test Result Database Speed 27 Test Result Database False Rate 32 Test Result Additinal Services 34 Test Result Default Deny Mde 38 Cnclusin 40 Appendix A 41 Appendix B 47 Disclaimer 52 Revisin Histry 53 Page 3 f 54 West Cast Labs 2011

4 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Intrductin Fllwing discussins between representatives f Kaspersky Lab (hereafter referred t as Kaspersky) and West Cast Labs (hereafter referred t as WCL) a test utline was cnstructed fr, and agreed with, Kaspersky related t the testing f the database used by Kaspersky t prvide their whitelisting technlgies. This dcument frms the reprt f thse tests carried ut against the Whitelist database, lking at several aspects f the slutin including database cverage, fullness, speed f access, false psitive/negative rates, default deny capabilities and additinal services that Kaspersky has made available t partners and prspective partners. It is imprtant t nte that the testing cnducted as part f this framewrk, and which is summarised in this reprt des nt verify the peratin f Kaspersky s whitelisting technlgy itself, r any implementatins f the technlgy, but nly assesses the underlying database. The technical lkup prtins f this test were cnducted using a web service based applicatin, prvided by Kaspersky t WCL, which allwed MD5 checksums (a methd f file identificatin used fr data integrity checking) f files t be passed t the service as a parameter, with the end result being the return f recrds frm the database related t each specific checksum that shwed the apprpriate metainfrmatin. The rest f this dcument details the high level test bjectives, the test envirnment that was used, the cmpsitin f the test cllectin, individual test cases, and their assciated methdlgy, test results, and finally a cnclusin drawing tgether all f the tests and prcesses. Page 4 f 54 West Cast Labs 2011

5 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Cmpetitive vendrs At Kaspersky s request, WCL apprached several ther vendrs in the whitelisting database space t request their invlvement in a cmparative test based arund the methdlgies, cllectins, and analysis as detailed belw. Each vendr was given the pprtunity t view their wn results against each f the tests and categries that were apprpriate t them, and had the ptin t request annymisatin within this final reprt. All three cmpetitive vendrs plitely declined t take part and, given that this srt f testing requires direct access int a vendr s databases fr accurate results, the matter culd unfrtunately nt be prgressed any further. The vendrs apprached were: Harris Crpratin (frmerly SignaCert) Lumensin Bit9 Page 5 f 54 West Cast Labs 2011

6 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Objectives The verall bjective f this testing prgramme was t determine the effectiveness f the whitelist database by cnducting a series f test cases, each designed t assess a specific area f the database and assciated prcesses. Results are prvided in the fllwing areas, alng with discussin: Test Case 01 - Database Cverage (TC01) This test case aimed t verify the cverage f the database in terms f cnsumer and crprate specific sftware, as well as regin specific sftware. Test Case 02 Database Fullness (TC02) This test case aimed t determine the quality and quantity f infrmatin and metainfrmatin included within the database fr the MD5 checksum (hereafter referred t as MD5) assciated with each f the files. Test Case 03 Database Speed (TC03) The aim f this test is t examine the respnsiveness f the Whitelisting database when returning data fr a given MD5 r grups f MD5s. Test Case 04 Database False Rate (TC04) This test case will examine the trustwrthiness r crrectness f the database when returning metainfrmatin fr bth knwn-bad and knwn-gd MD5s. Test Case 05 Additinal Services (TC05) This test case will examine and reprt upn the added functinality that is extended t partners f Kaspersky. Test Case 06 Default Deny (TC06) This test case will examine whether the database cntains infrmatin that is necessary fr running a default deny mde. Page 6 f 54 West Cast Labs 2011

7 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Netwrk Testing was carried ut at WCL s Eurpean Headquarters, lcated in Cardiff, UK. The Ubuntu variant f the Linux OS was installed t a number f machines lcated within the lab envirnment. Once installatin f the OS was cmplete, the Oracle VirtualBx applicatin was installed n each f the machines and cnfigured t run an instance f Windws XP patched t Service Pack 3. A Windws-based applicatin, prvided by Kaspersky Lab t enable direct access t their whitelist database, was then placed n the rt f each Windws XP virtual machine. The test cllectin f MD5 checksums was then parsed thrugh these applicatins, with the web traffic being transprted ver Cisc switches and ruters and a live Internet cnnectin as prvided by a UK-based ISP. Page 7 f 54 West Cast Labs 2011

8 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Cllectin The test cllectin prepared by WCL cnsisted f a series f MD5 checksums that were taken f Windws-based sftware. This sftware was surced frm different lcatins and cvered a number f different gegraphical areas. The number f checksums in the verall cllectin, after deduplicatin, was ver 843,000 and included all file types related t the applicatins being tested, nt just Prtable Executable types. The sftware surces used fr the cllectin were as fllws: Sftware distributed within WCL s usual ffice and testing envirnments, except where this was in breach f any existing NDAs r legal bligatins. Sftware distributed within WCL s parent cmpany Haymarket Media Grup, adhering t the same legal restrictins as abve. Cmmnly used sftware frm Adbe (such as Acrbat Reader) and frm Micrsft, including base versins f the Micrsft Operating Systems Windws XP, Windws Vista, and Windws 7, alng with varius Operating System patches frm Micrsft. Extracts frm the Kaspersky Security Netwrk f ppular sftware amng Kaspersky users (a separate and distinct entity frm the whitelisting database being tested). Ttal number f MD5s was nt mre than 6% f ttal cllectin. Sftware acquired prir t the testing by WCL. This cvered: The tp 100 ppular dwnlads frm sites such as tucws, surcefrge, dwnlad.cm, brthersft and sftpedia in the mnth immediately befre the testing cmmenced. Vide and audi cdecs. Tp games in the week prir t the cmmencement f testing purchased frm an ver the cunter retailer lcal t WCL in the UK. This list was cnstructed frm an nline retailer (Amazn), a specialist Page 8 f 54 West Cast Labs 2011

9 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Cllectins nline and high street games retailer (game.c.uk), and a UK supermarket (Tesc). Digital media sftware and system drivers such as audi, netwrk, graphics, and mtherbard. Lcalised dwnlads and sftware frm Americas, Eurpe Eastern, Eurpe Western, APAC, and Middle East and Africa regins. Malware surced frm WCL s cllectins just prir t the cmmencement f testing, representing 20,000 distinct samples, including malware captured in WCL s glbal hneypt netwrk. WCL als apprached sme f Haymarket Media grup s partners and licensees and requested that they prvide (and purchase, where apprpriate) sftware lcalised fr their particular market, and links frm which users in their regin regularly dwnlad sftware. Page 9 f 54 West Cast Labs 2011

10 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Methdlgy Test Case 01 - Database Cverage (TC01) TEST OBJECTIVE This test aims t verify the cverage f the database, bth fr specific gegraphic regins, and als fr particular areas f sftware, such as cnsumer r crprate. The % cverage rati f files assciated with specific regins/cuntries (Eastern Eurpe, Western Eurpe, Americas, Middle East and Africa, APAC) The % cverage rati f files assciated with the cnsumer market The % cverage rati f files assciated with the crprate market TEST DESCRIPTION The cmbined list f MD5s frm WCL s central strage database that hused the test cllectin was exprted int a number f flat text files cntaining 50,000 MD5s each. They were further brken dwn int regin and then by crprate and cnsumer grupings. The regins used in the test were as fllws: Americas Western Eurpe Eastern Eurpe Middle-East & Africa Asia Pacific Each f these flat-text files was then parsed against the whitelist database using the interface t the live database supplied by Kaspersky. The returned utputs against each MD5 were saved t a cmbined lg file fr analysis, and varius Page 10 f 54 West Cast Labs 2011

11 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Methdlgy measurements were als taken, such as the time taken t perfrm the lkup, and start/finish times fr the prcess. T further aid in this, netwrk traffic captures were taken t prvide later verificatin. Using varius SQL queries, alng with in-huse analysis tls and scripts, engineers then calculated the ttal number f MD5s fr which data was returned by the whitelist database. Test Case 02 - Database Fullness (TC02) TEST OBJECTIVE This test aimed at verifying the structure and value f metainfrmatin that is available in the database abut each file frm the test cllectin. The result als aimed t reflect the usefulness f any assciated infrmatin fr analytical purpses accrding t a predetermined weighting system agreed in advance with Kaspersky. Each metainfrmatin parameter was assigned a weight, and thus a metric based apprach t evaluate the quality f the Whitelisting database culd be assessed based upn the cmpsitin and perceived imprtance f each parameter. TEST DESCRIPTION Using the utput generated by TC01, scripts were develped that wuld cmpile a list f all thse MD5s fr which data had been returned. Fllwing this, the scripts then extracted all f the data n a per MD5 basis and made a recrd f hw many, if any, f a predetermined set f data flags were returned. The specific data sets, r flags, that were used in this test were then awarded a weighted value based n their level f imprtance, accrding t apprpriate weighting system agreed in advance with Kaspersky. This allwed fr a weighted measure (expressed as a percentage scre) f data cmpleteness t be awarded Page 11 f 54 West Cast Labs 2011

12 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Methdlgy t each returned MD5 s that a judgement n the value and quality f the data culd be determined. Specific details f the weightings are detailed in Appendix A. Test Case 03 - Database Speed (TC03) TEST OBJECTIVE This test aimed t examine the respnsiveness f the whitelisting database using the returning f requests frm queries made against the database. Specifically, this testing cnsidered: The time taken fr the Whitelist database t respnd t verificatin requests against individual checksums. The time required fr the additin f a previusly unknwn clean file int the whitelist after it was submitted t Kaspersky fr verificatin. The time required fr Kaspersky t respnd t the reprting f, and take apprpriate actins upn, a reprted false psitive result. TEST DESCRIPTION Testing f the database speed was cnducted utilising several separate stages. The first cnsidered the independent verificatin f the lkup time as reprted by the database/lkup applicatin. This cnsisted f a manual verificatin cnducted by a WCL engineer, alng with an autmatic verificatin cnducted by the scripts that mnitred the start and finish times f the assciated windws applicatins as the cllectins were being parsed. Netwrk traffic captures were als utilised t prvide further verificatin f the timings. Page 12 f 54 West Cast Labs 2011

13 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Methdlgy Fllwing this, the full cllectins were reprcessed thrugh the database, using the methdlgy as stated fr TC01. This time, hwever, the test sets were split int grups f bth 10,000 and 1,000 MD5s. The timings fr these were subsequently analysed in rder t determine the length f average time recrded fr prcessing each f these grupings, as well as t ascertain the average time per single MD5 when prcessed in these grups. Test Case 04 - Database False Rate (TC04) TEST OBJECTIVE This test aimed t examine the trustwrthiness r crrectness f the database. The testing was specifically fcussed n the fllwing areas: Number f False Negatives i.e. thse files classified as genuine by the slutin but in fact malicius, based against a set f samples drawn frm WCL s cllectins f recent malware. Number f False Psitives i.e. thse files classified by the slutin t be malicius, but in fact genuine. Availability f the database in terms f being able t respnd t a file lkup request, irrespective f the result f the lkup. TEST DESCRIPTION A list f 20,000 MD5s that are assciated with files cntained within WCL s malware cllectins was cmpiled, with the criteria fr this being that these had been cllected just prir t the start f the test. These included samples taken frm WCL s Page 13 f 54 West Cast Labs 2011

14 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Methdlgy glbal hneypt netwrk, and s these files were live and in the wild. Using the methdlgy that was emplyed fr TC01, each f these MD5s was then parsed thrugh the applicatin against the database, and the subsequent returns and utputs were recrded. Subsequent t this, further analysis was als cnducted against the utput taken frm TC01 in rder t determine hw many f the knwn-gd files were reprted by the whitelist database as infected. Test Case 05 Additinal Services (TC05) TEST OBJECTIVE This case aimed t examine and reprt upn the added functinality that is extended t partners f Kaspersky and crprate cnsumers. Kaspersky whitelisting service ( Kaspersky Trusted service ( Als available n The examinatin f the whitelisting service invlved lking at the means by which a partner cmpany can submit upcming sftware releases fr inclusin int the whitelist, including ging thrugh the prcess itself. Investigatins f the Kaspersky Trusted service cnsidered the prcess that allws fr the prvisin f a Kaspersky apprval lg n a sftware dwnlad page, alngside infrmatin relating t the file. TEST DESCRIPTION In rder t test the Kaspersky Whitelisting Service, WCL created a series f new files Page 14 f 54 West Cast Labs 2011

15 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Methdlgy that were previusly unknwn t Kaspersky Lab r t their Whitelisting database. Using accunt infrmatin specific t WCL and prvided by Kaspersky Lab, each f these files was upladed t the Whitelisting service FTP server and the uplad time nted. A series f in-huse scripts were simultaneusly started that wuld cntinuusly check the database nce every 10 secnds, this having been agreed as a reasnable timeframe. These scripts were cnfigured t lk fr the newly added infrmatin related the MD5s f the files that had been upladed. The timestamp fr when this infrmatin appeared was recrded n a per file basis. A subsequent test was then perfrmed that measured the time it takes fr Kaspersky Lab t reclassify a file s status nce they have been cntacted by a custmer with a query. T d this, a specific given address was cntacted with a request t reclassify thse same files frm clean t suspicius. The same scripts used in the earlier test were nce again run and cnfigured t recrd the timestamp at which the classificatin was seen t have altered. A validatin test was als perfrmed fr the use f the Kaspersky Trusted lg within the Kaspersky Trusted service. Using the MD5 infrmatin fr ne f the files supplied earlier in this test case, a web page was cnstructed and the HTML cde prvided was included, with the apprpriate MD5 and accunt cde infrmatin. It was then visually verified that bth the Trusted lg and respective Trusted service page were available. Finally, a series f questinnaires were sent ut t existing custmers f the service. Each questinnaire was designed t gain an insight as t the verall user experience f the service. Page 15 f 54 West Cast Labs 2011

16 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Methdlgy Test Case 06 Default Deny Mde (TC06) TEST OBJECTIVE This test aimed t determine whether the whitelisting database cntained the necessary infrmatin which is necessary fr running a default deny mde when using Kaspersky s Endpint prduct (which utilises Kaspersky Whitelisting database). Nte that the database nly was tested, and nt the implementatin Default Deny mde is, in this case, defined as a restricted mde f PC peratin when everything is blcked except fr certain particular pieces sftware which are necessary fr the basic peratin and general functinality f a given system - in ther wrds the Operating System and critical drivers. TEST DESCRIPTION The checksums f varius example perating systems as per the list belw were submitted t the database using the tls and methdlgies as in TC01, and the results returned were recrded as t whether they were included in the default deny list r nt: Windws XP Service Pack 3 (32 bit) Windws 7 Embedded Editin (32 bit) Windws 7 Hme Editin (64 bit) Windws 7 Prfessinal (32 bit) Windws 7 Enterprise (32-bit) Windws 7 Enterprise (64-bit) Page 16 f 54 West Cast Labs 2011

17 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results This sectin cntains the results fr all testing cnducted as part f this reprt. Tests are brken dwn accrding t each f the Test Cases utlined abve, and detailed as fllws: TC01 - Database Cverage TC02 - Database Fullness r quality f data TC03 - Database Speed TC04 - Database False Rate TC05 - Additinal Service TC06 Default Deny mde supprt Page 17 f 54 West Cast Labs 2011

18 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC01 - Database Cverage Cverage by Regins and type f sftware Regin Cnsumer Sftware Files requested Files detected Knwn (%) Americas 131, , % Western Eurpe 188, , % Eastern Eurpe 14,116 12, % Middle East & Africa 5,010 4, % Asia & Pacific 41,547 38, % All Regins 380, , % Table 1.0 Cnsumer sftware cverage by regin Regin Crprate Sftware Files requested Files detected Knwn (%) Americas 184, , % Western Eurpe 265, , % Eastern Eurpe 9,935 9, % Middle East & Africa % Asia & Pacific 2,087 2, % All Regins 462, , % Table 1.1 Crprate sftware cverage by regin Regin Cnsumer + Crprate Sftware Files requested Files detected Knwn (%) Americas 315, , % Western Eurpe 454, , % Eastern Eurpe 24,051 22, % Middle East & Africa 5,499 4, % Asia & Pacific 43,634 40, % All Regins 843, , % Table 1.2 Crprate and Cnsumer sftware cverage by regin Page 18 f 54 West Cast Labs 2011

19 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC01 - Database Cverage The belw graphs shw verall cverage statistics by regin (Figure 1.0) and verall cverage by market segment (Figure 1.1) Cverage by regin Asia Pacific 94% 100% Middle East & Africa 88% 100% East Eurpe 91% 100% Cverage Cllectin West Eurpe 91% 100% Americas 98% 100% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 1.0 Cverage by gegraphical regin Cverage by market segment Cnsumer 93% 100% Cverage Cllectin Crprate 94% 100% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 1.1 Cverage by gegraphical regin Page 19 f 54 West Cast Labs 2011

20 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC01 - Database Cverage File Type The fllwing table (Table 1.3) lists the database cverage statistics based n specific extensins with a graphical representatin in Figure 1.2. Extensin Files requested Files identified Percent Identified.exe %.dll %.cm %.cab %.sys %.msi % Other files (.pdf,.gpg,.dc, etc.) % Table 1.3 Cverage by file extensin type % 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00%.EXE.DLL.COM.CAB.SYS.MSI Other files Figure 1.2 Cverage by file extensin type 15% 14%.EXE.DLL 12% 16%.COM.CAB.SYS 16% 11% 16%.MSI Other files Figure 1.3 Cverage by file extensin type Page 20 f 54 West Cast Labs 2011

21 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC01 - Database Cverage TC01 Result Analysis When lking at the data fr the MD5 test cllectin, it is clear that the whitelist database prvides infrmatin, nt just fr a large number MD5s, but als fr a large variety f MD5s. Never des this becme mre bvius than when lking at MD5s belnging t sftware dwnladed frm ppular dwnlad lcatins/mirrrs that serve the Asia Pacific and Middle East/Africa regins. MD5 recgnitin rates remained cnsistently high thrughut all f the tests. The lwest f these, crprate sftware dwnladed frm the Middle East & Africa regin, still registered an impressive 86% cverage. Of the ther categries, the result that prbably stands ut mst is the 98% cverage f all sftware dwnladed frm the Americas regin. When lking at individual file types, the traditinal Prtable Executable (PE) files such as.exe,.dll, and.cm all recrded a high cverage rate. Page 21 f 54 West Cast Labs 2011

22 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC02 - Database Fullness The fllwing table and graph (table 2.0 and figure 2.0) shw the scres fr the three main fields f file infrmatin based upn the results returned frm the database related t the MetaData attached t each f the checksums. A full descriptin f hw these scres were arrived at can be fund in Appendix A. These scres are based n data returned nly fr thse files cntained in the test suite that are executable (.exe) files. Data Weighted percentage scre Expertise 87.17% Statistics 80.57% Raw Data 87.17% Table 2.0 Database fullness based upn Kaspersky metadata flags -.EXE files nly Raw Data Statistics Expertise 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 2.0 Database fullness based upn Kaspersky metadata flags -.EXE files nly Page 22 f 54 West Cast Labs 2011

23 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC02 - Database Fullness The fllwing table and graph (table 2.1 and figure 2.1) shw the scres fr a number f metrics based upn the results returned frm the database related t the MetaData attached t each f the checksums. A full descriptin f hw these scres were arrived at can be fund in Appendix A. Data Weighted percentage scre Weighted percentage scre File Data Inf KSN inf File Prduct inf File Surce & Package Inf 72.1 Signatures & Certificates Inf Verdict Type & Cmment Users Trusted Level Categrizatin File Gegraphy & Ppularity Zne Inf 89.3 Table 2.1 Database fullness based upn Kaspersky metadata flags -.EXE files nly File Data Inf File Prduct inf Signatures & Certificates Users Trusted Lev File Gegraphy & Ppul KSN inf File Surce & Package Verdict Type & Cmm Categrizatin Zne Inf Figure 2.1 Database fullness based upn Kaspersky metadata flags -.EXE files nly. Page 23 f 54 West Cast Labs 2011

24 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC02 - Database Fullness The fllwing table and graph (table 2.1 and figure 2.1) shw the scres fr the three main fields f file infrmatin based upn the results returned frm the database related t the MetaData attached t each f the checksums. A full descriptin f hw these scres were arrived at can be fund in Appendix A. Wrth nting with the tables and charts belw is that this data is based n all file types included within the test set and that sme f the fields, such as File Gegraphy, are related slely t executable files. Data Weighted percentage scre Expertise 97.37% Statistics 88.91% Raw Data 85.01% Table 2.0 Database fullness based upn Kaspersky metadata flags - all files Expertise Statistics Raw Data 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 2.0 Database fullness based upn Kaspersky metadata flags all files Page 24 f 54 West Cast Labs 2011

25 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC02 - Database Fullness The fllwing table and graph (table 2.1 and figure 2.1) shw the scres fr a number f metrics based upn the results returned frm the database related t the MetaData attached t each f the checksums. A full descriptin f hw these scres were arrived at can be fund in Appendix A. Data Weighted percentage scre Weighted percentage scre File Data Inf 97.3 KSN inf File Prduct inf File Surce & Package Inf Signatures & Certificates Inf Verdict Type & Cmment Users Trusted Level 3.89 Categrizatin File Gegraphy & Ppularity 2.47 Zne Inf Table 2.1 Database fullness based upn Kaspersky metadata flags all files File Data Inf File Prduct inf Signatures & Certificates Users Trusted Lev File Gegraphy & Ppul KSN inf File Surce & Package Verdict Type & Cmm Categrizatin Zne Inf Figure 2.1 Database fullness based upn Kaspersky metadata flags all files. Page 25 f 54 West Cast Labs 2011

26 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC02 - Database Fullness TC02 Result Analysis It can be seen frm the abve results that, whilst certain areas f the metadata in the database d still require backfilling, the imprtant sectins related t the File Data Inf, the File Surce and Package inf, the Verdict Types and Cmments, Zne inf and Expertise are all well ppulated these are the cre f the infrmatin n which decisins will likely be made within any implementatin. A marked difference can als be nted when cmparing the file infrmatin prvided specifically fr executable files. Page 26 f 54 West Cast Labs 2011

27 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC03 - Database Speed Whitelisting database reply speed (Gruped by 1,000 MD5s) The fllwing tables display the time taken fr the Whitelist database t respnd t verificatin requests against individual checksums. These are the times recrded by West Cast Labs. The specific applicatin prvided by Kaspersky fr use in the testing als returned a lkup time as part f its functinality, and the differences between the Kaspersky reprted time and the time measured by West Cast Labs were well within the margins f errr. These verificatins were cnducted acrss 10 separate runs, each cnsisting f 1,000 unique MD5s, taking the verificatin test set t 10,000 unique MD5s in ttal. Run f 1000 MD5s Recrded Time (mm:ss:00) Run f 1000 MD5s Recrded Time (mm:ss:00) 1 01: : : : : : : : : :14.90 Table 3.0 Recrded times fr 1000 unique MD5 lkups (mm:ss:00) Run 01 Run 02 Run 03 Run 04 Run 05 Run 06 Run 07 Run 08 Run 09 Run 10 Figure 3.0 Recrded times fr 1000 unique MD5 lkups in secnds Page 27 f 54 West Cast Labs 2011

28 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC03 - Database Speed - Gruped by 10,000 MD5s The fllwing graph (figure 3.1) shws the distributin lkup times recrded fr the entire MD5 test cllectin (rather than 10 individual runs f 1000 each) when parsed thrugh the whitelist tl in grups f 10,000. Average Lkup Time fr a grup f 10,000 MD5s: 5 minutes 13 secnds Average Lkup Time per MD5 when parsed in grups f 10,000: 0.03 secnds Time (Minute Runs Gruped by 10,000 MD5s Figure 3.1 runs f checksum lkups gruped by 10,000 MD5s ver entire test cllectin This shws the 73 runs gruped against the time taken in minutes - as can be seen, there were very few runs verall that tk substantially lnger than abut 7 minutes t return when querying 10,000 checksums at a time. Results appearing in the tp and bttm 10% f the verall range were discarded in rder t avid skew frm pssible transient netwrk cnditins. Page 28 f 54 West Cast Labs 2011

29 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC03 - Database Speed Gruped by 1,000 MD5s The fllwing graph (figure 3.2) displays the lkup times recrded fr the entire MD5 test cllectin (rather than 10 individual runs when parsed thrugh the whitelist tl in grups f 1, sets f runs. Average Lkup Time fr a grup f 1,000 MD5s: 18.5 secnds Average Lkup Time per MD5 when parsed in grups f 1,000: 0.02 secnds Time (Secnd Runs Gruped by 1,000 MD5s Figure 3.2 runs f checksum lkups gruped by 1,000 MD5s ver entire test cllectin Once again, it can be seen that the majrity f runs returned very quickly with nly a few that tk than abut 40 secnds lnger t return. Results appearing in the tp and bttm 10% f the verall range were discarded in rder t avid skew frm pssible transient netwrk cnditins. Page 29 f 54 West Cast Labs 2011

30 Cmmercial in Cnfidence Kaspersky Whitelisting - Test Reprt Test Results TC03 - Database Speed Additin f new files t Whitelist The fllwing graph (figure 3.3) shws the time taken in the prcess f the additin f a previusly unknwn clean file int the whitelist fllwing its submissin t Kaspersky fr verificatin. The average time in this case is just ver 1 hur and 13 minutes. The secnd graph (figure 3.4) shws the time required fr Kaspersky t respnd t the reprting f, and take apprpriate actins upn, a reprted false psitive result. The average time in this case is just ver 1 minute and 50 secnds. Time (Minutes) Files Figure 3.3 Time taken t add files t whitelist database Time (Secnds) Files Figure 3.4 Time taken t change file reputatin in whitelist database Page 30 f 54 West Cast Labs 2011