Design and Evalua.on of a Real- Time URL Spam Filtering Service

Transcription

1 Design and Evalua.on of a Real- Time URL Spam Filtering Service Kurt Thomas, Chris Grier, Jus.n Ma, Vern Paxson, Dawn Song University of California, Berkeley Interna.onal Computer Science Ins.tute

2 Mo.va.on Social Networks (Facebook, TwiMer) Spam Blogs, Services (Blogger, Yelp) Web Mail (Gmail, Live Mail)

3 Mo.va.on Exis.ng solu.ons: Blacklists Service- specific, account heuris.cs Develop new spam filter service: Filter spam: scams, phishing, malware Real-.me, fine- grained, generalizable

4 Overview Our system Monarch: Accepts millions of URLs from web service Crawls, labels each URL in real-.me Spam Classifica.on Decision based on URL content, page behavior, hos.ng Large- scale; distributed collec.on, classifica.on Implemented as a cloud service

5 Monarch in Ac.on URL Spam Account Social Network

6 Monarch in Ac.on URL Monarch Spam Account Social Network

7 3. Fetch Content Monarch in Ac.on URL Monarch Spam Account Social Network Spam URL Content

8 3. Fetch Content Monarch in Ac.on URL Monarch Spam Account Social Network Spam URL Content

9 3. Fetch Content Monarch in Ac.on URL Monarch Spam Account Social Network Message Recipients Spam URL Content

10 Challenges Accuracy Real- Time Scalability Tolerant to Feature Evolu.on

11 Outline Architecture Results & Performance Limita.ons Conclusion

12 System Architecture

13 System Architecture

14 System Architecture

15 System Architecture

16 URL Aggrega.on Source Spam URLs Blacklisted TwiMer URLs Non- spam TwiMer URLs Sample Size 1.25 million 567,000 9 million Collec.on period: 9/8/ /29/2010

17 Feature Collec.on High Fidelity Browser NavigaGon Lexical features of URLs (length, subdomains) Obfusca.on (directory opera.ons, nested encoding) HosGng IP/ASN A, NS, MX records Country, city if available

18 Feature Collec.on Content Common HTML templates, keywords Search engine op.miza.on Content of request, response headers Behavior Prevent naviga.ng away Pop- up windows Plugin, JavaScript redirects

19 Classifica.on Distributed LogisGc Regression Data overload for single machine

20 Classifica.on Distributed LogisGc Regression Data overload for single machine L1- regularizagon Reduces feature space, over- figng 50 million features - > 100,000 features

21 Implementa.on System implemented as a cloud service on Amazon EC2 AggregaGon: 1 machine Feature CollecGon: 20 machines Firefox, extension + modified source ClassificaGon & Feature ExtracGon: 50 machines Hadoop - Spark, Mesos Straighjorward to scale the architecture

22 Result Overview High- level summary: Performance Overall accuracy Highlight important features Feature evolu.on Spam independence between services

23 Performance Rate: 638,000 URLs/day Cost: $1,600/mo Process.me: 5.54 sec Network delay: 5.46 sec Can scale to 15 million URLs/day Es.mated $22,000/mo

24 Measuring Accuracy Dataset: 12 million URLs (<2 million spam) Sample 500K spam (half tweets, half ) Sample 500K non- spam Training, Tes.ng 5- fold valida.on Vary training folds non- spam:spam ra.o Test fold equal parts spam, non- spam

25 Overall Accuracy Training RaGo Accuracy False PosiGve Rate False NegaGve Rate 1:1 94% 4.23% 7.5% 4:1 91% 0.87% 17.6% 10:1 87% 0.29% 26.5% Correctly labeled samples Non- spam labeled as spam Spam labeled as non- spam

26 Overall Accuracy Training RaGo Accuracy False PosiGve Rate False NegaGve Rate 1:1 94% 4.23% 7.5% 4:1 91% 0.87% 17.6% 10:1 87% 0.29% 26.5% Correctly labeled samples Non- spam labeled as spam Spam labeled as non- spam

27 Error by Feature Error (%) Error False Posi.ve Rate Error = 1 - Accuracy

28 Error by Feature Error (%) Error False Posi.ve Rate Error = 1 - Accuracy

29 Error by Feature Error (%) Error False Posi.ve Rate Error = 1 - Accuracy

30 Feature Evolu.on Retraining Required Accuracy (%) Sep 16- Sep 20- Sep 24- Sep With Retraining Without Retraining

31 Spam Independence Unexpected result: TwiMer, spam qualita.vely different Training Set TesGng Set Accuracy False NegaGves TwiRer TwiRer 94% 22% TwiMer 81% 88% TwiMer 80% 99% 99% 4%

32 Spam Independence Unexpected result: TwiMer, spam qualita.vely different Training Set TesGng Set Accuracy False NegaGves TwiMer TwiMer 94% 22% TwiRer 81% 88% TwiRer 80% 99% 99% 4%

33 Dis.nct , TwiMer Features

34 Features Shorter Lived

35 Limita.ons Adversarial Machine Learning We provide oracle to spammers Can adversaries tweak content un.l passing? Time- based Evasion Change content aser URL submimed for verifica.on Crawler Fingerprin.ng Iden.fy IP space of Monarch, fingerprint Monarch browser client Dual- personality DNS, page behavior

36 Related Work C. WhiMaker, B. Ryner, and M. Nazif, Large- Scale Automa1c Classifica1on of Phishing Pages J. Ma, L. Saul, S. Savage, and G. Voelker, Iden1fying suspicious URLs: an applica1on of large- scale online learning Y. Zhang, J. Hong, and L. Cranor, Can1na: a content- based approach to detec1ng phishing web sites M. Cova, C. Kruegel, and G. Vigna, Detec1on and analysis of drive- by- download afacks and malicious JavaScript code

37 Conclusion Monarch provides: Real-.me scam, phishing, malware detec.on Experiments show 91% accuracy, 0.87% false posi.ves Readily scalable cloud service Applicable to all URL- based spam Spam not guaranteed to overlap between web services TwiMer, qualita.vely different Despite overlap, can s.ll provide generalizable filtering Require training data from each service