Using big data analytics to identify malicious content: a case study on spam s

Size: px

Start display at page:

Download "Using big data analytics to identify malicious content: a case study on spam emails"

Stuart Holmes
8 years ago
Views:

1 Using big data analytics to identify malicious content: a case study on spam s Mamoun Alazab & Roderic Broadhurst Mamoun.alazab@anu.edu.au

emails Mamoun Alazab & Roderic Broadhurst

2 2 Outline Background Cybercrime and SPAM? Importance of Big Data Analytics Data description & Analysis Summary Q&A

Regulation Information Security Malware Analysis Phishing

3 Background ( ANU Cybercrime Observatory 1 ) Team Research Interests: Criminology/Sociology Organised Crime Law & Regulation Information Security Malware Analysis Phishing attacks Police and media cases Computer Forensics 1 3

4 Spammed Messages Social Networking Websites Worm Malicious Websites Install Malware Become Zombie Removable Devices Spam as social engineering, enables malware to reach high volume low value targets that make it one of the popular means for spreading and injecting malware on computers. 4

engineering, enables malware to reach high volume low value targets

5 5 Cybercrime definition from legislation Definition of high tech crime (Australian Federal Police) 1 : High tech crime offences are defined in Commonwealth legislation in Part Computer Offences of the Criminal Code Act 1995 and include: computer intrusions (for example, malicious hacking) unauthorised modification of data, including destruction of data denial-of-service (DoS) attacks distributed denial of service (DDoS) attacks using botnets the creation and distribution of malicious software (for example, viruses, worms, trojans). 1 AFP, link:

7 - Computer Offences of the Criminal Code Act 1995 and include: computer intrusions (for example, malicious hacking) unauthorised modification of data,

6 6 Spam (Def.) it is hard to define the term spam accurately. Some distinct spam is an issue about consent not content, and while others believe it is the issue of content not the consent. Also some other believes it is about quantity or scale. In general, the word spam is commonly used to describe unsolicited s that are sent in bulk 1. Certain definitions also stress the commercial nature of spam 2. 1 Commission communication, on unsolicited commercial communications or "spam", p. 5 2 For example, the US CAN-SPAM act of 2003 establishes requirements for those who send commercial .

Also some other believes it is about quantity or scale.

7 7 Cont. In Australia, ACMA defined spam as unsolicited commercial electronic messages; also a single electronic message can also be considered spam under Australian law. On the other hand, Spamhaus defined spam differently and consider an is a spam only if it is both unsolicited and sent in bulk. "bulk", "commercial" and " unsolicited " are on themselves problematic, as they do not provide enough flexibility to deal with the variety of the content that is distributed using modern digital means of communications.

8 8 Why Big Data when fighting spam Dealing with spam introduces a number of Big Data challenges. The total size and scale of the data is enormous. In the 1990s, the average PC user received one or two spam messages a day. The amount of spam was currently estimated to be 200 billion messages sent per day (circa August 2010 see Josh Halliday, 2011; Syamtec, MAAWG, 2013 ). The suppression of spam involves the need to understand complex patterns of behavior and the capacity to identify new types of spam. Around 96% of all messages are estimated to be spam.

The amount of spam was currently estimated to be 200 billion messages sent per day (circa August 2010 see Josh Halliday, 2011; Syamtec,

9 9 Cont. Of all spam s sent on any one day, an average of 3.3% contained malicious attachments and higher for suspect web-pages (perhaps 1 in 5 ephemeral). Spammers collect gross world wide revenue on the order of $200 million per year ( Google, Microsoft, Yahoo, 2012) Spam now is associated with the recent crime toolkits. i.e. Blackhole, Zeus).

10 10 Focus s containing malicious contents, they attempt to compromise the security of a computer and try to lure the recipient to click on a fake or infected URL that links to a malicious Web site ( landing page ) or downloads a malicious attachment with a zero-day exploit. Regardless of the source i.e. phishing or spear phishing.

URL that links to a malicious Web site ( landing page ) or downloads a malicious

11 11 Data Set Data provided from the Australian Communication Media Authority's (ACMA) Spam Intelligence Database (SID) & the Computer Emergency Response Team (CERT) Australia. SID - Three sources received in anonymised. Only 2 data sources have been processed thus far (2012). Data (spam) are in the Millions in raw format. Our analysis only looked at the messages which appear to have been relayed through Australia, for example last hop IP address was located in Australia.

Only 2 data sources have been processed thus far (2012). Data (spam) are in the Millions in raw format.

12 Month Habul data set Botnet Data set # Spam s # spam s Jan 67 31,991 Feb ,085 Mar 75 45,413 Apr 65 33,311 May 83 28,415 Jun 94 11,587 Jul 72 16,251 Aug 85 21,970 Sep ,819 Oct 73 13,426 Nov ,145 Dec 95 20,696 Total 1, ,109 12

28,415 Jun 94 11,587 Jul 72 16,251 Aug 85 21,970 Sep 363 27,819

13 13 3 V s of Big Data Machine learning and data mining are well established techniques in the world of IT and especially among web companies and startups. Spam detection is made possible by mining the huge amount of data available and at play. However, big data is not only about Volume, but also about Velocity, and Variety (The 3V s of big data). Volume Data Quantity Velocity Data Speed Variety Data Type

Spam detection is made possible by mining the huge amount of data available and at play.

14 14 Attacks Malware and phishing are becoming combined Poisoned attachments (Ex. custom PDF exploits) Links to web sites with malware (web browser exploits) Install Trojan or remote access software Attackers use Fake domains: PayPal vs. PayPaI <= I not L Compromised Sites: hosting malicious software URL Shorting services: Hides real URL Droppers: malicious code on sites that drop malware upon visiting a site a webpage. Spear-phishing: targets specific groups or individuals usually attractive targets with limited guardianship Social engineered deceptive/tailored content (e.g. advanced fee frauds etc.)

PayPaI <= I not L Compromised Sites: hosting malicious software URL Shorting services: Hides real URL Droppers: malicious code on sites that drop malware

15 15 Trends (1/2) Spam and spam campaigns are often sent In large quantities At certain times or time frames Seemingly harmless URL that can redirect to compromised Web sites. Inconspicuous file names and extensions. tracking_instructions.pdf.zip Social Engineering tactics - Attackers user common business terms in the file names as spear phishing bait. i.e. ups, amazon, HP_document, etc.

Inconspicuous file names and extensions. tracking_instructions.pdf.

16 16 Trends (2/2) Same attachments with different body. Same body with different attachment ZIP files remain the preferred file of choice for malware delivery over (potentially delivers a high payload). Malware is delivered in ZIP file format in an estimated 91% of identified cases in our data. Malware authors (spammers) focus on evasion (e.g. double extensions, obfuscation, change code) URL seems to be not working - Evidence of a so-called Waterhole attack.

17 17 Zeus Virus Zeus code injection Legitimate webpage

18 Ransomware 18

19 19 Identifying malicious spam s Parsed the raw data to database. Then, extract attachments and URLs and upload to VirusTotal 1 1 Free online virus checker that offers support for academic researchers, to scan for viruses and suspicious content. VirusTotal uses over 40 different virus scanners, where we consider an attachment or URL to be malicious if at least one scanner shows a positive result.

support for academic researchers, to scan for viruses and suspicious content.

20 Attachment 20

21 21 URLs URL seems to be not working - Evidence of using Waterhole attack.

22 22 Waterhole attack - Blackhole Exploit Kit (finding) Malicious website

23 23 Conclusion Predicating spam messages containing malicious contents are not possible without the systematic analysis of big data, previous knowledge of current threats and likely development in modus operandi. Propose using only spam text to predict malicious attachments and URLs (ask for the paper) Novel features to capture text patterns Self-contained (no external resources) We show we can predict malicious attachments up to 95.2%, and up to 68.1% for URLs. Machine learning and data analytics based on Big Data will improve the discovery of targeted attacks and persistent threats. (ask for the 2 nd paper)

24 Thank you and visit

Phishing Scams Security Update Best Practices for General User

Phishing Scams Security Update Best Practices for General User hishing refers to the malicious attack Pmethod by attackers who imitate legitimate companies in sending emails in order to entice people to