Big Data Analysis in Cyber Security. K P Chow Law and Technology Center Center for Information Security and Cryptography

Transcription

1 Big Data Analysis in Cyber Security K P Chow Law and Technology Center Center for Information Security and Cryptography University of Hong Kong Nov

2 Size of Attack Data Growing DDoS size now mostly ranges from 1Gbps to over 100Gbps, thus creating huge data logs for analysis 2

3 Log Review vs Big Data Analytics Extensive Sources of Data to be analyzed => Big Data Analytics Source: 3

4 BIG DATA IN MASSIVE CYBER ATTACKS CASE STUDY: CYBER ATTACKS BEFORE & AFTER OCM 4

5 The Beginning From June 2014 (after the d-day 3), Hong Kong citizens were invited to vote on a referendum on constitutional reforms that would guarantee all citizens the right to vote in elections that determine who will be the city's Chief Executive. To build a public consensus around a recent civil proposal on universal suffrage, the civic group Occupy Central with Love and Peace (OCLP) appointed the Public Opinion Programme (HKUPOP) at the University of Hong Kong to host the civil referendum on their servers. 5

6 The First Attack The voting system was hosted by Amazon Web Services (AWS), Cloudflare and UDomain. All three web hosting services suffered from large scale DDoS attacks on June 14 and 15: AWS recorded 10 billion system requests with 20 hours CloudFare recorded a 75Gb DDoS per second UDomain recorded 10Gb per second The Feb 2000 Attack Yahoo! received in excess of 1GB per second during the peak of the attack Buy.Com received traffic quantities approximately to 8 times of their total capacity 6

7 As the scale of attack is tremendous, all three service providers were forced to temporarily suspend their services. An expert estimated that there could be at least 5,000 but possibly more than 10,000 computers involved in the attack. CISC 7

8 Oct 2: Operation Hong Kong 'Anonymous' hacker group declares cyber war on Hong Kong government, police Greetings, world. We are Anonymous. Operation Hong Kong engaged. We are Anonymous. We are legion. We do not forgive. We do not forget. Government of Hong Kong, expect 8 us.

9 Oct 2: Anonymous declares war against HK Police Hactivist collective Anonymous, Antisec division, declared a cyber-war against Hong Kong for the treatment of protestors there. The group has already defaced several Hong Kongbased websites, and that dozens more would be affected over the next few days.

10 Cyber Attack during OCM

11 Cyber Attack after OCM

12 Total Number of Attacks Targeting Hong Kong (September and October, 2014) CISC 12

13 Peak Attack Sizes per Day (Gbps) in the first week of November It is apparent that DDoS attacks have become the new normal during periods of political unrest worldwide CISC 13

14 How to defense massive cyber attacks? Content Distribution Network CDN 14

15 Problems in Investigation Usually required to analyze large amount of attack data: do we have the data? Network forensics challenges Possibility of collecting all attack data Prerequisite settings for incident response and cyber attack analysis Cross jurisdiction issues - traceback

16 From Online to Offline CISC 16

17 CYBER CRIMINAL PROFILING USING BIG DATA CASE STUDY: PROFILING ONLINE FLASH MOB ORGANIZER AFTER OCM 17

18 After Occupy Central More conflicts between Hong Kong locals and mainland China visitors 18

19 From online to offline Flash-mob-like activity callings posted on the forum : Asking people to go shopping at a specified location and disperse afterwards 19

20 Questions of Interests Identification of Flash Mob Organizers online What are the key online behavioral attributes these flash-mob-like organizer(s) might have? Is it possible to identify these individuals at an early stage based on these attributes? Social Influence Index Is it possible to measure the social influence of these flash-mob-like organizer(s)? 20

21 Methodology STEP 1 STEP 2 STEP 3 Collection of data Extraction of key online behavioral attributes Classification of Topic Authors Identification of most influential flash mob organizer Discussion Forum is selected instead of Facebook because the privacy setting of Facebook might limit the data to be collected for research The key online behavioural attributes will be explained in the following slides Clustering will be used for the classification process A Social Influence Index will be introduced to measure the most influential flash mob organizer 21

22 STEP 1 STEP 2 STEP 3 Collection of data Extraction of key online behavioral attributes Classification of Topic Authors Identification of most influential flash mob organizer METHODOLOGY 22

23 Online Forum Variable name Description Value Forum_ID Identifier of a forum Nominal Forum_subcat Identifier of a forum subcategory Nominal (i) Topic variables Topic_ID Identifier of a topic ID Topic_author Author a new topic ID Topic_date Creation date of a new topic Date Topic_time Creation time of a new topic Time Topic_title Title of a new topic Text (i) Post variables Post_ID Identifier of a post under the topic ID Post_author Author a post under the topic ID Post _date Creation date of a post under the topic Date Post _time Creation time of a post under the topic Time Post _content Contents of a post under the topic Text 23

24 Big Data??? 200, , ,000 50,000 0 Total number of posts (Jan-Apr, 2015) 199, , ,400135,806 Jan Feb Mar Apr 6,000 5,000 4,000 3,000 2,000 1,000 0 Total number of topics (Jan-Apr, 2015) 3,273 3,896 5,912 4,174 Jan Feb Mar Apr These are just data from ONE subcategory. On average, there are: 4,314 topics per month, i.e. 51,765 topics per year 157,414 posts per month, i.e. 1,888,971 posts per year There are 33 subcategories under just ONE forum! 24

25 2 types of online users 16,000 14,000 12,000 10,000 8,000 6,000 4,000 2, ,484 13,221 1,026 1,154 1,506 1,161 Jan Feb Mar Apr Total number topic authors 14,694 14,343 Total number post authors These are just data from ONE subcategory. On average, there are: 1,212 topic-authors per month, i.e. 14,541 topic-authors per year 13,936 post-authors per month, i.e. 167,226 post-authors per year There are 33 subcategories under just ONE forum! 25

27 Type of Posts Empty & Spam Typical front page extracted from a subcategory of the forum: In order to keep a topic on the front page, topic-authors or post-authors might create some empty/spam posts to a topic. A typical empty/spam posts Spam post with the word push but no other contents Empty post with emoticons only, no contents 27

28 Key Online Attributes of Topic-Author Motivation#1 a Motivation#2 a Ability#1 b Ability#2 b Attributes total number of topics created by the Topic Author total number of empty/spam posts created by the Topic Author total number of posts responded by other Post Authors (including both empty posts and nonempty posts) Descriptions this shows the Topic Author is frequently engaging the community this shows the Topic Author is very eager to keep the topic at a higher rank a Motivation: In order to influence others, a Topic Author total number might of have non-empty/spam higher motivation in posts created by Topic trying to engage others in discussing the topics Author created by him/her. this shows the Topic Author is good at engaging conversation with other Post Authors this is an indication of the ability of Topic Author to attract others to discuss the topic b Ability: In order to influence other people, a Topic Author might have higher abilities than others to engage people in discussions of the topics created. 28

29 Results: Classification#1 Moderate-Inactive n=876, 28.8% of the sample s population No significant difference in motivation score with Inactive- Silent-Majority group Showed slightly better ability in engaging in topic discussion Inactive-Silent-Majority n=1552, 51.1% of the sample s population least motivated with the lowest ability creating an average of 1 to 2 topics during this 4- month period 29

30 Results: Classification#2 Active-Vocal-Minority n=155, 5.1% of the sample s population received an average of 1,853 posts/topic replied an average of 162 non-empty posts/topic posted an average of 71 new topics continuously during the 4-month period on average they are creating one new topic every 2.5 days Moderate-active n=457, 15.0% of the sample s population ranked second best in terms of motivation and ability as topic authors n=278, 61% of this group has been active as topic authors 30 for 2 to 3 months duration

32 We define as follows: Social Influence Index Social Influence Index = Log(z-score) Receives n (=r+e) posts to the topic Topic-author X creates a topic on the discussion forum By definition, z-score = Non-empty post #1 Non-empty post #r Empty/spam post #1 Empty/spam post #e x μ σ n 2 where μ = n p = σ = n p 1 p = n/2 Assume a random Post Author who responses with non-empty posts of probability p=0.5 and with empty posts of probability 1 - p =0.5 Thus, z score = r n/2 n/2 = r e r + e 32

33 Results Inactive- Silent- Majority (n=1552) Moderate -Inactive (n=876) Moderate -Active (n=457) Active- Vocal- Minority (n=155) Social Influence Index Total number of flash-mob-like topics Total number of posts replied to flash-mob-like topics ,076 29, , , ,279 The highest Social Influence Index score The most number (1,076) of flash-mob-like-demonstrationtopics, which accounted for 57% of the total. The most number (334,279) of responses, which accounted for 53% of the total. 33

34 Sample Topic-Author from Active- Vocal-Minority group Created 183 topics in just one month (March 2015)! Responded 552 times in just one month (March 2015)! 34

35 Challenges of Social Media Mining 1. Data collection issue 2. Empirical evaluation issue 3. Noise issue 35

36 Conclusion Cyber attacks investigation Do we have the data? Do we know how to analyze the data? From online to offline We have lots of data! Big data!!! Do we know how to analyze them? 36

37 谢谢 Thank You