Cloud Computing Era. Trend Micro

Transcription

1 Cloud Computing Era Trend Micro

2 Three Major Trends to Chang the World Cloud Computing Big Data Mobile

3 什麼是雲端運算? 美國國家標準技術研究所 (NIST) 的定義 : Essential Characteristics Service Models Deployment Models 以服務 (as-a-service) 的商業模式, 透過 Internet 技術, 提供具有擴充性 (scalable) 和彈性 (elastic) 的 IT 相關功能給使用者

4 It s About the Ecosystem Structured, Semi-structured Cloud Computing Enterprise Data Warehouse SaaS PaaS IaaS Generate Big Data Lead Business Insights create Competition, Innovation, Productivity

5 What is Big Data?

6 What is the problem Getting the data to the processors becomes the bottleneck Quick calculation Typical disk data transfer rate: 75MB/sec Time taken to transfer 100GB of data to the processor: approx. 22 minutes!

7 The Era of Big Data Are You Ready Data for business commercial analysis 2011: multi-terabyte (TB) 2020: 35.2 ZB (1 ZB = 1 billion TB)

8 Who Needs It? Enterprise Database Hadoop When to use? Ad-hoc Reporting (<1sec) Multi-step Transactions Lots of Inserts/Updates/Deletes When to use? Affordable Storage/Compute Unstructured or Semi-structured Resilient Auto Scalability

9 Hadoop!

10 inspired by Apache Hadoop project inspired by Google's MapReduce and Google File System papers. Open sourced, flexible and available architecture for large scale computation and data processing on a network of commodity hardware Open Source Software + Hardware Commodity IT Costs Reduction

15 Word Count Example Key: offset Value: line Key: word Value: count Key: word Value: sum of count 0:The cat sat on the mat 22:The aardvark sat on the sofa

16 The Hadoop Ecosystems

17 The Ecosystem is the System Hadoop has become the kernel of the distributed operating system for Big Data No one uses the kernel alone A collection of projects at Apache

18 Relation Map Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)

19 Zookeeper Coordination Framework Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)

20 What is ZooKeeper A centralized service for maintaining Configuration information Providing distributed synchronization A set of tools to build distributed applications that can safely handle partial failures ZooKeeper was designed to store coordination data Status information Configuration Location information

21 Flume / Sqoop Data Integration Framework Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)

22 What s the problem for data collection Data collection is currently a priori and ad hoc A priori decide what you want to collect ahead of time Ad hoc each kind of data source goes through its own collection path

23 (and how can it help?) A distributed data collection service It efficiently collecting, aggregating, and moving large amounts of data Fault tolerant, many failover and recovery mechanism One-stop solution for data collection of all formats

26 Sqoop Easy, parallel database import/export What you want do? Insert data from RDBMS to HDFS Export data from HDFS back into RDBMS

28 Sqoop Examples $ sqoop import --connect jdbc:mysql://localhost/world -- username root --table City... $ hadoop fs -cat City/part-m ,Kabul,AFG,Kabol, ,Qandahar,AFG,Qandahar, ,He rat,afg,herat, ,mazar-e- Sharif,AFG,Balkh, ,Amsterdam,NLD,Noord-Holland, Cloudera, Inc. All Rights Reserved.

29 Pig / Hive Analytical Language Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)

30 Why Hive and Pig? Although MapReduce is very powerful, it can also be complex to master Many organizations have business or data analysts who are skilled at writing SQL queries, but not at writing Java code Many organizations have programmers who are skilled at writing code in scripting languages Hive and Pig are two projects which evolved separately to help such people analyze huge amounts of data via MapReduce Hive was initially developed at Facebook, Pig at Yahoo!

31 Hive Developed by What is Hive? An SQL-like interface to Hadoop Data Warehouse infrastructure that provides data summarization and ad hoc querying on top of Hadoop MapRuduce for execution HDFS for storage Hive Query Language Basic-SQL : Select, From, Join, Group-By Equi-Join, Muti-Table Insert, Multi-Group-By Batch query SELECT * FROM purchases WHERE price > 100 GROUP BY storeid

33 Pig Initiated by A high-level scripting language (Pig Latin) Process data one step at a time Simple to write MapReduce program Easy understand Easy debug A = load a.txt as (id, name, age,...) B = load b.txt as (id, address,...) C = JOIN A BY id, B BY id;store C into c.txt

35 Hive vs. Pig Hive Language HiveQL (SQL-like) Pig Latin, a scripting language Schema Table definitions that are stored in a metastore Programmait Access JDBC, ODBC Pig A schema is optionally defined at runtime PigServer

36 WordCount Example Input Hello World Bye World Hello Hadoop Goodbye Hadoop For the given sample input the map emits < Hello, 1> < World, 1> < Bye, 1> < World, 1> < Hello, 1> < Hadoop, 1> < Goodbye, 1> < Hadoop, 1> < Bye, 1> < Goodbye, 1> < Hadoop, 2> < Hello, 2> < World, 2> the reduce just sums up the values

37 WordCount Example In MapReduce public class WordCount { public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(longwritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.tostring(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasmoretokens()) { word.set(tokenizer.nexttoken()); context.write(word, one); } } } public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } context.write(key, new IntWritable(sum)); } } public static void main(string[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); job.setoutputkeyclass(text.class); job.setoutputvalueclass(intwritable.class); job.setmapperclass(map.class); job.setreducerclass(reduce.class); job.setinputformatclass(textinputformat.class); job.setoutputformatclass(textoutputformat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitforcompletion(true); }

38 WordCount Example By Pig A = LOAD 'wordcount/input' USING PigStorage as (token:chararray); B = GROUP A BY token; C = FOREACH B GENERATE group, COUNT(A) as count; DUMP C;

39 WordCount Example By Hive CREATE TABLE wordcount (token STRING); LOAD DATA LOCAL INPATH wordcount/input' OVERWRITE INTO TABLE wordcount; SELECT count(*) FROM wordcount GROUP BY token;

41 Hbase Column NoSQL DB Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)

42 Structured-data vs Raw-data

43 I Inspired by Coordinated by Zookeeper Low Latency Random Reads And Writes Distributed Key/Value Store Simple API PUT GET DELETE SCAN

44 Hbase Data Model Cells are versioned Table rows are sorted by row key Region a row range [start-key:end-key]

45 HBase Examples hbase> create 'mytable', 'mycf hbase> list hbase> put 'mytable', 'row1', 'mycf:col1', 'val1 hbase> put 'mytable', 'row1', 'mycf:col2', 'val2 hbase> put 'mytable', 'row2', 'mycf:col1', 'val3 hbase> scan 'mytable hbase> disable 'mytable hbase> drop 'mytable' 2011 Cloudera, Inc. All Rights Reserved.

46 Oozie Job Workflow & Scheduling Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)

47 What is? A Java Web Application Oozie is a workflow scheduler for Hadoop Crond for Hadoop Triggered Time Data Job 1 Job 3 Job 2 Job 4 Job 5

49 Mahout Data Mining Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)

50 What is Machine-learning tool Distributed and scalable machine learning algorithms on the Hadoop platform Building intelligent applications easier and faster

52 Hue developed by Hadoop User Experience Apache Open source project HUE is a web UI for Hadoop Platform for building custom applications with a nice UI library

53 Hue HUE comes with a suite of applications File Browser: Browse HDFS; change permissions and ownership; upload, download, view and edit files. Job Browser: View jobs, tasks, counters, logs, etc. Beeswax: Wizards to help create Hive tables, load data, run and manage Hive queries, and download results in Excel format.

54 Hue: File Browser UI

55 Hue: Beewax UI

56 Use case Example Predict what the user likes based on His/Her historical behavior Aggregate behavior of people similar to him

57 Conclusion Today, we introduced: Why Hadoop is needed The basic concepts of HDFS and MapReduce What sort of problems can be solved with Hadoop What other projects are included in the Hadoop ecosystem

58 Recap Hadoop Ecosystem Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)

59 趨勢科技雲端防毒 Case Study

60 Collaboration in the underground

61 網路威脅呈現爆炸性的成長各式各樣的變種病毒垃圾郵件不明的下載來源等等, 這些來自網路上的威脅, 躲過傳統安全防護系統的偵測, 一直持續呈現爆炸性的成長, 形成嚴重的資安威脅 New Unique Malware Discovered 1M unique Malwares every month

62

63 New Design Concept for Threat Intelligence CDN / xsp Human Intelligence Honeypot Web Crawler Trend Micro Mail Protection Trend Micro Web Protection 150M+ Worldwide Endpoints/Sensors Trend Micro Endpoint Protection

64 Challenges We Are Faced The Concept is Great but. 6TB of data and 15B lines of logs received daily by It becomes the Big Data Challenge!

65 Issues to Address Raw Data Information Threat Intelligence/Solution Volume: Infinite Time: No Delay Target: Keep Changing Threats

66 SPN Feedback HTTP POST SPAM CDN Log SPN High Level Architecture L4 Log Receiver Log Receiver L4 Log Post Processing Log Post Processing Log Post Processing SPN infrastructure Application MapReduce HBase Hadoop Distributed File System (HDFS) Feedback Information Reputation Service Adhoc-Query (Pig) Circus (Ambari) Message Bus Web Reputation Service File Reputation Service

67 Trend Micro Big Data process capacity 雲端防毒每日需要處理的資料量 85 億個 Web Reputation 查詢 30 億個 Reputation 查詢 70 億個 File Reputation 查詢處理 6 TB 從全世界收集到的 raw logs 來自 1.5 億台終端裝置的連線

68 Trend Micro: Web Reputation Services Technology Process Operation Trend Micro Products / Technology User Traffic Honeypot 8 billions/day CDN Cache High Throughput Web Service Hadoop Cluster Web Crawling Akamai Rating Server for Known Threats Unknown & Prefilter Page Download 4.8 billions/day 860 millions/day 40% filtered 82% filtered 15 Minutes Machine Learning Data Mining Threat Analysis 99.98% filtered 25,000 malicious URL /day Block malicious URL within 15 minutes once it goes online!

69 Big Data Cases

70 Google vs. Hadoop Ecosystem Chubby vs. Zookeeper MapReduce vs. MapReduce BigTable vs. HBase GFS vs. HDFS Ref: Google Cluster

71 Pioneer of Big Data Infrastructure Google

72 Hbase use - Messages HBase Use Facebook Messages Facebook Insights Self-service Hashout Operational Data Store More Analytics/Hashout apps Site Integrity Social Graph Search Indexing Realtime Hive Updates Cross-system Tracing and more

73 Flagship App:Facebook Messages Monthly data volume prior to launch Monthly data volume prior to launch 15B x 1,024 bytes = 14TB 120B x 100 bytes = 11TB

74 Facebook Messages Now ook Messages NOW tats Quick Stats 11B+ messages/day messages/day B+ data accesses ak: 1.5M ops/sec 5%Rd, 45% Wr 90B+ data accesses Peak:1.5M ops/sec ~55% Read, 45% Write 20PB+ of total data Grows 400TB/month + of total data ows 400TB/month Messages s Chats SMS

75 Facebook Messages:Requirements Very High Write Volume Previously, chat was not persisted to disk Ever-growing data sets(old data rarely gets accessed) Elasticity & automatic failover Strong consistency within a single data center Large scans/map-reduce support for migrations & schema conversions Bulk import data

76 Physical Multi-tenancy Real-time Ads Insights Real-time analytics for social plugins on top of Hbase Publishers get real-time distribution/engagement metrics: # of impressions, likes analytics by domain/url/demographics and time periods Uses HBase capabilities: Efficient counters (single-rpc increments) TTL for purging old data Needs massive write throughput & low latencies Billions of URLs Millions of counter increments/second Operational Data Store

77 Facebook Open Source Stack Memcached --> App Server Cache ZooKeeper --> Small Data Coordination Service HBase --> Database Storage Engine HDFS --> Distributed FileSystem Hadoop --> Asynchronous Map-Reduce Jobs

78

79 Questions?

80 Thank you!