Cloud Computing Era Trend Micro
Three Major Trends to Chang the World Cloud Computing Big Data Mobile
什 麼 是 雲 端 運 算? 美 國 國 家 標 準 技 術 研 究 所 (NIST) 的 定 義 : Essential Characteristics Service Models Deployment Models 以 服 務 (as-a-service) 的 商 業 模 式, 透 過 Internet 技 術, 提 供 具 有 擴 充 性 (scalable) 和 彈 性 (elastic) 的 IT 相 關 功 能 給 使 用 者
It s About the Ecosystem Structured, Semi-structured Cloud Computing Enterprise Data Warehouse SaaS PaaS IaaS Generate Big Data Lead Business Insights create Competition, Innovation, Productivity
What is Big Data?
What is the problem Getting the data to the processors becomes the bottleneck Quick calculation Typical disk data transfer rate: 75MB/sec Time taken to transfer 100GB of data to the processor: approx. 22 minutes!
The Era of Big Data Are You Ready Data for business commercial analysis 2011: multi-terabyte (TB) 2020: 35.2 ZB (1 ZB = 1 billion TB)
Who Needs It? Enterprise Database Hadoop When to use? Ad-hoc Reporting (<1sec) Multi-step Transactions Lots of Inserts/Updates/Deletes When to use? Affordable Storage/Compute Unstructured or Semi-structured Resilient Auto Scalability
Hadoop!
inspired by Apache Hadoop project inspired by Google's MapReduce and Google File System papers. Open sourced, flexible and available architecture for large scale computation and data processing on a network of commodity hardware Open Source Software + Hardware Commodity IT Costs Reduction
Hadoop Core MapReduce HDFS 2011 Cloudera, Inc. All Rights Reserved.
HDFS Hadoop Distributed File System Redundancy Fault Tolerant Scalable Self Healing Write Once, Read Many Times Java API Command Line Tool 2011 Cloudera, Inc. All Rights Reserved.
MapReduce Two Phases of Functional Programming Redundancy Fault Tolerant Scalable Self Healing Java API 13 2011 Cloudera, Inc. All Rights Reserved.
Hadoop Core Java Java MapReduce HDFS Java Java 14 2011 Cloudera, Inc. All Rights Reserved.
Word Count Example Key: offset Value: line Key: word Value: count Key: word Value: sum of count 0:The cat sat on the mat 22:The aardvark sat on the sofa
The Hadoop Ecosystems
The Ecosystem is the System Hadoop has become the kernel of the distributed operating system for Big Data No one uses the kernel alone A collection of projects at Apache
Relation Map Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)
Zookeeper Coordination Framework Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)
What is ZooKeeper A centralized service for maintaining Configuration information Providing distributed synchronization A set of tools to build distributed applications that can safely handle partial failures ZooKeeper was designed to store coordination data Status information Configuration Location information
Flume / Sqoop Data Integration Framework Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)
What s the problem for data collection Data collection is currently a priori and ad hoc A priori decide what you want to collect ahead of time Ad hoc each kind of data source goes through its own collection path
(and how can it help?) A distributed data collection service It efficiently collecting, aggregating, and moving large amounts of data Fault tolerant, many failover and recovery mechanism One-stop solution for data collection of all formats
Flume Architecture Log Flume Node... Log Flume Node HDFS 2011 Cloudera, Inc. All Rights Reserved.
Flume Sources and Sinks Local Files HDFS Stdin, Stdout Twitter IRC IMAP 2011 Cloudera, Inc. All Rights Reserved.
Sqoop Easy, parallel database import/export What you want do? Insert data from RDBMS to HDFS Export data from HDFS back into RDBMS
Sqoop HDFS Sqoop RDBMS 28 2011 Cloudera, Inc. All Rights Reserved.
Sqoop Examples $ sqoop import --connect jdbc:mysql://localhost/world -- username root --table City... $ hadoop fs -cat City/part-m-00000 1,Kabul,AFG,Kabol,17800002,Qandahar,AFG,Qandahar,2375003,He rat,afg,herat,1868004,mazar-e- Sharif,AFG,Balkh,1278005,Amsterdam,NLD,Noord-Holland,731200... 29 2011 Cloudera, Inc. All Rights Reserved.
Pig / Hive Analytical Language Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)
Why Hive and Pig? Although MapReduce is very powerful, it can also be complex to master Many organizations have business or data analysts who are skilled at writing SQL queries, but not at writing Java code Many organizations have programmers who are skilled at writing code in scripting languages Hive and Pig are two projects which evolved separately to help such people analyze huge amounts of data via MapReduce Hive was initially developed at Facebook, Pig at Yahoo!
Hive Developed by What is Hive? An SQL-like interface to Hadoop Data Warehouse infrastructure that provides data summarization and ad hoc querying on top of Hadoop MapRuduce for execution HDFS for storage Hive Query Language Basic-SQL : Select, From, Join, Group-By Equi-Join, Muti-Table Insert, Multi-Group-By Batch query SELECT * FROM purchases WHERE price > 100 GROUP BY storeid
Hive SQL Hive MapReduce 33 2011 Cloudera, Inc. All Rights Reserved.
Pig Initiated by A high-level scripting language (Pig Latin) Process data one step at a time Simple to write MapReduce program Easy understand Easy debug A = load a.txt as (id, name, age,...) B = load b.txt as (id, address,...) C = JOIN A BY id, B BY id;store C into c.txt
Pig Script Pig MapReduce 2011 Cloudera, Inc. All Rights Reserved.
Hive vs. Pig Hive Language HiveQL (SQL-like) Pig Latin, a scripting language Schema Table definitions that are stored in a metastore Programmait Access JDBC, ODBC Pig A schema is optionally defined at runtime PigServer
WordCount Example Input Hello World Bye World Hello Hadoop Goodbye Hadoop For the given sample input the map emits < Hello, 1> < World, 1> < Bye, 1> < World, 1> < Hello, 1> < Hadoop, 1> < Goodbye, 1> < Hadoop, 1> < Bye, 1> < Goodbye, 1> < Hadoop, 2> < Hello, 2> < World, 2> the reduce just sums up the values
WordCount Example In MapReduce public class WordCount { public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(longwritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.tostring(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasmoretokens()) { word.set(tokenizer.nexttoken()); context.write(word, one); } } } public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } context.write(key, new IntWritable(sum)); } } public static void main(string[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); job.setoutputkeyclass(text.class); job.setoutputvalueclass(intwritable.class); job.setmapperclass(map.class); job.setreducerclass(reduce.class); job.setinputformatclass(textinputformat.class); job.setoutputformatclass(textoutputformat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitforcompletion(true); }
WordCount Example By Pig A = LOAD 'wordcount/input' USING PigStorage as (token:chararray); B = GROUP A BY token; C = FOREACH B GENERATE group, COUNT(A) as count; DUMP C;
WordCount Example By Hive CREATE TABLE wordcount (token STRING); LOAD DATA LOCAL INPATH wordcount/input' OVERWRITE INTO TABLE wordcount; SELECT count(*) FROM wordcount GROUP BY token;
The Story So Far SQL Hive Pig Script Java Java MapReduce HDFS Sqoop Flume SQL RDBMS FS Posix 4 1 2011 Cloudera, Inc. All Rights Reserved.
Hbase Column NoSQL DB Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)
Structured-data vs Raw-data
I Inspired by Coordinated by Zookeeper Low Latency Random Reads And Writes Distributed Key/Value Store Simple API PUT GET DELETE SCAN
Hbase Data Model Cells are versioned Table rows are sorted by row key Region a row range [start-key:end-key]
HBase Examples hbase> create 'mytable', 'mycf hbase> list hbase> put 'mytable', 'row1', 'mycf:col1', 'val1 hbase> put 'mytable', 'row1', 'mycf:col2', 'val2 hbase> put 'mytable', 'row2', 'mycf:col1', 'val3 hbase> scan 'mytable hbase> disable 'mytable hbase> drop 'mytable' 2011 Cloudera, Inc. All Rights Reserved.
Oozie Job Workflow & Scheduling Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)
What is? A Java Web Application Oozie is a workflow scheduler for Hadoop Crond for Hadoop Triggered Time Data Job 1 Job 3 Job 2 Job 4 Job 5
Oozie Features Component Independent MapReduce Hive Pig SqoopStreaming 2011 Cloudera, Inc. All Rights Reserved.
Mahout Data Mining Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)
What is Machine-learning tool Distributed and scalable machine learning algorithms on the Hadoop platform Building intelligent applications easier and faster
Mahout Use Cases Yahoo: Spam Detection Foursquare: Recommendations SpeedDate.com: Recommendations Adobe: User Targetting Amazon: Personalization Platform 2011 Cloudera, Inc. All Rights Reserved.
Hue developed by Hadoop User Experience Apache Open source project HUE is a web UI for Hadoop Platform for building custom applications with a nice UI library
Hue HUE comes with a suite of applications File Browser: Browse HDFS; change permissions and ownership; upload, download, view and edit files. Job Browser: View jobs, tasks, counters, logs, etc. Beeswax: Wizards to help create Hive tables, load data, run and manage Hive queries, and download results in Excel format.
Hue: File Browser UI
Hue: Beewax UI
Use case Example Predict what the user likes based on His/Her historical behavior Aggregate behavior of people similar to him
Conclusion Today, we introduced: Why Hadoop is needed The basic concepts of HDFS and MapReduce What sort of problems can be solved with Hadoop What other projects are included in the Hadoop ecosystem
Recap Hadoop Ecosystem Hue (Web Console) Mahout (Data Mining) Oozie (Job Workflow & Scheduling) Zookeeper (Coordination) Sqoop/Flume (Data integration) MapReduce Runtime (Dist. Programming Framework) Pig/Hive (Analytical Language) Hbase (Column NoSQL DB) Hadoop Distributed File System (HDFS)
趨 勢 科 技 雲 端 防 毒 Case Study
Collaboration in the underground
網 路 威 脅 呈 現 爆 炸 性 的 成 長 各 式 各 樣 的 變 種 病 毒 垃 圾 郵 件 不 明 的 下 載 來 源 等 等, 這 些 來 自 網 路 上 的 威 脅, 躲 過 傳 統 安 全 防 護 系 統 的 偵 測, 一 直 持 續 呈 現 爆 炸 性 的 成 長, 形 成 嚴 重 的 資 安 威 脅 New Unique Malware Discovered 1M unique Malwares every month
New Design Concept for Threat Intelligence CDN / xsp Human Intelligence Honeypot Web Crawler Trend Micro Mail Protection Trend Micro Web Protection 150M+ Worldwide Endpoints/Sensors Trend Micro Endpoint Protection
Challenges We Are Faced The Concept is Great but. 6TB of data and 15B lines of logs received daily by It becomes the Big Data Challenge!
Issues to Address Raw Data Information Threat Intelligence/Solution Volume: Infinite Time: No Delay Target: Keep Changing Threats
SPN Feedback HTTP POST SPAM CDN Log SPN High Level Architecture L4 Log Receiver Log Receiver L4 Log Post Processing Log Post Processing Log Post Processing SPN infrastructure Application MapReduce HBase Hadoop Distributed File System (HDFS) Feedback Information Email Reputation Service Adhoc-Query (Pig) Circus (Ambari) Message Bus Web Reputation Service File Reputation Service
Trend Micro Big Data process capacity 雲 端 防 毒 每 日 需 要 處 理 的 資 料 量 85 億 個 Web Reputation 查 詢 30 億 個 Email Reputation 查 詢 70 億 個 File Reputation 查 詢 處 理 6 TB 從 全 世 界 收 集 到 的 raw logs 來 自 1.5 億 台 終 端 裝 置 的 連 線
Trend Micro: Web Reputation Services Technology Process Operation Trend Micro Products / Technology User Traffic Honeypot 8 billions/day CDN Cache High Throughput Web Service Hadoop Cluster Web Crawling Akamai Rating Server for Known Threats Unknown & Prefilter Page Download 4.8 billions/day 860 millions/day 40% filtered 82% filtered 15 Minutes Machine Learning Data Mining Threat Analysis 99.98% filtered 25,000 malicious URL /day Block malicious URL within 15 minutes once it goes online!
Big Data Cases
Google vs. Hadoop Ecosystem Chubby vs. Zookeeper MapReduce vs. MapReduce BigTable vs. HBase GFS vs. HDFS Ref: Google Cluster
Pioneer of Big Data Infrastructure Google
Hbase use Case@Facebook - Messages HBase Use Cases @ Facebook Messages Facebook Insights Self-service Hashout Operational Data Store More Analytics/Hashout apps Site Integrity 2010 2011 2012 2013 Social Graph Search Indexing Realtime Hive Updates Cross-system Tracing and more
Flagship App:Facebook Messages Monthly data volume prior to launch Monthly data volume prior to launch 15B x 1,024 bytes = 14TB 120B x 100 bytes = 11TB
Facebook Messages Now ook Messages NOW tats Quick Stats 11B+ messages/day messages/day B+ data accesses ak: 1.5M ops/sec 5%Rd, 45% Wr 90B+ data accesses Peak:1.5M ops/sec ~55% Read, 45% Write 20PB+ of total data Grows 400TB/month + of total data ows 400TB/month Messages Emails Chats SMS
Facebook Messages:Requirements Very High Write Volume Previously, chat was not persisted to disk Ever-growing data sets(old data rarely gets accessed) Elasticity & automatic failover Strong consistency within a single data center Large scans/map-reduce support for migrations & schema conversions Bulk import data
Physical Multi-tenancy Real-time Ads Insights Real-time analytics for social plugins on top of Hbase Publishers get real-time distribution/engagement metrics: # of impressions, likes analytics by domain/url/demographics and time periods Uses HBase capabilities: Efficient counters (single-rpc increments) TTL for purging old data Needs massive write throughput & low latencies Billions of URLs Millions of counter increments/second Operational Data Store
Facebook Open Source Stack Memcached --> App Server Cache ZooKeeper --> Small Data Coordination Service HBase --> Database Storage Engine HDFS --> Distributed FileSystem Hadoop --> Asynchronous Map-Reduce Jobs
Questions?
Thank you!