Age of Big data. Presented by: Mohammad Iqbal BCM -2014

Transcription

1 Age of Presented by: Mohammad Iqbal BCM -2014

2 Agenda Big? Big evolution from

3 Big? Name Symbol Value Kilobyte KB 10^3 BIG DATA Megabyte MB 10^6 Gigabyte GB 10^9 Terabyte TB 10^12 Petabyte PB 10^15 So large data that it becomes difficult to process it using the traditional system Exabyte EB 10^18 Zettabyte ZB 10^21 Yottabyte YB 10^24 Big? Big

4 Difficult to process by Traditional System Unable to send Unable to View 100 MB document Unable to Edit 100 GB document Depends on capability of system 100 TB document Big? Big

5 Organization/Context Specific 500 TB Text,Audio,Video data per day Big Date NOT a Big data Depends on capabilities of the organization Company A Company B Big? Big

6 Areas of Challenges Capture search Curation Sharing Storage Transfer Anlaysis Visualization Big? Big

7 Big Big Large & growing files At High speed In various Format V^3 comes at high speed result in large file This files comes in various formats VELOCITY VOLUME VARIETY Big? Big

8 Structured / Unstructured Challenge /Opportunity Mostly wasted Used in decision making Unstructured 90% Structured 10% To analyze & extract meaningful information Big? Big

9 Users Applications Systems Large & growing files ( files) Sensors Big? Big

10 Generation point Examples Mobile devices Machine Sensors Microphones cameras Readers/Scanners Social Media Science facilities Software/program Big? Big

11 Sample Events generating Every day, we create 2.5 Exabytes of data i.e 2.5 billion GB, so much that 90% of the data in the world today has been created in the last few years alone. CERN Atomic facility generates 40 TB data per second. Twitter generates 12 TB of data every day. Airbus A380 generates 10 TB every 30 minutes of flight. About 650TB generated in one flight. In 2009 total data in world was estimated to be 1 ZB. By 2020 estimated to be 35 ZB. (Source :IBM.com) Big? Big

12 Collect Analyze Understand Big? Big

13 Applications Companies gaining edge by collecting,analyzing and understanding information. Government forecasting events and taking proactive actions. Big? Big

14 Not able to handle Big data Created to handle big data Traditional Systems (e.g RDBMS,SQL) tool (e.g NoSQL) Time Big? Big

15 Traditional Enterprise Approach Only So much data could be processed Processing Limit Powerful Computer Big? Big

16 Modern s approach Computation Combined result Computation Computation Computation Big? Big

17 s s Hive Map Reduce HBase Mahout File System HDFS Pig Oozie Projects Source :hortonworks/hadoop/hdfs/.com/ Flume Scoop Big? Big

18 MASTER Task tracker Job Tracker DATA Application Node Name Node Slaves Task tracker Task tracker Task tracker Task tracker Node Node Node Node

19 MASTER can be taken directly Task tracker Job Tracker DATA Application Node Name Node Know where data residing Slaves Task tracker Task tracker Task tracker Task tracker Node Node Node Node

20 HDFS vs GFS Similarity with file system (GFS) MapReduce Back in 1990 search engine supported by: Excite Altavista Lycos Infoseek Big? Big

21 Victory 1995 Excite 2000 Altavista Lycos Big? Big

22 evolution from GFS paper released by released paper on MapReduce created by Doug & Cafarella at Yahoo! (Nutch search engine) Yahoo donated the project to Apache Source : & Nutch white papers Big? Big

23 is here!! Big? Big

24 scientists with just two years' experience can earn between $200,000 and $300,000 a year (wall street journel). Anyone with "data science" in his or her job title on a LinkedIn page is going to get "100 recruiter s a day,.(wall street journel). is a super hot up-and-coming "big data" technology. (Business insider.com). Many other data scientists, especially at data-driven companies such as, Amazon, Microsoft, Walmart, ebay, LinkedIn, and Twitter, have added to and looking for developing the tool kit. (Harvard business review). "People are slapping buzzwords as on résumés and looking to get 50 or 100 percent more, and they're getting it," said Scott Gnau, president of Teradata Lab. Big? Big

25 References Dean & Sanjay (2004)> MapReduce: Simplied Processing on Large Clusters.google.com Dogh Cutting Nutch(2005): A Flexible and Scalable Open-Source Web Search Engine.yahoo.com Sanjay & Howard (2003): The File System, google.com [Accessed date 27 th nov 2014] salary ?op=1[accessed date 27 th nov 2014] Big 's High-Priests of Algorithms, date 27 th nov 2014]

26 Thank you for your attention Q/A