Distributed Calculus with Hadoop MapReduce inside Orange Search Engine. mardi 3 juillet 12

Transcription

1 Distributed Calculus with Hadoop MapReduce inside Orange Search Engine

2 What is Big Data?

3 $ 5 billions (2012) to $ 50 billions (by 2017) Forbes

4 «Big Data is the new definitive source of competitive advantage across all industries» Jeff Kelly

5 Product success «The days are over when you build a product once and it just works. You have to take ideas, test them, iterate them, use data and analytics to understand what works and what doesn t in order to be successful» 2012

6 Big Data

7 What about Hadoop?

8 Beliefs «We believe that by 2015, more than half the world s data will be processed by Apache Hadoop» 2012

9 Actors

10 Hadoop eco-system

11 Context at Orange (more than 2 years ago)

12 Orange Search Engine

13 Search Engine Architecture Internet Collecte du WEB 24/24 (Crawl) Indexation Pré-calcul de score Infrastructure de Recherche PageRank 5 milliards de documents 750 millions de documents Français

14 Main issue PageRank calculus on billions nodes and 10s billions edges regularly failed! (hardware...) 4 to 8 weeks calculus unscalable failure rate aroud 80% One person full time to supervise

15 Answer

16 PageRank portable to Hadoop / MapReduce? Simple programing model: Map(in_k,in_v) => list(out_k,intermed_v) Reduce(out_k,intermed_v) => list(out_v) Scalable Batch Processing YES!

17 Hadoop Axioms System shall manage and heal himself Performance shall scale linearly Compute shall move to data Modular and extensible

18 Our install?

19 ZooKeeper Our install Oozie HIVE PIG MapReduce HDFS Mahout Khiops

20 HDFS?

21 Hadoop - HDFS Client Master Nœud données Nœud données Nœud données Nœud données Nœud données Read/Write Read Bloc ops replication COTS - replication - big blocks maximize throughput - Metadata in RAM

22 Map Reduce?

23 MapReduce cat «your map» sort -u «your reduce» Programming paradigm

24 MapReduce cat «your map» sort -u «your reduce» FrameWork

25 MapReduce cat «your map» sort -u «your reduce» your Job

26 MapReduce Insides

27 Interfaces Java API Pipes Streaming (python, perl, C/C++,...)

28 PIG High level data analysis script language extensible via UDF Structure of a Pig script load filter foreach group by join your functions order store

29 HIVE High level SQL-like query and analysis language extensible via UDF Structure of a Hive script create table load data select... from... insert group by join

30 Application domains?

31 Projects Scoring User profiling Log analysis and statistics... and many others to come

32 ROI?

33 ROI Lines Of Code 10X gain Development Time 2X gain IT cost 4X gain less bug, automatic, scalable...

34 Perfect World? YES Run cost Development cost Scalable Stable Heterogenity NO SPOF (almost solved) Fastidious debugging Localy non optimum mono-site

35 Thank you Olivier Varene -

36 thanks to Hadoop - Apache ( Khiops - Orange ( Shauwn Connoly - HortonWorks ( Forbes - article ( Living Social (sentence) Terradata (Volumetry Graph) (Words Cloud)