Обработка больших данных: Map Reduce (Python) + Hadoop (Streaming) Максим Щербаков ВолгГТУ 8/10/2014

Transcription

1 Обработка больших данных: Map Reduce (Python) + Hadoop (Streaming) Максим Щербаков ВолгГТУ 8/10/2014 1

2 Содержание Бигдайта: распределенные вычисления и тренды MapReduce: концепция и примеры реализации MapReduce over Hadoop (Python + Streaming) 2

3 In one second on the Internet there are

4 Big Data Проблемы: 5V Volume Velocity Variety Veracity Алгоритмы real time data stream processing batch processing (map reduce) Технологии distributed computation: Hadoop tech. stack 4

5 Big Data: Gartner's Hype Cycles for Emerging Technologies, / 5

6 6

7 Distributed Architecture 7

8 Map Reduce Paradigm (input) csv; text (separated); v1 = input map(k1; v1) -> list(k2;v2) shuffle/sort * reduce(k2; list(v2)) -> list(k3;v3) output = k3;v3 8

9 Poker Game 9

10 Пример 10

11 Hadoop 11

12 MapReduce over Hadoop. v1 1. Job clients submit job to JobTracker copy binaries w/ MR 2. JobTracker talks to NameNode 3. Job Tracker creates the execution plan 4. Job Tracker submits work to Task Tracker 5. TaskTracker report progress via heartbeats 6. JobTracker manages phases 7. JobTracker updates status 12

13 YARN MapReduce 2.0 (MRv2) idea of MRv2's YARN architecture: is to split up the JobTracker a global ResourceManager (RM) per-application ApplicationMasters (AM). ResourceManager (RM) and per-node NodeManagers (NM), form the datacomputation framework: ResourceManager as JobTracker NodeManagers run on slave nodes instead of TaskTracker. 13

14 Hadoop Distributions Apache Hadoop Cloudera udera/en/home.html Hortonworks Microsoft HDInsight 14

15 Hadoop (CDH4 with YARN on a Single Linux Node in Pseudo-distributed mode ) Download RPM (Red Hat/CentOS/Oracle 6 (64-bit)) Download RPM. Install the RPM. sudo yum --nogpgcheck localinstall cloudera-cdh-4-0.x86_64.rpm Install CDH4 sudo yum install hadoop-conf-pseudo on/cdh4/latest/cdh4-quick-start/cdh4qs_topic_3_3.html 15

16 Starting Hadoop Step 1: Форматирование NameNode $ sudo -u hdfs hdfs namenode -format Step 2: Запуск HDFS $ for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done Step 3: Создание /tmp, Staging and Log Directories $ sudo -u hdfs hadoop fs -rm -r /tmp $ sudo -u hdfs hadoop fs -mkdir -p /tmp/hadoopyarn/staging/history/done_intermediate $ sudo -u hdfs hadoop fs -chown -R mapred:mapred /tmp/hadoopyarn/staging $ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp $ sudo -u hdfs hadoop fs -mkdir -p /var/log/hadoop-yarn $ sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn 16

17 Starting Hadoop (cont.) Step 4: Верификация HDFS File Structure: $ sudo -u hdfs hadoop fs -ls -R / Step 5: Запуск YARN $ sudo service hadoop-yarn-resourcemanager start $ sudo service hadoop-yarn-nodemanager start $ sudo service hadoop-mapreduce-historyserver start Step 6: Create User Directories $ sudo -u hdfs hadoop fs -mkdir /user/$user $ sudo -u hdfs hadoop fs -chown $USER /user/$user 17

18 Hints don't forget to create user hdfs + set the password; be careful with Data Node name (hdnode <- ok) be careful with paths in HDFS 18

19 Running an example application with YARN Create a home directory on HDFS for the user who will be running the job $ sudo -u hdfs hadoop fs -mkdir /user/joe $ sudo -u hdfs hadoop fs -chown user/joe /user/joe 19

20 Make a directory in HDFS called input and copy some XML files $ hadoop fs -mkdir input $ hadoop fs -put /etc/hadoop/conf/*.xml input $ hadoop fs -ls input Found 3 items: -rw-r--r-- 1 joe supergroup :21 input/core-site.xml -rw-r--r-- 1 joe supergroup :21 input/hdfs-site.xml -rw-r--r-- 1 joe supergroup :21 input/mapred-site.xml 20

21 Set HADOOP_MAPRED_HOME for user $ export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce 21

22 Run an example Hadoop job $ hadoop jar /usr/lib/hadoop-mapreduce/hadoopmapreduce-examples.jar grep input output23 'dfs[a-z.]+' $ hadoop fs -ls Found 2 items drwxr-xr-x - joe supergroup :36 /user/maximshcherbakov/input drwxr-xr-x - joe supergroup :38 /user/joe/output23 $ hadoop fs -ls output23 Found 2 items drwxr-xr-x - joe supergroup :33 /user/joe/output23/_success -rw-r--r-- 1 joe supergroup :33 /user/joe/output23/part-r

23 Read the results in the output file. $ hadoop fs -cat output23/part-r head 1 dfs.safemode.min.datanodes 1 dfs.safemode.extension 1 dfs.replication 1 dfs.permissions.enabled 1 dfs.namenode.name.dir 1 dfs.namenode.checkpoint.dir 1 dfs.datanode.data.dir 23

24 Java Example Given C[,]SV files with temperature for different location line[0]: timestamp; line[1]: date; line[10]: temperature 24

25 AvgTemperatureMapper 25

26 AvgTemperatureReducer 26

27 AvgTemperature 27

28 28

29 WordCount Example. Mapper 29

30 Reducer 30

31 Launch under Streaming hadoop jar /usr/lib/hadoop-mapreduce/hadoopstreaming-2*.jar -file /tmp/python/mapper.py -mapper "python /tmp/python/mapper.py" -file /tmp/python/reducer.py -reducer "python /tmp/python/reducer.py" -input data/small/* -output data/small/output2 31

32 Более интеллектуальное: идентификация выбросов 32

33 Mapper 33

34 Reducer version#1 34

35 Reducer version#2 35

36 Streaming 36

37 Hello, mrjob Write multi-step MapReduce jobs in pure Python Test on your local machine Run on a Hadoop cluster Run in the cloud using Amazon Elastic MapReduce (EMR) 37

38 Hadoop + BI Browser Data Sources Layer Devices (Sensors) External Web Services Actuators Mobile Application Web Application & Services (UI Layer) Web Application View Web Application Controllers & Models Data Collector Engine Web Services Business Layer Real Time Data Stream Processing Engine Supervised Control Engine Decision Support Engine Intelligent Data Processing Engine Data Layer Data Base Access Abstract Engine Real Data Warehouse Metadata & DSS Results Database 38

39 Demo

40 Спасибо