Install Hadoop on Ubuntu and run as standalone

Transcription

1 Welcome, this document is a record of my installation of Hadoop for study purpose. Version Version Date Content and Change Dec Initialize study Hadoop Install basic environment, run first word count program May Expend Hadoop to master+slave (1 master + 2 slave nodes) on RHEL Run word count on bigger file, see improvement Study Hadoop based projects Hive, Pig, Hbase Install Hbase on Hadoop as full distribute mode Install Hadoop on Ubuntu and run as standalone Hadoop version: OS: Ubuntu 32 bit hduser@ubuntu:~$ cat /etc/issue Ubuntu \n \l Host: VMWARE workstation on Windows 8 Please be advice this is a draft, and just fit my own circumstance. References: Developer list!! Following things need to be verified:

2 1. Job tracker interface can t show Fixed 2. Where is job tracker? It seems they changed job tracker to YARN (Next generation Map-reduce) Use local:8088 instead of localhost features.html Web Interfaces Once the Hadoop cluster is up and running check the web-ui of the components as described below: NameNode ResourceManager MapReduce JobHistory Server Apache Hadoop NextGen MapReduce (YARN) MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2) or YARN. So when check with JPS, there is no longer has jobtracker Web UI In MR1, the JobTracker web UI served detailed information about the state of the cluster and the jobs currently and recently running on it. It also contained the job history page, which served information from disk about older jobs.

3 The MR2 web UI provides the same information structured in the same way, but has been revamped with a new look and feel. The ResourceManager UI, which includes information about running applications and the state of the cluster, is now located by default at:8088. The job history UI is now located by default at: Jobs can be searched and viewed there just as they could in MR1. Because the ResourceManager is meant to be agnostic to many of the concepts in MapReduce, it cannot host job information directly. Instead, it proxies to a web UI that can. If the job is running, this is the relevant MapReduce Application Master; if it has completed, this is the JobHistoryServer. In this sense, the user experience is similar to that of MR1, but the information is coming from different places. Some conceptions What Is Apache Hadoop? The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing. The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver highavailability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. The project includes these modules: Hadoop Common: The common utilities that support the other Hadoop modules. Hadoop Distributed File System (HDFS ): A distributed file system that provides high-throughput access to application data. Hadoop YARN: A framework for job scheduling and cluster resource management. Hadoop MapReduce: A YARN-based system for parallel processing of large data sets. Other Hadoop-related projects at Apache include: Ambari : A web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters which includes support for Hadoop HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides a dashboard for viewing cluster health such as heatmaps and ability to view MapReduce, Pig and Hive applications visually alongwith features to diagnose their performance characteristics in a user-friendly manner. Avro : A data serialization system. Cassandra : A scalable multi-master database with no single points of failure. Chukwa : A data collection system for managing large distributed systems. HBase : A scalable, distributed database that supports structured data storage for large tables. Hive : A data warehouse infrastructure that provides data summarization and ad hoc querying. Mahout : A Scalable machine learning and data mining library. Pig : A high-level data-flow language and execution framework for parallel computation.

4 Spark : A fast and general compute engine for Hadoop data. Spark provides a simple and expressive programming model that supports a wide range of applications, including ETL, machine learning, stream processing, and graph computation. ZooKeeper : A high-performance coordination service for distributed applications. Hadoop follows the idea of job map and reduce. It has a DFS file system to support distribution Important ports 1.Job Tracker :50030 (no longer in 2.20?) 2.HDFS : HDFS communication: MapReducecommunication:9001 Management 1. HDFS web interface 2. MapReduce interface Install Ubuntu You may like to install it by a flash disk (on a new tower) Install required packages For example: JRE/JDK $ sudo apt-get update $ sudo apt-get install sun-java6-jdk Add user and user group $ sudo addgroup hadoop

5 $ sudo adduser --ingroup hadoop hduser This will add the user hduser and the group hadoop to your local machine. Configuring SSH su - hduser hduser@ubuntu:~$ ssh-keygen -t rsa -P "" Generating public/private rsa key pair. Enter file in which to save the key (/home/hduser/.ssh/id_rsa): Created directory '/home/hduser/.ssh'. Your identification has been saved in /home/hduser/.ssh/id_rsa. Your public key has been saved in /home/hduser/.ssh/id_rsa.pub. The key fingerprint is: 9b:82:ea:58:b4:e0:35:d7:ff:19:66:a6:ef:ae:0e:d2 hduser@ubuntu The key's randomart image is: [...snipp...] hduser@ubuntu:~$ If success, you may ssh to your host without input credentials Disable IPV6 (not proved whether necessary) # disable ipv6 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 You have to reboot your machine in order to make the changes take effect. You can check whether IPv6 is enabled on your machine with the following command: $ cat /proc/sys/net/ipv6/conf/all/disable_ipv6

6 Install Hadoop files $ cd /usr/local $ sudo tar xzf hadoop tar.gz $ sudo mv hadoop hadoop $ sudo chown -R hduser:hadoop hadoop You may also think to make you directory where to place DFS data file, in my case it is /hadoopfs. If necessary, grant owner to hduser:hadoop Update environment variables vim /home/hduser/.rcbash export HADOOP_HOME=/usr/local/hadoop unalias fs &> /dev/null alias fs="hadoop fs" unalias hls &> /dev/null alias hls="fs -ls" lzohead () { } hadoop fs -cat $1 lzop -dc head less export PATH=$PATH:$HADOOP_HOME/bin export JAVA_HOME=/usr/lib/jvm/default-java Configure site Modify core-site.xml

7 pwd /usr/local/hadoop/etc/hadoop Add following lines:?xml version="1.0" encoding="utf-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <name>hadoop.tmp.dir</name> <value>/hadoopfs/tmp</value> <description>a base for other temporary directories.</description> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> <description>the name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.scheme.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description> <final>true</final> </configuration> Configure DHS Modify hdsf-site.xml. This file indicates information to DFS duser@ubuntu:/usr/local/hadoop/etc/hadoop$ cat hdfs-site.xml

8 <?xml version="1.0" encoding="utf-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <name>dfs.replication</name> <value>1</value> <description>default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time. </description> <name>dfs.permissions</name>

9 <value>false</value> <name>dfs.namenode.name.dir</name> <value>file:/hadoopfs/dfs/name</value> <final>true</final> <name>dfs.datanode.data.dir</name> <value>file:/hadoopfs/dfs/data</value> <final>true</final> </configuration> Configure Map-reduce cat mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

10 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <name>mapreduce.framework.name</name> <value>yarn</value> <name>mapreduce.jobtracker.system.dir</name> <value>file:/hadoopfs/dfs/system</value> <final>true</final> <name>mapreduce.jobtracker.address</name> <value>localhost:9001</value> <description>the host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task. </description>

11 <name>mapreduce.jobtracker.http.address</name> <value>localhost:50030</value> <name>mapreduce.cluster.local.dir</name> <value>file:/hadoopfs/dfs/local</value> <final>true</final> <name>mapreduce.cluster.temp.dir</name> <value>file:/hadoopfs/dfs/tmp</value> <description>no description</description> <final>true</final> </configuration> Yarn-site.xml <?xml version="1.0"?> <configuration> <!-- Site specific YARN configuration properties -->

12 <name>yarn.resourcemanager.hostname</name> <value>localhost</value> <description>the host is the hostname of the ResourceManager and the port is the port on which the clients can talk to the Resource Manager. </description> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> <value>org.apache.hadoop.mapred.shufflehandler</value> <description>shuffle service that needs to be set for Map Reduce to run </description> <name>yarn.nodemanager.aux-services</name> <value>mapreduce.shuffle</value> </configuration> Initialize DFS $ hdfs namenode -format In some cases if there is some problem with node and namenode DFS, you need to re-format the DFS $ rm -rf /hadoopfs/dfs/* $ rm -rf /hadoopfs/tmp/* Then re-format namenode DFS. Start and stop the server(s) Should be done one by one namenode, datanode and Yarn(resource manager), but you can use startall.sh and stop-all.sh to enable one time start/stop /usr/local/hadoop/sbin

13 $ hadoop-daemon.sh start namenode $ hadoop-daemon.sh start datanode Monitor the node(s) General nodes information can be checked by web browser at port (by default) You can also use jps to see following instances are running hduser@ubuntu:/usr/local/hadoop/sbin$ jps 3135 ResourceManager 2678 DataNode 2428 NameNode 5044 Jps 2961 SecondaryNameNode Run a test Let s try this word count example. Don t try to run the randomwriter example by default if you want waste 10 GB

14 RandomWriter example writes 10 gig (by default) of random data/host to DFS using Map/Reduce. Try to find some txt such like this cool stuff $ wget $ unzip 4300.zip hduser@ubuntu:/usr/local/hadoop/share/hadoop/mapreduce$ hadoop dfs -mkdir /tmp hduser@ubuntu:/usr/local/hadoop/share/hadoop/mapreduce$ hadoop dfs -copyfromlocal 4300.txt /tmp Now ready to roll! hduser@ubuntu:/usr/local/hadoop/share/hadoop/mapreduce$ hadoop jar hadoop-mapreduceexamples beta.jar wordcount /tmp/4300.txt /tmp/output3 14/01/28 20:04:56 WARN conf.configuration: session.id is deprecated. Instead, use dfs.metrics.session-id 14/01/28 20:04:56 INFO jvm.jvmmetrics: Initializing JVM Metrics with processname=jobtracker, sessionid= 14/01/28 20:04:56 INFO input.fileinputformat: Total input paths to process : 1 14/01/28 20:04:56 INFO mapreduce.jobsubmitter: number of splits:1 14/01/28 20:04:56 WARN conf.configuration: user.name is deprecated. Instead, use mapreduce.job.user.name 14/01/28 20:04:56 WARN conf.configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar 14/01/28 20:04:56 WARN conf.configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 14/01/28 20:04:56 WARN conf.configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class 14/01/28 20:04:56 WARN conf.configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 14/01/28 20:04:56 WARN conf.configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name 14/01/28 20:04:56 WARN conf.configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class

15 14/01/28 20:04:56 WARN conf.configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 14/01/28 20:04:56 WARN conf.configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 14/01/28 20:04:56 WARN conf.configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 14/01/28 20:04:56 WARN conf.configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 14/01/28 20:04:56 WARN conf.configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 14/01/28 20:04:56 INFO mapreduce.jobsubmitter: Submitting tokens for job: job_local _ /01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser /.staging/job_local _0001/job.xml:an attempt to override final parameter: mapreduce.local.dir; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser /.staging/job_local _0001/job.xml:an attempt to override final parameter: dfs.namenode.name.dir; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser /.staging/job_local _0001/job.xml:an attempt to override final parameter: mapreduce.job.endnotification.max.retry.interval; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser /.staging/job_local _0001/job.xml:an attempt to override final parameter: dfs.datanode.data.dir; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser /.staging/job_local _0001/job.xml:an attempt to override final parameter: mapreduce.job.endnotification.max.attempts; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser /.staging/job_local _0001/job.xml:an attempt to override final parameter: fs.defaultfs; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser /.staging/job_local _0001/job.xml:an attempt to override final parameter: mapreduce.system.dir; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local _0001.xml:an attempt to override final parameter: mapreduce.local.dir; Ignoring.

16 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local _0001.xml:an attempt to override final parameter: dfs.namenode.name.dir; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local _0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local _0001.xml:an attempt to override final parameter: dfs.datanode.data.dir; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local _0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local _0001.xml:an attempt to override final parameter: fs.defaultfs; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local _0001.xml:an attempt to override final parameter: mapreduce.system.dir; Ignoring. 14/01/28 20:04:56 INFO mapreduce.job: The url to track the job: 14/01/28 20:04:56 INFO mapreduce.job: Running job: job_local _ /01/28 20:04:56 INFO mapred.localjobrunner: OutputCommitter set in config null 14/01/28 20:04:56 INFO mapred.localjobrunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.fileoutputcommitter 14/01/28 20:04:56 INFO mapred.localjobrunner: Waiting for map tasks 14/01/28 20:04:56 INFO mapred.localjobrunner: Starting task: attempt_local _0001_m_000000_0 14/01/28 20:04:56 INFO mapred.task: Using ResourceCalculatorProcessTree : [ ] 14/01/28 20:04:56 INFO mapred.maptask: Processing split: hdfs://localhost:9000/tmp/4300.txt: /01/28 20:04:57 INFO mapred.maptask: Map output collector class = org.apache.hadoop.mapred.maptask$mapoutputbuffer 14/01/28 20:04:57 INFO mapred.maptask: (EQUATOR) 0 kvi ( ) 14/01/28 20:04:57 INFO mapred.maptask: mapreduce.task.io.sort.mb: /01/28 20:04:57 INFO mapred.maptask: soft limit at

17 14/01/28 20:04:57 INFO mapred.maptask: bufstart = 0; bufvoid = /01/28 20:04:57 INFO mapred.maptask: kvstart = ; length = /01/28 20:04:57 INFO mapred.localjobrunner: 14/01/28 20:04:57 INFO mapred.maptask: Starting flush of map output 14/01/28 20:04:57 INFO mapred.maptask: Spilling map output 14/01/28 20:04:57 INFO mapred.maptask: bufstart = 0; bufend = ; bufvoid = /01/28 20:04:57 INFO mapred.maptask: kvstart = ( ); kvend = ( ); length = / /01/28 20:04:57 INFO mapreduce.job: Job job_local _0001 running in uber mode : false 14/01/28 20:04:57 INFO mapreduce.job: map 0% reduce 0% 14/01/28 20:04:58 INFO mapred.maptask: Finished spill 0 14/01/28 20:04:58 INFO mapred.task: Task:attempt_local _0001_m_000000_0 is done. And is in the process of committing 14/01/28 20:04:58 INFO mapred.localjobrunner: map 14/01/28 20:04:58 INFO mapred.task: Task 'attempt_local _0001_m_000000_0' done. 14/01/28 20:04:58 INFO mapred.localjobrunner: Finishing task: attempt_local _0001_m_000000_0 14/01/28 20:04:58 INFO mapred.localjobrunner: Map task executor complete. 14/01/28 20:04:58 INFO mapred.task: Using ResourceCalculatorProcessTree : [ ] 14/01/28 20:04:58 INFO mapred.merger: Merging 1 sorted segments 14/01/28 20:04:58 INFO mapred.merger: Down to the last merge-pass, with 1 segments left of total size: bytes 14/01/28 20:04:58 INFO mapred.localjobrunner: 14/01/28 20:04:58 WARN conf.configuration: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 14/01/28 20:04:58 INFO mapreduce.job: map 100% reduce 0% 14/01/28 20:04:58 INFO mapred.task: Task:attempt_local _0001_r_000000_0 is done. And is in the process of committing 14/01/28 20:04:58 INFO mapred.localjobrunner:

18 14/01/28 20:04:58 INFO mapred.task: Task attempt_local _0001_r_000000_0 is allowed to commit now 14/01/28 20:04:58 INFO output.fileoutputcommitter: Saved output of task 'attempt_local _0001_r_000000_0' to hdfs://localhost:9000/tmp/output3/_temporary/0/task_local _0001_r_ /01/28 20:04:58 INFO mapred.localjobrunner: reduce > reduce 14/01/28 20:04:58 INFO mapred.task: Task 'attempt_local _0001_r_000000_0' done. 14/01/28 20:04:59 INFO mapreduce.job: map 100% reduce 100% 14/01/28 20:04:59 INFO mapreduce.job: Job job_local _0001 completed successfully 14/01/28 20:04:59 INFO mapreduce.job: Counters: 32 File System Counters FILE: Number of bytes read= FILE: Number of bytes written= FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read= HDFS: Number of bytes written= HDFS: Number of read operations=13 HDFS: Number of large read operations=0 HDFS: Number of write operations=4 Map-Reduce Framework Map input records=33056 Map output records= Map output bytes= Map output materialized bytes= Input split bytes=99 Combine input records= Combine output records=50095

19 Reduce input groups=50095 Reduce shuffle bytes=0 Reduce input records=50095 Reduce output records=50095 Spilled Records= Shuffled Maps =0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=37 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)= File Input Format Counters Bytes Read= File Output Format Counters Bytes Written= see the result by hadoop dfs -cat /tmp/output3/part* Or from web console

20 Remove a directory hduser@ubuntu:/usr/local/hadoop/share/hadoop/mapreduce$ hadoop dfs -rmr /tmp/output DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. rmr: DEPRECATED: Please use 'rm -r' instead. 14/01/30 09:26:27 INFO fs.trashpolicydefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /tmp/output hduser@ubuntu:/usr/local/hadoop/share/hadoop/mapreduce$ hadoop dfs -rmr /tmp/output2 DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it.

21 rmr: DEPRECATED: Please use 'rm -r' instead. 14/01/30 09:26:56 INFO fs.trashpolicydefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /tmp/output2 Install Hadoop as Master + Slave on RHEL 6.5 General steps 1. Preparation Master Slave Hostname master.hadoop.advol node.hadoop.advol edit /etc/hosts JDK /usr/java/jdk As master Download rpm from oracle $HADOOP_HOME /data/hadoop/hadoop As master Can use separated logic volume as hdfs user hadoop hadoop Adduser hadoop Also grant owner of $HADOOP_HOME to hadoop Hardware Standalone hardware, RHEL 6.5 RHEL 6.5 on VMWARE on my laptop 2. DO THE SAME ON MASTER NODE AND SLAVE NODES TO INSTALL JDK and ENVIRONMENT CONFIGURATION 3. MODIFY HOSTNAME TO ENABLE PING BY HOSTNAME TO EACH OTHER 4. MODIFY FIREWALL IPTABLES 5. ADD SSH PUBLIC KEY FROM MASTER TO SLAVE AS WELL AS FROM SLAVE TO MASTER 6. KEEP DIRECTORY BE THE SAME 7. MODIFY SLAVE FILE IN $HADOOPHOME/ETC 8. FORMAT NAMENODE AGAIN 9. START-ALL FROM MASTER Hadoop-env.sh Add JAVA_HOME # The java implementation to use. export JAVA_HOME=/usr/java/jdk1.7.0_55

22 core-site.xml <configuration> <name>hadoop.tmp.dir</name> <value>/data/hadoop/hadoopfs/tmp</value> <description>a base for other temporary directories.</description> <name>fs.default.name</name> <value>hdfs://master.hadoop.advol:9000</value> <final>true</final> </configuration> hdfs-site.xml <configuration> <name>dfs.replication</name> <value>2</value> <name>dfs.permissions</name> <value>false</value> <name>dfs.namenode.name.dir</name> <value>file:/data/hadoop/hadoopfs/dfs/name</value>

23 <final>true</final> <name>dfs.datanode.data.dir</name> <value>file:/data/hadoop/hadoopfs/dfs/data</value> <final>true</final> <name>dfs.datanode.max.xcievers</name> <value>4096</value> </configuration> yarn-site.xml <configuration> <name>dfs.replication</name> <value>2</value> <name>dfs.permissions</name> <value>false</value> <name>dfs.namenode.name.dir</name> <value>file:/data/hadoop/hadoopfs/dfs/name</value> <final>true</final> <name>dfs.datanode.data.dir</name>

24 <value>file:/data/hadoop/hadoopfs/dfs/data</value> <final>true</final> <name>dfs.datanode.max.xcievers</name> <value>4096</value> </configuration> mapred-site.xml <configuration> <name>mapreduce.framework.name</name> <value>yarn</value> <name>mapreduce.cluster.local.dir</name> <value>file:/data/hadoop/hadoopfs/dfs/local</value> <final>true</final> <name>mapreduce.cluster.temp.dir</name> <value>file:/data/hadoop/hadoopfs/dfs/tmp</value> <description>no description</description> <final>true</final> </configuration>

25 After start (~/$HADOOP_HOME/sbin/start-all.sh ), use JPS to check JAVA threads, at Master: sbin]$ jps ResourceManager NodeManager Jps SecondaryNameNode NameNode sbin]$ JPS at SLAVE ~]$ jps 3934 Jps 3480 DataNode 3583 NodeManager ~]$ If have issue with copyfromlocal (only one node accept, while another raise a I/O issue), check firewall Check hdfs [hadoop@master hadoop]$ hdfs dfsadmin -report Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/05/09 17:38:44 WARN util.nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Configured Capacity: ( GB) Present Capacity: ( GB) DFS Remaining: ( GB) DFS Used: (3.11 MB) DFS Used%: 0.00%

26 Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: Datanodes available: 2 (2 total, 0 dead) Live datanodes: Name: :50010 (node.hadoop.advol) Hostname: node.hadoop.advol Decommission Status : Normal Configured Capacity: (17.43 GB) DFS Used: (28 KB) Non DFS Used: (4.23 GB) DFS Remaining: (13.20 GB) DFS Used%: 0.00% DFS Remaining%: 75.73% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: % Cache Remaining%: 0.00% Last contact: Fri May 09 17:38:43 EDT 2014 Name: :50010 (master.hadoop.advol) Hostname: master.hadoop.advol Decommission Status : Normal Configured Capacity: ( GB) DFS Used: (3.09 MB)

27 Non DFS Used: (15.76 GB) DFS Remaining: ( GB) DFS Used%: 0.00% DFS Remaining%: 89.55% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: % Cache Remaining%: 0.00% Last contact: Fri May 09 17:38:43 EDT 2014 [hadoop@master hadoop]$ Run word count on 1 master + 2 slave nodes Can also check application job tracker at master server localhost not accepted [hadoop@master ~]$ hadoop jar /data/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/hadoopmapreduce-examples jar wordcount /tmp/4300.txt /tmp/output4 Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/05/12 13:04:13 WARN util.nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/05/12 13:04:14 INFO client.rmproxy: Connecting to ResourceManager at master.hadoop.advol/ : /05/12 13:04:15 INFO input.fileinputformat: Total input paths to process : 1 14/05/12 13:04:15 INFO mapreduce.jobsubmitter: number of splits:1

28 14/05/12 13:04:15 INFO mapreduce.jobsubmitter: Submitting tokens for job: job_ _ /05/12 13:04:16 INFO impl.yarnclientimpl: Submitted application application_ _ /05/12 13:04:16 INFO mapreduce.job: The url to track the job: 14/05/12 13:04:16 INFO mapreduce.job: Running job: job_ _ /05/12 13:04:26 INFO mapreduce.job: Job job_ _0001 running in uber mode : false 14/05/12 13:04:26 INFO mapreduce.job: map 0% reduce 0% 14/05/12 13:04:35 INFO mapreduce.job: map 100% reduce 0% 14/05/12 13:04:41 INFO mapreduce.job: map 100% reduce 100% 14/05/12 13:04:41 INFO mapreduce.job: Job job_ _0001 completed successfully 14/05/12 13:04:42 INFO mapreduce.job: Counters: 49 File System Counters FILE: Number of bytes read= FILE: Number of bytes written= FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read= HDFS: Number of bytes written= HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=6084

29 Total time spent by all reduces in occupied slots (ms)=3660 Total time spent by all map tasks (ms)=6084 Total time spent by all reduce tasks (ms)=3660 Total vcore-seconds taken by all map tasks=6084 Total vcore-seconds taken by all reduce tasks=3660 Total megabyte-seconds taken by all map tasks= Total megabyte-seconds taken by all reduce tasks= Map-Reduce Framework Map input records=33056 Map output records= Map output bytes= Map output materialized bytes= Input split bytes=109 Combine input records= Combine output records=50095 Reduce input groups=50095 Reduce shuffle bytes= Reduce input records=50095 Reduce output records=50095 Spilled Records= Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=286 CPU time spent (ms)=4990 Physical memory (bytes) snapshot= Virtual memory (bytes) snapshot= Total committed heap usage (bytes)= Shuffle Errors BAD_ID=0

30 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read= File Output Format Counters Bytes Written= ~]$ Run word count on master+slave (1 node) sbin]$ hadoop jar /data/hadoop/hadoop /share/hadoop/mapreduce/hadoop-mapreduce-examples jar wordcount /tmp/4300.txt /tmp/output5 Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/05/12 13:29:24 WARN util.nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/05/12 13:29:25 INFO client.rmproxy: Connecting to ResourceManager at master.hadoop.advol/ : /05/12 13:29:26 INFO input.fileinputformat: Total input paths to process : 1 14/05/12 13:29:26 INFO mapreduce.jobsubmitter: number of splits:1 14/05/12 13:29:26 INFO mapreduce.jobsubmitter: Submitting tokens for job: job_ _ /05/12 13:29:27 INFO impl.yarnclientimpl: Submitted application application_ _0001

31 14/05/12 13:29:27 INFO mapreduce.job: The url to track the job: 14/05/12 13:29:27 INFO mapreduce.job: Running job: job_ _ /05/12 13:29:34 INFO mapreduce.job: Job job_ _0001 running in uber mode : false 14/05/12 13:29:34 INFO mapreduce.job: map 0% reduce 0% 14/05/12 13:29:41 INFO mapreduce.job: map 100% reduce 0% 14/05/12 13:29:47 INFO mapreduce.job: map 100% reduce 100% 14/05/12 13:29:48 INFO mapreduce.job: Job job_ _0001 completed successfully 14/05/12 13:29:48 INFO mapreduce.job: Counters: 49 File System Counters FILE: Number of bytes read= FILE: Number of bytes written= FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read= HDFS: Number of bytes written= HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=4254 Total time spent by all reduces in occupied slots (ms)=3904 Total time spent by all map tasks (ms)=4254 Total time spent by all reduce tasks (ms)=3904 Total vcore-seconds taken by all map tasks=4254

32 Total vcore-seconds taken by all reduce tasks=3904 Total megabyte-seconds taken by all map tasks= Total megabyte-seconds taken by all reduce tasks= Map-Reduce Framework Map input records=33056 Map output records= Map output bytes= Map output materialized bytes= Input split bytes=109 Combine input records= Combine output records=50095 Reduce input groups=50095 Reduce shuffle bytes= Reduce input records=50095 Reduce output records=50095 Spilled Records= Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=119 CPU time spent (ms)=5290 Physical memory (bytes) snapshot= Virtual memory (bytes) snapshot= Total committed heap usage (bytes)= Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0

33 WRONG_REDUCE=0 File Input Format Counters Bytes Read= File Output Format Counters Bytes Written= sbin]$ Observation: I don t see any improvement on the result of running on standalone or master+slave Run word count on bible on standalone and cluster Try to work on a bigger file [hadoop@master hadoop]$ wget [hadoop@master hadoop]$ unzip AV_txt.zip [hadoop@master hadoop]$ mv AV1611Bible.txt bible.txt [hadoop@master hadoop]$ hdfs dfs -copyfromlocal bible.txt /tmp/bible.txt [hadoop@master hadoop]$ hadoop jar /data/hadoop/hadoop /share/hadoop/mapreduce/hadoop-mapreduce-examples jar wordcount /tmp/bible.txt /tmp/bible/ [hadoop@master hadoop]$ hdfs dfs -getmerge /tmp/bible/./bible.result [hadoop@master hadoop]$ cat bible.result grep god Gudgodah 1 Gudgodah; 1 [god] 1 [god]: 1 [gods 1 [gods], 2 ergodiwkthv 1 god 20 god, 19 god. 9 god: 4

34 god; 3 goddess 4 goddess. 1 godliness 4 godliness) 1 godliness, 4 godliness. 1 godliness: 2 godliness; 3 godly 16 godly, 1 godly-learned 1 gods 93 gods, 88 gods. 28 gods: 15 gods; 8 gods? 7 ungodliness 3 ungodliness. 1 ungodly 17 ungodly, 5 ungodly. 2 ungodly; 2 ungodly? 1

35 Run it on Master + 2 slave nodes [hadoop@master sbin]$./start-all.sh [hadoop@master sbin]$ jps 853 SecondaryNameNode 598 DataNode 1009 ResourceManager 466 NameNode 1452 Jps 1120 NodeManager [hadoop@master sbin]$ hadoop jar /data/hadoop/hadoop /share/hadoop/mapreduce/hadoop-mapreduce-examples jar wordcount /tmp/bible.txt /tmp/bible2/ Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/05/12 14:01:50 WARN util.nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/05/12 14:01:51 INFO client.rmproxy: Connecting to ResourceManager at master.hadoop.advol/ : /05/12 14:01:52 INFO input.fileinputformat: Total input paths to process : 1 14/05/12 14:01:52 INFO mapreduce.jobsubmitter: number of splits:1 14/05/12 14:01:52 INFO mapreduce.jobsubmitter: Submitting tokens for job: job_ _ /05/12 14:01:52 INFO impl.yarnclientimpl: Submitted application application_ _ /05/12 14:01:52 INFO mapreduce.job: The url to track the job: 14/05/12 14:01:53 INFO mapreduce.job: Running job: job_ _ /05/12 14:02:00 INFO mapreduce.job: Job job_ _0001 running in uber mode : false

36 14/05/12 14:02:00 INFO mapreduce.job: map 0% reduce 0% 14/05/12 14:02:12 INFO mapreduce.job: map 100% reduce 0% 14/05/12 14:02:21 INFO mapreduce.job: map 100% reduce 100% 14/05/12 14:02:22 INFO mapreduce.job: Job job_ _0001 completed successfully 14/05/12 14:02:22 INFO mapreduce.job: Counters: 49 File System Counters FILE: Number of bytes read= FILE: Number of bytes written= FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read= HDFS: Number of bytes written= HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=10832 Total time spent by all reduces in occupied slots (ms)=5080 Total time spent by all map tasks (ms)=10832 Total time spent by all reduce tasks (ms)=5080 Total vcore-seconds taken by all map tasks=10832 Total vcore-seconds taken by all reduce tasks=5080 Total megabyte-seconds taken by all map tasks= Total megabyte-seconds taken by all reduce tasks= Map-Reduce Framework

37 Map input records=33979 Map output records= Map output bytes= Map output materialized bytes= Input split bytes=110 Combine input records= Combine output records=32635 Reduce input groups=32635 Reduce shuffle bytes= Reduce input records=32635 Reduce output records=32635 Spilled Records=65270 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=760 CPU time spent (ms)=7800 Physical memory (bytes) snapshot= Virtual memory (bytes) snapshot= Total committed heap usage (bytes)= Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read= File Output Format Counters

38 Bytes Written= sbin]$ hdfs dfs -getmerge /tmp/bible/./bible2.result Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/05/12 14:07:39 WARN util.nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable sbin]$ cat bible2.result grep god Gudgodah 1 Gudgodah; 1 [god] 1 [god]: 1 [gods 1 [gods], 2 ergodiwkthv 1 god 20 god, 19 god. 9 god: 4 god; 3 goddess 4 goddess. 1 godliness 4 godliness) 1 godliness, 4 godliness. 1 godliness: 2 godliness; 3 godly 16

39 godly, 1 godly-learned 1 gods 93 gods, 88 gods. 28 gods: 15 gods; 8 gods? 7 ungodliness 3 ungodliness. 1 ungodly 17 ungodly, 5 ungodly. 2 ungodly; 2 ungodly? 1 [hadoop@master sbin]$ Run word count on my s master + 2 slave nodes Try to run another bigger file My s.txt is a export from my outlook which includes 3 years [hadoop@master hadoop]$ hadoop jar /data/hadoop/hadoop /share/hadoop/mapreduce/hadoop-mapreduce-examples jar wordcount /tmp/my s.txt /tmp/my / Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.

40 It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/05/12 14:54:45 WARN util.nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/05/12 14:54:46 INFO client.rmproxy: Connecting to ResourceManager at master.hadoop.advol/ : /05/12 14:54:46 INFO input.fileinputformat: Total input paths to process : 1 14/05/12 14:54:46 INFO mapreduce.jobsubmitter: number of splits:1 14/05/12 14:54:47 INFO mapreduce.jobsubmitter: Submitting tokens for job: job_ _ /05/12 14:54:47 INFO impl.yarnclientimpl: Submitted application application_ _ /05/12 14:54:47 INFO mapreduce.job: The url to track the job: 14/05/12 14:54:47 INFO mapreduce.job: Running job: job_ _ /05/12 14:54:54 INFO mapreduce.job: Job job_ _0002 running in uber mode : false 14/05/12 14:54:54 INFO mapreduce.job: map 0% reduce 0% 14/05/12 14:55:10 INFO mapreduce.job: map 21% reduce 0% 14/05/12 14:55:13 INFO mapreduce.job: map 24% reduce 0% 14/05/12 14:55:16 INFO mapreduce.job: map 43% reduce 0% 14/05/12 14:55:19 INFO mapreduce.job: map 47% reduce 0% 14/05/12 14:55:22 INFO mapreduce.job: map 62% reduce 0% 14/05/12 14:55:25 INFO mapreduce.job: map 67% reduce 0% 14/05/12 14:55:26 INFO mapreduce.job: map 100% reduce 0% 14/05/12 14:55:33 INFO mapreduce.job: map 100% reduce 100% 14/05/12 14:55:33 INFO mapreduce.job: Job job_ _0002 completed successfully 14/05/12 14:55:33 INFO mapreduce.job: Counters: 49 File System Counters FILE: Number of bytes read= FILE: Number of bytes written=

41 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read= HDFS: Number of bytes written= HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=29563 Total time spent by all reduces in occupied slots (ms)=4477 Total time spent by all map tasks (ms)=29563 Total time spent by all reduce tasks (ms)=4477 Total vcore-seconds taken by all map tasks=29563 Total vcore-seconds taken by all reduce tasks=4477 Total megabyte-seconds taken by all map tasks= Total megabyte-seconds taken by all reduce tasks= Map-Reduce Framework Map input records= Map output records= Map output bytes= Map output materialized bytes= Input split bytes=113 Combine input records= Combine output records= Reduce input groups= Reduce shuffle bytes=

42 Reduce input records= Reduce output records= Spilled Records= Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=604 CPU time spent (ms)=32020 Physical memory (bytes) snapshot= Virtual memory (bytes) snapshot= Total committed heap usage (bytes)= Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read= File Output Format Counters Bytes Written= hadoop]$

43 Start history server Not start by default sbin]$./mr-jobhistory-daemon.sh start historyserver starting historyserver, logging to /data/hadoop/hadoop-2.4.0/logs/mapred-hadoophistoryserver-master.hadoop.advol.out sbin]$ Install Hbase Understand what is Hadoop Hbase Hive Pig hive 与 hbase 区 别 1 hive 是 sql 语 言, 通 过 数 据 库 的 方 式 来 操 作 hdfs 文 件 系 统, 为 了 简 化 编 程, 底 层 计 算 方 式 为 mapreduce 2 hive 是 面 向 行 存 储 的 数 据 库 3 Hive 本 身 不 存 储 和 计 算 数 据, 它 完 全 依 赖 于 HDFS 和 MapReduce,Hive 中 的 表 纯 逻 辑 4 HBase 为 查 询 而 生 的, 它 通 过 组 织 起 节 点 內 所 有 机 器 的 內 存, 提 供 一 個 超 大 的 內 存 Hash 表 5 hbase 不 是 关 系 型 数 据 库, 而 是 一 个 在 hdfs 上 开 发 的 面 向 列 的 分 布 式 数 据 库, 不 支 持 sql 6 hbase 是 物 理 表, 不 是 逻 辑 表, 提 供 一 个 超 大 的 内 存 hash 表, 搜 索 引 擎 通 过 它 来 存 储 索 引, 方 便 查 询 操 作

44 7 hbase 是 列 存 储 Pig Pig 是 一 种 数 据 流 语 言, 用 来 快 速 轻 松 的 处 理 巨 大 的 数 据 Pig 包 含 两 个 部 分 :Pig Interface,Pig Latin Pig 可 以 非 常 方 便 的 处 理 HDFS 和 HBase 的 数 据, 和 Hive 一 样,Pig 可 以 非 常 高 效 的 处 理 其 需 要 做 的, 通 过 直 接 操 作 Pig 查 询 可 以 节 省 大 量 的 劳 动 和 时 间 当 你 想 在 你 的 数 据 上 做 一 些 转 换, 并 且 不 想 编 写 MapReduce jobs 就 可 以 用 Pig. Hive 起 源 于 FaceBook,Hive 在 Hadoop 中 扮 演 数 据 仓 库 的 角 色 建 立 在 Hadoop 集 群 的 最 顶 层, 对 存 储 在 Hadoop 群 上 的 数 据 提 供 类 SQL 的 接 口 进 行 操 作 你 可 以 用 HiveQL 进 行 select,join, 等 等 操 作 如 果 你 有 数 据 仓 库 的 需 求 并 且 你 擅 长 写 SQL 并 且 不 想 写 MapReduce jobs 就 可 以 用 Hive 代 替 HBase HBase 作 为 面 向 列 的 数 据 库 运 行 在 HDFS 之 上,HDFS 缺 乏 随 即 读 写 操 作,HBase 正 是 为 此 而 出 现 HBase 以 Google BigTable 为 蓝 本, 以 键 值 对 的 形 式 存 储 项 目 的 目 标 就 是 快 速 在 主 机 内 数 十 亿 行 数 据 中 定 位 所 需 的 数 据 并 访 问 它 HBase 是 一 个 数 据 库, 一 个 NoSql 的 数 据 库, 像 其 他 数 据 库 一 样 提 供 随 即 读 写 功 能,Hadoop 不 能 满 足 实 时 需 要,HBase 正 可 以 满 足 如 果 你 需 要 实 时 访 问 一 些 数 据, 就 把 它 存 入 HBase 你 可 以 用 Hadoop 作 为 静 态 数 据 仓 库,HBase 作 为 数 据 存 储, 放 那 些 进 行 一 些 操 作 会 改 变 的 数 据 Pig VS Hive Hive 更 适 合 于 数 据 仓 库 的 任 务,Hive 主 要 用 于 静 态 的 结 构 以 及 需 要 经 常 分 析 的 工 作 Hive 与 SQL 相 似 促 使 其 成 为 Hadoop 与 其 他 BI 工 具 结 合 的 理 想 交 集 Pig 赋 予 开 发 人 员 在 大 数 据 集 领 域 更 多 的 灵 活 性, 并 允 许 开 发 简 洁 的 脚 本 用 于 转 换 数 据 流 以 便 嵌 入 到 较 大 的 应 用 程 序 Pig 相 比 Hive 相 对 轻 量, 它 主 要 的 优 势 是 相 比 于 直 接 使 用 Hadoop Java APIs 可 大 幅 削 减 代 码 量 正 因 为 如 此,Pig 仍 然 是 吸 引 大 量 的 软 件 开 发 人 员 Hive 和 Pig 都 可 以 与 HBase 组 合 使 用,Hive 和 Pig 还 为 HBase 提 供 了 高 层 语 言 支 持, 使 得 在 HBase 上 进 行 数 据 统 计 处 理 变 的 非 常 简 单 Hive VS HBase Hive 是 建 立 在 Hadoop 之 上 为 了 减 少 MapReduce jobs 编 写 工 作 的 批 处 理 系 统,HBase 是 为 了 支 持 弥 补 Hadoop 对 实 时 操 作 的 缺 陷 的 项 目 想 象 你 在 操 作 RMDB 数 据 库, 如 果 是 全 表 扫 描, 就 用 Hive+Hadoop, 如 果 是 索 引 访 问, 就 用 HBase+Hadoop Hive query 就 是 MapReduce jobs 可 以 从 5 分 钟 到 数 小 时 不 止,HBase 是 非 常 高 效 的, 肯 定 比 Hive 高 效 的 多 If see:

45 org.apache.hadoop.ipc.remoteexception: Server IPC version 9 cannot communicate with client version 4 Check Hbase version and Hadoop version [hadoop@master download]$ wget /hbase hadoop2-bin.tar.gz :54: /hbase hadoop2-bin.tar.gz Resolving mirror.csclub.uwaterloo.ca Connecting to mirror.csclub.uwaterloo.ca :80... connected. HTTP request sent, awaiting response OK Length: (78M) [application/x-gzip] Saving to: 鈥 渉 base hadoop2-bin.tar.gz 鈥? 100%[==================================================================>] 82,246, M/s in 3.9s :54:22 (20.0 MB/s) - 鈥 渉 base hadoop2-bin.tar.gz 鈥?saved [ / ] [hadoop@master download]$ tar xfz hbase hadoop2-bin.tar.gz -C /data/hbase/ Download and modify [hadoop@master hbase hadoop2]$ cd conf [hadoop@master conf]$ ls hadoop-metrics2-hbase.properties hbase-env.sh hbase-site.xml regionservers hbase-env.cmd hbase-policy.xml log4j.properties [hadoop@master conf]$ more hbase-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration>

46 <name>hbase.cluster.distributed</name> <value>true</value> <name>hbase.rootdir</name> <value>hdfs://master.hadoop.advol:9000/hbase</value> </configuration> Create a table card and add a record [hadoop@master bin]$./hbase shell hbase(main):019:0> create 'card','mycf' 0 row(s) in seconds => Hbase::Table - card hbase(main):020:0> list 'card' TABLE card 1 row(s) in seconds => ["card"] hbase(main):021:0> put 'card','rowid1','mycf:a','sword of Truth' 0 row(s) in seconds hbase(main):022:0> list 'card' TABLE card 1 row(s) in seconds => ["card"]

47 hbase(main):024:0> scan 'card' ROW rowid1 Truth COLUMN+CELL column=mycf:a, timestamp= , value=sword of 1 row(s) in seconds hbase(main):025:0> [hadoop@master bin]$ [hadoop@master bin]$ [hadoop@master bin]$./stop-hbase.sh stopping hbase... localhost: stopping zookeeper. [hadoop@master bin]$ jps 8707 NameNode 9248 ResourceManager 9092 SecondaryNameNode Jps 8837 DataNode 9359 NodeManager 5532 JobHistoryServer [hadoop@master bin]$./start-hbase.sh localhost: starting zookeeper, logging to /data/hbase/hbase hadoop2/bin/../logs/hbase-hadoop-zookeeper-master.hadoop.advol.out starting master, logging to /data/hbase/hbase hadoop2/bin/../logs/hbase-hadoopmaster-master.hadoop.advol.out localhost: starting regionserver, logging to /data/hbase/hbase hadoop2/bin/../logs/hbase-hadoop-regionserver-master.hadoop.advol.out [hadoop@master bin]$./hbase shell :10:51,394 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available HBase Shell; enter 'help<return>' for list of supported commands.