Welcome, this document is a record of my installation of Hadoop for study purpose. Version Version Date Content and Change 1.0 2013 Dec Initialize study Hadoop Install basic environment, run first word count program 2.0 2014 May Expend Hadoop to master+slave (1 master + 2 slave nodes) on RHEL Run word count on bigger file, see improvement Study Hadoop based projects Hive, Pig, Hbase Install Hbase on Hadoop as full distribute mode Install Hadoop on Ubuntu and run as standalone Hadoop version: 2.2.0 OS: Ubuntu 32 bit hduser@ubuntu:~$ cat /etc/issue Ubuntu 12.10 \n \l Host: VMWARE workstation on Windows 8 Please be advice this is a draft, and just fit my own circumstance. References: http://blog.csdn.net/focusheart/article/details/14005893 http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/ http://cs.smith.edu/dftwiki/index.php/hadoop_tutorial_1_--_running_wordcount http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/commandsmanual.html Developer http://www.osedu.net/article/nosql/2012-05-02/435.html http://www.slideshare.net/waue/hadoop-map-reduce-3019713!!to-do list!! Following things need to be verified:
1. Job tracker interface can t show Fixed 2. Where is job tracker? It seems they changed job tracker to YARN (Next generation Map-reduce) Use local:8088 instead of localhost 50030 http://docs.aws.amazon.com/elasticmapreduce/latest/developerguide/emr-hadoop-2.2.0- features.html https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoopcommon/clustersetup.html Web Interfaces Once the Hadoop cluster is up and running check the web-ui of the components as described below: NameNode http://nn_host:port/ ResourceManager http://rm_host:port/ MapReduce JobHistory Server http://jhs_host:port/ Apache Hadoop NextGen MapReduce (YARN) MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2) or YARN. So when check with JPS, there is no longer has jobtracker http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/ Web UI In MR1, the JobTracker web UI served detailed information about the state of the cluster and the jobs currently and recently running on it. It also contained the job history page, which served information from disk about older jobs.
The MR2 web UI provides the same information structured in the same way, but has been revamped with a new look and feel. The ResourceManager UI, which includes information about running applications and the state of the cluster, is now located by default at:8088. The job history UI is now located by default at:19888. Jobs can be searched and viewed there just as they could in MR1. Because the ResourceManager is meant to be agnostic to many of the concepts in MapReduce, it cannot host job information directly. Instead, it proxies to a web UI that can. If the job is running, this is the relevant MapReduce Application Master; if it has completed, this is the JobHistoryServer. In this sense, the user experience is similar to that of MR1, but the information is coming from different places. Some conceptions What Is Apache Hadoop? The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing. The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver highavailability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. The project includes these modules: Hadoop Common: The common utilities that support the other Hadoop modules. Hadoop Distributed File System (HDFS ): A distributed file system that provides high-throughput access to application data. Hadoop YARN: A framework for job scheduling and cluster resource management. Hadoop MapReduce: A YARN-based system for parallel processing of large data sets. Other Hadoop-related projects at Apache include: Ambari : A web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters which includes support for Hadoop HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides a dashboard for viewing cluster health such as heatmaps and ability to view MapReduce, Pig and Hive applications visually alongwith features to diagnose their performance characteristics in a user-friendly manner. Avro : A data serialization system. Cassandra : A scalable multi-master database with no single points of failure. Chukwa : A data collection system for managing large distributed systems. HBase : A scalable, distributed database that supports structured data storage for large tables. Hive : A data warehouse infrastructure that provides data summarization and ad hoc querying. Mahout : A Scalable machine learning and data mining library. Pig : A high-level data-flow language and execution framework for parallel computation.
Spark : A fast and general compute engine for Hadoop data. Spark provides a simple and expressive programming model that supports a wide range of applications, including ETL, machine learning, stream processing, and graph computation. ZooKeeper : A high-performance coordination service for distributed applications. Hadoop follows the idea of job map and reduce. It has a DFS file system to support distribution Important ports 1.Job Tracker :50030 (no longer in 2.20?) 2.HDFS :50070 3.HDFS communication:9000 4.MapReducecommunication:9001 Management 1. HDFS web interface http://hostname:50070 2. MapReduce interface http://hostname:50030 Install Ubuntu You may like to install it by a flash disk (on a new tower) Install required packages For example: JRE/JDK $ sudo apt-get update $ sudo apt-get install sun-java6-jdk Add user and user group $ sudo addgroup hadoop
$ sudo adduser --ingroup hadoop hduser This will add the user hduser and the group hadoop to your local machine. Configuring SSH user@ubuntu:~$ su - hduser hduser@ubuntu:~$ ssh-keygen -t rsa -P "" Generating public/private rsa key pair. Enter file in which to save the key (/home/hduser/.ssh/id_rsa): Created directory '/home/hduser/.ssh'. Your identification has been saved in /home/hduser/.ssh/id_rsa. Your public key has been saved in /home/hduser/.ssh/id_rsa.pub. The key fingerprint is: 9b:82:ea:58:b4:e0:35:d7:ff:19:66:a6:ef:ae:0e:d2 hduser@ubuntu The key's randomart image is: [...snipp...] hduser@ubuntu:~$ If success, you may ssh to your host without input credentials Disable IPV6 (not proved whether necessary) # disable ipv6 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 You have to reboot your machine in order to make the changes take effect. You can check whether IPv6 is enabled on your machine with the following command: $ cat /proc/sys/net/ipv6/conf/all/disable_ipv6
Install Hadoop files $ cd /usr/local $ sudo tar xzf hadoop-1.0.3.tar.gz $ sudo mv hadoop-1.0.3 hadoop $ sudo chown -R hduser:hadoop hadoop You may also think to make you directory where to place DFS data file, in my case it is /hadoopfs. If necessary, grant owner to hduser:hadoop Update environment variables vim /home/hduser/.rcbash export HADOOP_HOME=/usr/local/hadoop unalias fs &> /dev/null alias fs="hadoop fs" unalias hls &> /dev/null alias hls="fs -ls" lzohead () { } hadoop fs -cat $1 lzop -dc head -1000 less export PATH=$PATH:$HADOOP_HOME/bin export JAVA_HOME=/usr/lib/jvm/default-java Configure site Modify core-site.xml
hduser@ubuntu:/usr/local/hadoop/etc/hadoop$ pwd /usr/local/hadoop/etc/hadoop Add following lines:?xml version="1.0" encoding="utf-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <name>hadoop.tmp.dir</name> <value>/hadoopfs/tmp</value> <description>a base for other temporary directories.</description> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> <description>the name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.scheme.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description> <final>true</final> </configuration> Configure DHS Modify hdsf-site.xml. This file indicates information to DFS duser@ubuntu:/usr/local/hadoop/etc/hadoop$ cat hdfs-site.xml
<?xml version="1.0" encoding="utf-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/license-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <name>dfs.replication</name> <value>1</value> <description>default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time. </description> <name>dfs.permissions</name>
<value>false</value> <name>dfs.namenode.name.dir</name> <value>file:/hadoopfs/dfs/name</value> <final>true</final> <name>dfs.datanode.data.dir</name> <value>file:/hadoopfs/dfs/data</value> <final>true</final> </configuration> Configure Map-reduce hduser@ubuntu:/usr/local/hadoop/etc/hadoop$ cat mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/license-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <name>mapreduce.framework.name</name> <value>yarn</value> <name>mapreduce.jobtracker.system.dir</name> <value>file:/hadoopfs/dfs/system</value> <final>true</final> <name>mapreduce.jobtracker.address</name> <value>localhost:9001</value> <description>the host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task. </description>
<name>mapreduce.jobtracker.http.address</name> <value>localhost:50030</value> <name>mapreduce.cluster.local.dir</name> <value>file:/hadoopfs/dfs/local</value> <final>true</final> <name>mapreduce.cluster.temp.dir</name> <value>file:/hadoopfs/dfs/tmp</value> <description>no description</description> <final>true</final> </configuration> Yarn-site.xml <?xml version="1.0"?> <configuration> <!-- Site specific YARN configuration properties -->
<name>yarn.resourcemanager.hostname</name> <value>localhost</value> <description>the host is the hostname of the ResourceManager and the port is the port on which the clients can talk to the Resource Manager. </description> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> <value>org.apache.hadoop.mapred.shufflehandler</value> <description>shuffle service that needs to be set for Map Reduce to run </description> <name>yarn.nodemanager.aux-services</name> <value>mapreduce.shuffle</value> </configuration> Initialize DFS $ hdfs namenode -format In some cases if there is some problem with node and namenode DFS, you need to re-format the DFS $ rm -rf /hadoopfs/dfs/* $ rm -rf /hadoopfs/tmp/* Then re-format namenode DFS. Start and stop the server(s) Should be done one by one namenode, datanode and Yarn(resource manager), but you can use startall.sh and stop-all.sh to enable one time start/stop /usr/local/hadoop/sbin
$ hadoop-daemon.sh start namenode $ hadoop-daemon.sh start datanode Monitor the node(s) General nodes information can be checked by web browser at port 50070 (by default) You can also use jps to see following instances are running hduser@ubuntu:/usr/local/hadoop/sbin$ jps 3135 ResourceManager 2678 DataNode 2428 NameNode 5044 Jps 2961 SecondaryNameNode Run a test Let s try this word count example. Don t try to run the randomwriter example by default if you want waste 10 GB
http://wiki.apache.org/hadoop/randomwriter RandomWriter example writes 10 gig (by default) of random data/host to DFS using Map/Reduce. Try to find some txt such like this cool stuff $ wget http://www.gutenberg.org/files/4300/4300.zip $ unzip 4300.zip hduser@ubuntu:/usr/local/hadoop/share/hadoop/mapreduce$ hadoop dfs -mkdir /tmp hduser@ubuntu:/usr/local/hadoop/share/hadoop/mapreduce$ hadoop dfs -copyfromlocal 4300.txt /tmp Now ready to roll! hduser@ubuntu:/usr/local/hadoop/share/hadoop/mapreduce$ hadoop jar hadoop-mapreduceexamples-2.1.0-beta.jar wordcount /tmp/4300.txt /tmp/output3 14/01/28 20:04:56 WARN conf.configuration: session.id is deprecated. Instead, use dfs.metrics.session-id 14/01/28 20:04:56 INFO jvm.jvmmetrics: Initializing JVM Metrics with processname=jobtracker, sessionid= 14/01/28 20:04:56 INFO input.fileinputformat: Total input paths to process : 1 14/01/28 20:04:56 INFO mapreduce.jobsubmitter: number of splits:1 14/01/28 20:04:56 WARN conf.configuration: user.name is deprecated. Instead, use mapreduce.job.user.name 14/01/28 20:04:56 WARN conf.configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar 14/01/28 20:04:56 WARN conf.configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 14/01/28 20:04:56 WARN conf.configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class 14/01/28 20:04:56 WARN conf.configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 14/01/28 20:04:56 WARN conf.configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name 14/01/28 20:04:56 WARN conf.configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/01/28 20:04:56 WARN conf.configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 14/01/28 20:04:56 WARN conf.configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 14/01/28 20:04:56 WARN conf.configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 14/01/28 20:04:56 WARN conf.configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 14/01/28 20:04:56 WARN conf.configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 14/01/28 20:04:56 INFO mapreduce.jobsubmitter: Submitting tokens for job: job_local513207422_0001 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser513207422/.staging/job_local513207422_0001/job.xml:an attempt to override final parameter: mapreduce.local.dir; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser513207422/.staging/job_local513207422_0001/job.xml:an attempt to override final parameter: dfs.namenode.name.dir; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser513207422/.staging/job_local513207422_0001/job.xml:an attempt to override final parameter: mapreduce.job.endnotification.max.retry.interval; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser513207422/.staging/job_local513207422_0001/job.xml:an attempt to override final parameter: dfs.datanode.data.dir; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser513207422/.staging/job_local513207422_0001/job.xml:an attempt to override final parameter: mapreduce.job.endnotification.max.attempts; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser513207422/.staging/job_local513207422_0001/job.xml:an attempt to override final parameter: fs.defaultfs; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/staging/hduser513207422/.staging/job_local513207422_0001/job.xml:an attempt to override final parameter: mapreduce.system.dir; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local513207422_0001.xml:an attempt to override final parameter: mapreduce.local.dir; Ignoring.
14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local513207422_0001.xml:an attempt to override final parameter: dfs.namenode.name.dir; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local513207422_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local513207422_0001.xml:an attempt to override final parameter: dfs.datanode.data.dir; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local513207422_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local513207422_0001.xml:an attempt to override final parameter: fs.defaultfs; Ignoring. 14/01/28 20:04:56 WARN conf.configuration: file:/hadoopfs/tmp/mapred/local/localrunner/job_local513207422_0001.xml:an attempt to override final parameter: mapreduce.system.dir; Ignoring. 14/01/28 20:04:56 INFO mapreduce.job: The url to track the job: http://localhost:8080/ 14/01/28 20:04:56 INFO mapreduce.job: Running job: job_local513207422_0001 14/01/28 20:04:56 INFO mapred.localjobrunner: OutputCommitter set in config null 14/01/28 20:04:56 INFO mapred.localjobrunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.fileoutputcommitter 14/01/28 20:04:56 INFO mapred.localjobrunner: Waiting for map tasks 14/01/28 20:04:56 INFO mapred.localjobrunner: Starting task: attempt_local513207422_0001_m_000000_0 14/01/28 20:04:56 INFO mapred.task: Using ResourceCalculatorProcessTree : [ ] 14/01/28 20:04:56 INFO mapred.maptask: Processing split: hdfs://localhost:9000/tmp/4300.txt:0+1573078 14/01/28 20:04:57 INFO mapred.maptask: Map output collector class = org.apache.hadoop.mapred.maptask$mapoutputbuffer 14/01/28 20:04:57 INFO mapred.maptask: (EQUATOR) 0 kvi 26214396(104857584) 14/01/28 20:04:57 INFO mapred.maptask: mapreduce.task.io.sort.mb: 100 14/01/28 20:04:57 INFO mapred.maptask: soft limit at 83886080
14/01/28 20:04:57 INFO mapred.maptask: bufstart = 0; bufvoid = 104857600 14/01/28 20:04:57 INFO mapred.maptask: kvstart = 26214396; length = 6553600 14/01/28 20:04:57 INFO mapred.localjobrunner: 14/01/28 20:04:57 INFO mapred.maptask: Starting flush of map output 14/01/28 20:04:57 INFO mapred.maptask: Spilling map output 14/01/28 20:04:57 INFO mapred.maptask: bufstart = 0; bufend = 2601826; bufvoid = 104857600 14/01/28 20:04:57 INFO mapred.maptask: kvstart = 26214396(104857584); kvend = 25142480(100569920); length = 1071917/6553600 14/01/28 20:04:57 INFO mapreduce.job: Job job_local513207422_0001 running in uber mode : false 14/01/28 20:04:57 INFO mapreduce.job: map 0% reduce 0% 14/01/28 20:04:58 INFO mapred.maptask: Finished spill 0 14/01/28 20:04:58 INFO mapred.task: Task:attempt_local513207422_0001_m_000000_0 is done. And is in the process of committing 14/01/28 20:04:58 INFO mapred.localjobrunner: map 14/01/28 20:04:58 INFO mapred.task: Task 'attempt_local513207422_0001_m_000000_0' done. 14/01/28 20:04:58 INFO mapred.localjobrunner: Finishing task: attempt_local513207422_0001_m_000000_0 14/01/28 20:04:58 INFO mapred.localjobrunner: Map task executor complete. 14/01/28 20:04:58 INFO mapred.task: Using ResourceCalculatorProcessTree : [ ] 14/01/28 20:04:58 INFO mapred.merger: Merging 1 sorted segments 14/01/28 20:04:58 INFO mapred.merger: Down to the last merge-pass, with 1 segments left of total size: 725062 bytes 14/01/28 20:04:58 INFO mapred.localjobrunner: 14/01/28 20:04:58 WARN conf.configuration: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 14/01/28 20:04:58 INFO mapreduce.job: map 100% reduce 0% 14/01/28 20:04:58 INFO mapred.task: Task:attempt_local513207422_0001_r_000000_0 is done. And is in the process of committing 14/01/28 20:04:58 INFO mapred.localjobrunner:
14/01/28 20:04:58 INFO mapred.task: Task attempt_local513207422_0001_r_000000_0 is allowed to commit now 14/01/28 20:04:58 INFO output.fileoutputcommitter: Saved output of task 'attempt_local513207422_0001_r_000000_0' to hdfs://localhost:9000/tmp/output3/_temporary/0/task_local513207422_0001_r_000000 14/01/28 20:04:58 INFO mapred.localjobrunner: reduce > reduce 14/01/28 20:04:58 INFO mapred.task: Task 'attempt_local513207422_0001_r_000000_0' done. 14/01/28 20:04:59 INFO mapreduce.job: map 100% reduce 100% 14/01/28 20:04:59 INFO mapreduce.job: Job job_local513207422_0001 completed successfully 14/01/28 20:04:59 INFO mapreduce.job: Counters: 32 File System Counters FILE: Number of bytes read=1268626 FILE: Number of bytes written=2363894 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=3146156 HDFS: Number of bytes written=527555 HDFS: Number of read operations=13 HDFS: Number of large read operations=0 HDFS: Number of write operations=4 Map-Reduce Framework Map input records=33056 Map output records=267980 Map output bytes=2601826 Map output materialized bytes=725074 Input split bytes=99 Combine input records=267980 Combine output records=50095
Reduce input groups=50095 Reduce shuffle bytes=0 Reduce input records=50095 Reduce output records=50095 Spilled Records=100190 Shuffled Maps =0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=37 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=508559360 File Input Format Counters Bytes Read=1573078 File Output Format Counters Bytes Written=527555 hduser@ubuntu:/usr/local/hadoop/share/hadoop/mapreduce$ see the result by hduser@ubuntu:/usr/local/hadoop/share/hadoop/mapreduce$ hadoop dfs -cat /tmp/output3/part* Or from web console
Remove a directory hduser@ubuntu:/usr/local/hadoop/share/hadoop/mapreduce$ hadoop dfs -rmr /tmp/output DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. rmr: DEPRECATED: Please use 'rm -r' instead. 14/01/30 09:26:27 INFO fs.trashpolicydefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /tmp/output hduser@ubuntu:/usr/local/hadoop/share/hadoop/mapreduce$ hadoop dfs -rmr /tmp/output2 DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it.
rmr: DEPRECATED: Please use 'rm -r' instead. 14/01/30 09:26:56 INFO fs.trashpolicydefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /tmp/output2 Install Hadoop as Master + Slave on RHEL 6.5 General steps 1. Preparation Master Slave Hostname master.hadoop.advol node.hadoop.advol edit /etc/hosts JDK /usr/java/jdk-1.7.0 As master Download rpm from oracle $HADOOP_HOME /data/hadoop/hadoop- 2.4.0 As master Can use separated logic volume as hdfs user hadoop hadoop Adduser hadoop Also grant owner of $HADOOP_HOME to hadoop Hardware Standalone hardware, RHEL 6.5 RHEL 6.5 on VMWARE on my laptop 2. DO THE SAME ON MASTER NODE AND SLAVE NODES TO INSTALL JDK and ENVIRONMENT CONFIGURATION 3. MODIFY HOSTNAME TO ENABLE PING BY HOSTNAME TO EACH OTHER 4. MODIFY FIREWALL IPTABLES 5. ADD SSH PUBLIC KEY FROM MASTER TO SLAVE AS WELL AS FROM SLAVE TO MASTER 6. KEEP DIRECTORY BE THE SAME 7. MODIFY SLAVE FILE IN $HADOOPHOME/ETC 8. FORMAT NAMENODE AGAIN 9. START-ALL FROM MASTER Hadoop-env.sh Add JAVA_HOME # The java implementation to use. export JAVA_HOME=/usr/java/jdk1.7.0_55
core-site.xml <configuration> <name>hadoop.tmp.dir</name> <value>/data/hadoop/hadoopfs/tmp</value> <description>a base for other temporary directories.</description> <name>fs.default.name</name> <value>hdfs://master.hadoop.advol:9000</value> <final>true</final> </configuration> hdfs-site.xml <configuration> <name>dfs.replication</name> <value>2</value> <name>dfs.permissions</name> <value>false</value> <name>dfs.namenode.name.dir</name> <value>file:/data/hadoop/hadoopfs/dfs/name</value>
<final>true</final> <name>dfs.datanode.data.dir</name> <value>file:/data/hadoop/hadoopfs/dfs/data</value> <final>true</final> <name>dfs.datanode.max.xcievers</name> <value>4096</value> </configuration> yarn-site.xml <configuration> <name>dfs.replication</name> <value>2</value> <name>dfs.permissions</name> <value>false</value> <name>dfs.namenode.name.dir</name> <value>file:/data/hadoop/hadoopfs/dfs/name</value> <final>true</final> <name>dfs.datanode.data.dir</name>
<value>file:/data/hadoop/hadoopfs/dfs/data</value> <final>true</final> <name>dfs.datanode.max.xcievers</name> <value>4096</value> </configuration> mapred-site.xml <configuration> <name>mapreduce.framework.name</name> <value>yarn</value> <name>mapreduce.cluster.local.dir</name> <value>file:/data/hadoop/hadoopfs/dfs/local</value> <final>true</final> <name>mapreduce.cluster.temp.dir</name> <value>file:/data/hadoop/hadoopfs/dfs/tmp</value> <description>no description</description> <final>true</final> </configuration>
After start (~/$HADOOP_HOME/sbin/start-all.sh ), use JPS to check JAVA threads, at Master: [hadoop@master sbin]$ jps 15673 ResourceManager 15782 NodeManager 16101 Jps 15519 SecondaryNameNode 15145 NameNode [hadoop@master sbin]$ JPS at SLAVE [hadoop@node ~]$ jps 3934 Jps 3480 DataNode 3583 NodeManager [hadoop@node ~]$ If have issue with copyfromlocal (only one node accept, while another raise a I/O issue), check firewall Check hdfs [hadoop@master hadoop]$ hdfs dfsadmin -report Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/05/09 17:38:44 WARN util.nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Configured Capacity: 180645183488 (168.24 GB) Present Capacity: 159183417344 (148.25 GB) DFS Remaining: 159180152832 (148.25 GB) DFS Used: 3264512 (3.11 MB) DFS Used%: 0.00%
Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 ------------------------------------------------- Datanodes available: 2 (2 total, 0 dead) Live datanodes: Name: 130.113.68.191:50010 (node.hadoop.advol) Hostname: node.hadoop.advol Decommission Status : Normal Configured Capacity: 18713219072 (17.43 GB) DFS Used: 28672 (28 KB) Non DFS Used: 4541681664 (4.23 GB) DFS Remaining: 14171508736 (13.20 GB) DFS Used%: 0.00% DFS Remaining%: 75.73% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Last contact: Fri May 09 17:38:43 EDT 2014 Name: 130.113.68.234:50010 (master.hadoop.advol) Hostname: master.hadoop.advol Decommission Status : Normal Configured Capacity: 161931964416 (150.81 GB) DFS Used: 3235840 (3.09 MB)
Non DFS Used: 16920084480 (15.76 GB) DFS Remaining: 145008644096 (135.05 GB) DFS Used%: 0.00% DFS Remaining%: 89.55% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Last contact: Fri May 09 17:38:43 EDT 2014 [hadoop@master hadoop]$ Run word count on 1 master + 2 slave nodes Can also check application job tracker at master server http://130.113.68.234:8088/cluster localhost not accepted [hadoop@master ~]$ hadoop jar /data/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/hadoopmapreduce-examples-2.4.0.jar wordcount /tmp/4300.txt /tmp/output4 Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/05/12 13:04:13 WARN util.nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/05/12 13:04:14 INFO client.rmproxy: Connecting to ResourceManager at master.hadoop.advol/130.113.68.234:8032 14/05/12 13:04:15 INFO input.fileinputformat: Total input paths to process : 1 14/05/12 13:04:15 INFO mapreduce.jobsubmitter: number of splits:1
14/05/12 13:04:15 INFO mapreduce.jobsubmitter: Submitting tokens for job: job_1399671198293_0001 14/05/12 13:04:16 INFO impl.yarnclientimpl: Submitted application application_1399671198293_0001 14/05/12 13:04:16 INFO mapreduce.job: The url to track the job: http://master.hadoop.advol:8088/proxy/application_1399671198293_0001/ 14/05/12 13:04:16 INFO mapreduce.job: Running job: job_1399671198293_0001 14/05/12 13:04:26 INFO mapreduce.job: Job job_1399671198293_0001 running in uber mode : false 14/05/12 13:04:26 INFO mapreduce.job: map 0% reduce 0% 14/05/12 13:04:35 INFO mapreduce.job: map 100% reduce 0% 14/05/12 13:04:41 INFO mapreduce.job: map 100% reduce 100% 14/05/12 13:04:41 INFO mapreduce.job: Job job_1399671198293_0001 completed successfully 14/05/12 13:04:42 INFO mapreduce.job: Counters: 49 File System Counters FILE: Number of bytes read=725074 FILE: Number of bytes written=1635839 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=1573187 HDFS: Number of bytes written=527555 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=6084
Total time spent by all reduces in occupied slots (ms)=3660 Total time spent by all map tasks (ms)=6084 Total time spent by all reduce tasks (ms)=3660 Total vcore-seconds taken by all map tasks=6084 Total vcore-seconds taken by all reduce tasks=3660 Total megabyte-seconds taken by all map tasks=6230016 Total megabyte-seconds taken by all reduce tasks=3747840 Map-Reduce Framework Map input records=33056 Map output records=267980 Map output bytes=2601826 Map output materialized bytes=725074 Input split bytes=109 Combine input records=267980 Combine output records=50095 Reduce input groups=50095 Reduce shuffle bytes=725074 Reduce input records=50095 Reduce output records=50095 Spilled Records=100190 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=286 CPU time spent (ms)=4990 Physical memory (bytes) snapshot=436609024 Virtual memory (bytes) snapshot=1763209216 Total committed heap usage (bytes)=274726912 Shuffle Errors BAD_ID=0
CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=1573078 File Output Format Counters Bytes Written=527555 [hadoop@master ~]$ Run word count on master+slave (1 node) [hadoop@master sbin]$ hadoop jar /data/hadoop/hadoop- 2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar wordcount /tmp/4300.txt /tmp/output5 Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/05/12 13:29:24 WARN util.nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/05/12 13:29:25 INFO client.rmproxy: Connecting to ResourceManager at master.hadoop.advol/130.113.68.234:8032 14/05/12 13:29:26 INFO input.fileinputformat: Total input paths to process : 1 14/05/12 13:29:26 INFO mapreduce.jobsubmitter: number of splits:1 14/05/12 13:29:26 INFO mapreduce.jobsubmitter: Submitting tokens for job: job_1399915743300_0001 14/05/12 13:29:27 INFO impl.yarnclientimpl: Submitted application application_1399915743300_0001
14/05/12 13:29:27 INFO mapreduce.job: The url to track the job: http://master.hadoop.advol:8088/proxy/application_1399915743300_0001/ 14/05/12 13:29:27 INFO mapreduce.job: Running job: job_1399915743300_0001 14/05/12 13:29:34 INFO mapreduce.job: Job job_1399915743300_0001 running in uber mode : false 14/05/12 13:29:34 INFO mapreduce.job: map 0% reduce 0% 14/05/12 13:29:41 INFO mapreduce.job: map 100% reduce 0% 14/05/12 13:29:47 INFO mapreduce.job: map 100% reduce 100% 14/05/12 13:29:48 INFO mapreduce.job: Job job_1399915743300_0001 completed successfully 14/05/12 13:29:48 INFO mapreduce.job: Counters: 49 File System Counters FILE: Number of bytes read=725074 FILE: Number of bytes written=1635839 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=1573187 HDFS: Number of bytes written=527555 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=4254 Total time spent by all reduces in occupied slots (ms)=3904 Total time spent by all map tasks (ms)=4254 Total time spent by all reduce tasks (ms)=3904 Total vcore-seconds taken by all map tasks=4254
Total vcore-seconds taken by all reduce tasks=3904 Total megabyte-seconds taken by all map tasks=4356096 Total megabyte-seconds taken by all reduce tasks=3997696 Map-Reduce Framework Map input records=33056 Map output records=267980 Map output bytes=2601826 Map output materialized bytes=725074 Input split bytes=109 Combine input records=267980 Combine output records=50095 Reduce input groups=50095 Reduce shuffle bytes=725074 Reduce input records=50095 Reduce output records=50095 Spilled Records=100190 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=119 CPU time spent (ms)=5290 Physical memory (bytes) snapshot=442953728 Virtual memory (bytes) snapshot=1764282368 Total committed heap usage (bytes)=305135616 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0
WRONG_REDUCE=0 File Input Format Counters Bytes Read=1573078 File Output Format Counters Bytes Written=527555 [hadoop@master sbin]$ Observation: I don t see any improvement on the result of running on standalone or master+slave Run word count on bible on standalone and cluster Try to work on a bigger file [hadoop@master hadoop]$ wget http://printkjv.ifbweb.com/av_txt.zip [hadoop@master hadoop]$ unzip AV_txt.zip [hadoop@master hadoop]$ mv AV1611Bible.txt bible.txt [hadoop@master hadoop]$ hdfs dfs -copyfromlocal bible.txt /tmp/bible.txt [hadoop@master hadoop]$ hadoop jar /data/hadoop/hadoop- 2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar wordcount /tmp/bible.txt /tmp/bible/ [hadoop@master hadoop]$ hdfs dfs -getmerge /tmp/bible/./bible.result [hadoop@master hadoop]$ cat bible.result grep god Gudgodah 1 Gudgodah; 1 [god] 1 [god]: 1 [gods 1 [gods], 2 ergodiwkthv 1 god 20 god, 19 god. 9 god: 4
god; 3 goddess 4 goddess. 1 godliness 4 godliness) 1 godliness, 4 godliness. 1 godliness: 2 godliness; 3 godly 16 godly, 1 godly-learned 1 gods 93 gods, 88 gods. 28 gods: 15 gods; 8 gods? 7 ungodliness 3 ungodliness. 1 ungodly 17 ungodly, 5 ungodly. 2 ungodly; 2 ungodly? 1
Run it on Master + 2 slave nodes [hadoop@master sbin]$./start-all.sh [hadoop@master sbin]$ jps 853 SecondaryNameNode 598 DataNode 1009 ResourceManager 466 NameNode 1452 Jps 1120 NodeManager [hadoop@master sbin]$ hadoop jar /data/hadoop/hadoop- 2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar wordcount /tmp/bible.txt /tmp/bible2/ Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/05/12 14:01:50 WARN util.nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/05/12 14:01:51 INFO client.rmproxy: Connecting to ResourceManager at master.hadoop.advol/130.113.68.234:8032 14/05/12 14:01:52 INFO input.fileinputformat: Total input paths to process : 1 14/05/12 14:01:52 INFO mapreduce.jobsubmitter: number of splits:1 14/05/12 14:01:52 INFO mapreduce.jobsubmitter: Submitting tokens for job: job_1399917682766_0001 14/05/12 14:01:52 INFO impl.yarnclientimpl: Submitted application application_1399917682766_0001 14/05/12 14:01:52 INFO mapreduce.job: The url to track the job: http://master.hadoop.advol:8088/proxy/application_1399917682766_0001/ 14/05/12 14:01:53 INFO mapreduce.job: Running job: job_1399917682766_0001 14/05/12 14:02:00 INFO mapreduce.job: Job job_1399917682766_0001 running in uber mode : false
14/05/12 14:02:00 INFO mapreduce.job: map 0% reduce 0% 14/05/12 14:02:12 INFO mapreduce.job: map 100% reduce 0% 14/05/12 14:02:21 INFO mapreduce.job: map 100% reduce 100% 14/05/12 14:02:22 INFO mapreduce.job: Job job_1399917682766_0001 completed successfully 14/05/12 14:02:22 INFO mapreduce.job: Counters: 49 File System Counters FILE: Number of bytes read=467766 FILE: Number of bytes written=1121223 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=4397316 HDFS: Number of bytes written=343441 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=10832 Total time spent by all reduces in occupied slots (ms)=5080 Total time spent by all map tasks (ms)=10832 Total time spent by all reduce tasks (ms)=5080 Total vcore-seconds taken by all map tasks=10832 Total vcore-seconds taken by all reduce tasks=5080 Total megabyte-seconds taken by all map tasks=11091968 Total megabyte-seconds taken by all reduce tasks=5201920 Map-Reduce Framework
Map input records=33979 Map output records=840245 Map output bytes=7722868 Map output materialized bytes=467766 Input split bytes=110 Combine input records=840245 Combine output records=32635 Reduce input groups=32635 Reduce shuffle bytes=467766 Reduce input records=32635 Reduce output records=32635 Spilled Records=65270 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=760 CPU time spent (ms)=7800 Physical memory (bytes) snapshot=445231104 Virtual memory (bytes) snapshot=1754845184 Total committed heap usage (bytes)=308281344 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=4397206 File Output Format Counters
Bytes Written=343441 [hadoop@master sbin]$ hdfs dfs -getmerge /tmp/bible/./bible2.result Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/05/12 14:07:39 WARN util.nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [hadoop@master sbin]$ cat bible2.result grep god Gudgodah 1 Gudgodah; 1 [god] 1 [god]: 1 [gods 1 [gods], 2 ergodiwkthv 1 god 20 god, 19 god. 9 god: 4 god; 3 goddess 4 goddess. 1 godliness 4 godliness) 1 godliness, 4 godliness. 1 godliness: 2 godliness; 3 godly 16
godly, 1 godly-learned 1 gods 93 gods, 88 gods. 28 gods: 15 gods; 8 gods? 7 ungodliness 3 ungodliness. 1 ungodly 17 ungodly, 5 ungodly. 2 ungodly; 2 ungodly? 1 [hadoop@master sbin]$ Run word count on my emails master + 2 slave nodes Try to run another bigger file Myemails.txt is a export from my outlook which includes 3 years email [hadoop@master hadoop]$ hadoop jar /data/hadoop/hadoop- 2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar wordcount /tmp/myemails.txt /tmp/myemail/ Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/05/12 14:54:45 WARN util.nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/05/12 14:54:46 INFO client.rmproxy: Connecting to ResourceManager at master.hadoop.advol/130.113.68.234:8032 14/05/12 14:54:46 INFO input.fileinputformat: Total input paths to process : 1 14/05/12 14:54:46 INFO mapreduce.jobsubmitter: number of splits:1 14/05/12 14:54:47 INFO mapreduce.jobsubmitter: Submitting tokens for job: job_1399917682766_0002 14/05/12 14:54:47 INFO impl.yarnclientimpl: Submitted application application_1399917682766_0002 14/05/12 14:54:47 INFO mapreduce.job: The url to track the job: http://master.hadoop.advol:8088/proxy/application_1399917682766_0002/ 14/05/12 14:54:47 INFO mapreduce.job: Running job: job_1399917682766_0002 14/05/12 14:54:54 INFO mapreduce.job: Job job_1399917682766_0002 running in uber mode : false 14/05/12 14:54:54 INFO mapreduce.job: map 0% reduce 0% 14/05/12 14:55:10 INFO mapreduce.job: map 21% reduce 0% 14/05/12 14:55:13 INFO mapreduce.job: map 24% reduce 0% 14/05/12 14:55:16 INFO mapreduce.job: map 43% reduce 0% 14/05/12 14:55:19 INFO mapreduce.job: map 47% reduce 0% 14/05/12 14:55:22 INFO mapreduce.job: map 62% reduce 0% 14/05/12 14:55:25 INFO mapreduce.job: map 67% reduce 0% 14/05/12 14:55:26 INFO mapreduce.job: map 100% reduce 0% 14/05/12 14:55:33 INFO mapreduce.job: map 100% reduce 100% 14/05/12 14:55:33 INFO mapreduce.job: Job job_1399917682766_0002 completed successfully 14/05/12 14:55:33 INFO mapreduce.job: Counters: 49 File System Counters FILE: Number of bytes read=15513937 FILE: Number of bytes written=22690574
FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=86012550 HDFS: Number of bytes written=6281696 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=29563 Total time spent by all reduces in occupied slots (ms)=4477 Total time spent by all map tasks (ms)=29563 Total time spent by all reduce tasks (ms)=4477 Total vcore-seconds taken by all map tasks=29563 Total vcore-seconds taken by all reduce tasks=4477 Total megabyte-seconds taken by all map tasks=30272512 Total megabyte-seconds taken by all reduce tasks=4584448 Map-Reduce Framework Map input records=3597155 Map output records=10216704 Map output bytes=125299964 Map output materialized bytes=6990938 Input split bytes=113 Combine input records=10489013 Combine output records=456869 Reduce input groups=184560 Reduce shuffle bytes=6990938
Reduce input records=184560 Reduce output records=184560 Spilled Records=641429 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=604 CPU time spent (ms)=32020 Physical memory (bytes) snapshot=457576448 Virtual memory (bytes) snapshot=1773580288 Total committed heap usage (bytes)=304611328 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=86012437 File Output Format Counters Bytes Written=6281696 [hadoop@master hadoop]$
Start history server Not start by default [hadoop@master sbin]$./mr-jobhistory-daemon.sh start historyserver starting historyserver, logging to /data/hadoop/hadoop-2.4.0/logs/mapred-hadoophistoryserver-master.hadoop.advol.out [hadoop@master sbin]$ Install Hbase Understand what is Hadoop Hbase Hive Pig hive 与 hbase 区 别 1 hive 是 sql 语 言, 通 过 数 据 库 的 方 式 来 操 作 hdfs 文 件 系 统, 为 了 简 化 编 程, 底 层 计 算 方 式 为 mapreduce 2 hive 是 面 向 行 存 储 的 数 据 库 3 Hive 本 身 不 存 储 和 计 算 数 据, 它 完 全 依 赖 于 HDFS 和 MapReduce,Hive 中 的 表 纯 逻 辑 4 HBase 为 查 询 而 生 的, 它 通 过 组 织 起 节 点 內 所 有 机 器 的 內 存, 提 供 一 個 超 大 的 內 存 Hash 表 5 hbase 不 是 关 系 型 数 据 库, 而 是 一 个 在 hdfs 上 开 发 的 面 向 列 的 分 布 式 数 据 库, 不 支 持 sql 6 hbase 是 物 理 表, 不 是 逻 辑 表, 提 供 一 个 超 大 的 内 存 hash 表, 搜 索 引 擎 通 过 它 来 存 储 索 引, 方 便 查 询 操 作
7 hbase 是 列 存 储 Pig Pig 是 一 种 数 据 流 语 言, 用 来 快 速 轻 松 的 处 理 巨 大 的 数 据 Pig 包 含 两 个 部 分 :Pig Interface,Pig Latin Pig 可 以 非 常 方 便 的 处 理 HDFS 和 HBase 的 数 据, 和 Hive 一 样,Pig 可 以 非 常 高 效 的 处 理 其 需 要 做 的, 通 过 直 接 操 作 Pig 查 询 可 以 节 省 大 量 的 劳 动 和 时 间 当 你 想 在 你 的 数 据 上 做 一 些 转 换, 并 且 不 想 编 写 MapReduce jobs 就 可 以 用 Pig. Hive 起 源 于 FaceBook,Hive 在 Hadoop 中 扮 演 数 据 仓 库 的 角 色 建 立 在 Hadoop 集 群 的 最 顶 层, 对 存 储 在 Hadoop 群 上 的 数 据 提 供 类 SQL 的 接 口 进 行 操 作 你 可 以 用 HiveQL 进 行 select,join, 等 等 操 作 如 果 你 有 数 据 仓 库 的 需 求 并 且 你 擅 长 写 SQL 并 且 不 想 写 MapReduce jobs 就 可 以 用 Hive 代 替 HBase HBase 作 为 面 向 列 的 数 据 库 运 行 在 HDFS 之 上,HDFS 缺 乏 随 即 读 写 操 作,HBase 正 是 为 此 而 出 现 HBase 以 Google BigTable 为 蓝 本, 以 键 值 对 的 形 式 存 储 项 目 的 目 标 就 是 快 速 在 主 机 内 数 十 亿 行 数 据 中 定 位 所 需 的 数 据 并 访 问 它 HBase 是 一 个 数 据 库, 一 个 NoSql 的 数 据 库, 像 其 他 数 据 库 一 样 提 供 随 即 读 写 功 能,Hadoop 不 能 满 足 实 时 需 要,HBase 正 可 以 满 足 如 果 你 需 要 实 时 访 问 一 些 数 据, 就 把 它 存 入 HBase 你 可 以 用 Hadoop 作 为 静 态 数 据 仓 库,HBase 作 为 数 据 存 储, 放 那 些 进 行 一 些 操 作 会 改 变 的 数 据 Pig VS Hive Hive 更 适 合 于 数 据 仓 库 的 任 务,Hive 主 要 用 于 静 态 的 结 构 以 及 需 要 经 常 分 析 的 工 作 Hive 与 SQL 相 似 促 使 其 成 为 Hadoop 与 其 他 BI 工 具 结 合 的 理 想 交 集 Pig 赋 予 开 发 人 员 在 大 数 据 集 领 域 更 多 的 灵 活 性, 并 允 许 开 发 简 洁 的 脚 本 用 于 转 换 数 据 流 以 便 嵌 入 到 较 大 的 应 用 程 序 Pig 相 比 Hive 相 对 轻 量, 它 主 要 的 优 势 是 相 比 于 直 接 使 用 Hadoop Java APIs 可 大 幅 削 减 代 码 量 正 因 为 如 此,Pig 仍 然 是 吸 引 大 量 的 软 件 开 发 人 员 Hive 和 Pig 都 可 以 与 HBase 组 合 使 用,Hive 和 Pig 还 为 HBase 提 供 了 高 层 语 言 支 持, 使 得 在 HBase 上 进 行 数 据 统 计 处 理 变 的 非 常 简 单 Hive VS HBase Hive 是 建 立 在 Hadoop 之 上 为 了 减 少 MapReduce jobs 编 写 工 作 的 批 处 理 系 统,HBase 是 为 了 支 持 弥 补 Hadoop 对 实 时 操 作 的 缺 陷 的 项 目 想 象 你 在 操 作 RMDB 数 据 库, 如 果 是 全 表 扫 描, 就 用 Hive+Hadoop, 如 果 是 索 引 访 问, 就 用 HBase+Hadoop Hive query 就 是 MapReduce jobs 可 以 从 5 分 钟 到 数 小 时 不 止,HBase 是 非 常 高 效 的, 肯 定 比 Hive 高 效 的 多 If see:
org.apache.hadoop.ipc.remoteexception: Server IPC version 9 cannot communicate with client version 4 Check Hbase version and Hadoop version [hadoop@master download]$ wget http://mirror.csclub.uwaterloo.ca/apache/hbase/hbase- 0.98.2/hbase-0.98.2-hadoop2-bin.tar.gz --2014-05-13 18:54:18-- http://mirror.csclub.uwaterloo.ca/apache/hbase/hbase- 0.98.2/hbase-0.98.2-hadoop2-bin.tar.gz Resolving mirror.csclub.uwaterloo.ca... 129.97.134.71 Connecting to mirror.csclub.uwaterloo.ca 129.97.134.71 :80... connected. HTTP request sent, awaiting response... 200 OK Length: 82246622 (78M) [application/x-gzip] Saving to: 鈥 渉 base-0.98.2-hadoop2-bin.tar.gz 鈥? 100%[==================================================================>] 82,246,622 20.5M/s in 3.9s 2014-05-13 18:54:22 (20.0 MB/s) - 鈥 渉 base-0.98.2-hadoop2-bin.tar.gz 鈥?saved [82246622/82246622] [hadoop@master download]$ tar xfz hbase-0.98.2-hadoop2-bin.tar.gz -C /data/hbase/ Download and modify [hadoop@master hbase-0.98.2-hadoop2]$ cd conf [hadoop@master conf]$ ls hadoop-metrics2-hbase.properties hbase-env.sh hbase-site.xml regionservers hbase-env.cmd hbase-policy.xml log4j.properties [hadoop@master conf]$ more hbase-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration>
<name>hbase.cluster.distributed</name> <value>true</value> <name>hbase.rootdir</name> <value>hdfs://master.hadoop.advol:9000/hbase</value> </configuration> Create a table card and add a record [hadoop@master bin]$./hbase shell hbase(main):019:0> create 'card','mycf' 0 row(s) in 0.7670 seconds => Hbase::Table - card hbase(main):020:0> list 'card' TABLE card 1 row(s) in 0.0360 seconds => ["card"] hbase(main):021:0> put 'card','rowid1','mycf:a','sword of Truth' 0 row(s) in 0.1260 seconds hbase(main):022:0> list 'card' TABLE card 1 row(s) in 0.0330 seconds => ["card"]
hbase(main):024:0> scan 'card' ROW rowid1 Truth COLUMN+CELL column=mycf:a, timestamp=1400090875012, value=sword of 1 row(s) in 0.0400 seconds hbase(main):025:0> [hadoop@master bin]$ [hadoop@master bin]$ [hadoop@master bin]$./stop-hbase.sh stopping hbase... localhost: stopping zookeeper. [hadoop@master bin]$ jps 8707 NameNode 9248 ResourceManager 9092 SecondaryNameNode 11217 Jps 8837 DataNode 9359 NodeManager 5532 JobHistoryServer [hadoop@master bin]$./start-hbase.sh localhost: starting zookeeper, logging to /data/hbase/hbase-0.98.2- hadoop2/bin/../logs/hbase-hadoop-zookeeper-master.hadoop.advol.out starting master, logging to /data/hbase/hbase-0.98.2-hadoop2/bin/../logs/hbase-hadoopmaster-master.hadoop.advol.out localhost: starting regionserver, logging to /data/hbase/hbase-0.98.2- hadoop2/bin/../logs/hbase-hadoop-regionserver-master.hadoop.advol.out [hadoop@master bin]$./hbase shell 2014-05-14 14:10:51,394 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available HBase Shell; enter 'help<return>' for list of supported commands.
Type "exit<return>" to leave the HBase Shell Version 0.98.2-hadoop2, r1591526, Wed Apr 30 20:17:33 PDT 2014 hbase(main):001:0> list TABLE SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/hbase/hbase-0.98.2-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/staticloggerbinder.class] SLF4J: Found binding in [jar:file:/data/hadoop/hadoop- 2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop/hadoop-2.4.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 2014-05-14 14:10:54,583 WARN [main] util.nativecodeloader: Unable to load nativehadoop library for your platform... using builtin-java classes where applicable card 1 row(s) in 1.4430 seconds => ["card"] hbase(main):003:0> scan 'card' ROW rowid1 Truth COLUMN+CELL column=mycf:a, timestamp=1400090875012, value=sword of 1 row(s) in 0.1200 seconds hbase(main):004:0> Also can find a file was created in hdfs