Programing Map-Reduce. ( Hadoop ) with Eclipse

Transcription

1 Programing Map-Reduce ( Hadoop ) with + Wei - Yu Chen NCHC 2008 / 05 / 2 7 see m ore : htt p: / / t rac.nchc.org.tw /clou d /

2 1. Prepare : System : Ubunt u 7.10 Hadoop 0.16 Requirement : (3.2.2) $ apt - get install eclipse java 6 $ apt - get install sun - java6 - bin sun - java6 - jdk s un - java6 - jre s un - java6 - pl ugin s uggest to re move the default java compiler gcj $ apt - get p urge java - gcj - com pat Appen d two codes to / e tc / bas h.bashrc to setup Java Class path export JAVA_HOME= / u sr /lib /jvm / java s un export HADOOP_HOME= / h o me / waue /works pace / hadoop / export CLASSPATH =.:$JAVA_HOME/lib / d t.jar:$java_home/lib / tools.jar Building UP Path Nam e Path Hadoop Home / h o me /waue /works pace / hadoop / Java Home / u s r /lib /jvm / java sun

3 2. Hadoop Setup 1. Generate an SSH key for the user. $ ssh - keygen - t rsa - P "" $ cat ~ /.ssh /id_rsa.pub > > ~.ss h / a uthorized_keys $ ssh localhost $ exit 2. Installation Hadoop $ cd / h o me / wa ue / works pace $ s u do tar xzf ha doop tar.gz $ s u do mv hadoop ha doo p $ s u do chown - R waue:waue ha doop $ cd ha doop 3. Configura tio n 1. hadoop - env.sh ($HADOOP_HOME/conf /) Cha nge # The java imple mentation to use. Required. # export JAVA_HOME= / u sr /lib /j2sdk1.5 - sun to # The java imple mentation to use. Required. export JAVA_HOME= / u sr /lib /jvm /java sun export HADOOP_HOME= / h o me / wa ue / workspace / ha doop exp or t HADOOP_LOG_DIR = $HADOOP_HOME/logs export HADOOP_SLAVES=$HADOOP_HOME/conf / slaves 2. hadoop - site.xml ($HADOOP_HOME/conf /) m o dify the contents of conf / ha doop - site.xml as below < c onfiguration > < p ro perty > < n a me >fs.default.na me < / n a me > < value > localhost:9000< / value > < d e scription > < / d escription > < / p r o perty > < p ro perty > < n a me > m a pred.job.tracker < / na me > < value >localhost:9001 < /value > < d e scription > < / d escription > < p ro perty >

4 < n a me > m a pred.map.tasks < / n a me > < value > 1 < / value > < d e scription > define m a pred.ma p tasks to be n u m ber of slave hosts < / d escription > < / p r o perty > < p ro perty > < n a me > m a pred.reduce.tasks < / n a me > < value > 1 < / value > < d e scription > define m a pred.reduce tasks to be nu m ber of slave hosts < / d escription > < / p r o perty > < p ro perty > < n a me > dfs.replication < / n a me > < value > 1 < / value > < / p r o perty > < / c o nfiguration > 4. Start Up Hadoop $ cd $HADOOP_HOME $ bin / hadoop na menode - for mat 08 / 0 5 / :52:16 INFO dfs.na menode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = Dx7200 / STARTUP_MSG: args = [-for mat] STARTUP_MSG: ver sio n = STARTUP_MSG: build = http: / / svn.apache.org / re pos / a sf / ha doop /core / branches / branch r ; compiled by 'ha doopqa' on Fri May 2 00:18:12 UTC 2008 ************************************************************/ 08 / 0 5 / :52:17 INFO fs.fsnamesyste m: fsowner = waue,waue,ad m,dialout,cdro m,floppy,audio,dip,video,plugdev,staff,scanner,lpa d min,a d min,netdev,power dev,vboxusers 08 / 0 5 / :52:17 INFO fs.fsnamesyste m: supergroup = s u pergroup 08 / 0 5 / :52:17 INFO fs.fsnamesyste m: ispermissionenabled = true 08 / 0 5 / :52:17 INFO dfs.storage: Storage directory / t m p / h a doo p - waue / dfs / n a me has been successfully for matte d. 08 / 0 5 / :52:17 INFO dfs.na menode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at Dx7200 / ************************************************************/ $ / bin / s tart - all.sh starting na menode, logging to / h o me / wa ue / workspace / ha doop /logs / ha doo p - waue - na menode - Dx7200.out localhost: starting data node, logging to / h o me / wa ue /workspace / ha doop /logs / h a doo p - waue - data node - Dx7200.out localhost: starting secon daryna meno de, logging to / h o me / wa ue /works pace / h a doo p /logs / h a doop - waue - secon daryna meno de - Dx7200.out starting jobtracker, logging to / h o me / waue /works pace / h a doo p /logs / h a doop - waue - jobtracker - Dx7200.out localhost: starting tasktracker, logging to / h o me / waue /workspace / ha doo p /logs / ha doop - waue - tasktracker - Dx7200.out

5 Then m ake s ure htt p: / /localhost:50030 / by your explorer is on going. Ps : if your syste m ha d error after restart, you could do there for resolving and renewing one. $ cd $HADOOP_HOME $ bin / s to p - all.sh $ r m - rf / t m p /* $ r m - rf logs /* And repeat to 4. start u p Hadoop

6 3. Setup 3.1 install IBM mapreduce tool 1. Download the IBM MapReduce Tools zip file an d extract to / t m p /. 2. Make sure is closed and... $ cd / t m p / $ u n zip m a preduce_tools.zip $ mv plugins /co m.ibm.hipods.mapreduce* / u s r /lib /eclipse / plugins / 3. Restart Check IBM MapReduce Tools plugin installing well File > New > Project see MapReduce category 3.2 configure Window > Preferences > java > compiler set compiler compliance level to 5.0 Some eclipse - plugin m ay exhaust m uch resource, you m ay happe n to out of me m ory error. We suggest to execute eclipse with so me para meters as that : $ eclipse - v margs - Xmx 512 m

7 4. Run on 4.1 map-reduce sample code File > ne w > project > map - reduce project > next > project name : sample use default location : V use default Hadoop : V > Finish at Project explorer, y ou will se e sample tree. Now, y ou should create a sample code. right click sample > new > file > file na m e : WordCount.java the sa m ple code is here http: / / trac.nchc.org.tw / cloud /attachment / wiki / hadoop - sample - code /WordCount.java paste the contents to your new adding file WordCount.java 4.2. Connect to Hadoop File System Enable the MapReduce servers window Window > Show View > Other... > MapReduce Tools > MapReduce Servers At the botto m of your window, you s hould have a "MapReduce Servers" tab. If not, see second bullet above. Switch to that tab. At the top right edge of the tab, you should see a little blue elephant icons. Click blue elephant to add a new MapReduce server location. Server name : any_you_want Hostname : localhost Installation directory: / h o me / waue / w orkspace / nutch /

8 Username : waue If any password pro m pt, please input the pass w ord which you login to local It should show u p un der a little elephant icon in the Project Explorer (on the left side of ). ps : Pleast m ake sure your Hadoop is working on local syste m. If not, please refer session 2 Hadoop Setup for debuging, or you can not pass through. $ cd / h o me / wa ue / works pace / hadoop / $ wget htt p: / / /etext / 132 / 132.txt $ bin / hadoop dfs - m k dir input $ bin / hadoop dfs - ls Foun d 1 ite ms / u ser / wa ue /in p ut < dir > :15 rwxr - xr - x waue s u pergroup $ bin / hadoop dfs - p ut 132.txt input 4.3 Run sample > right click WordCount.java > run as... > run on Hadoop > choose an existing s erver from the list below > finish A console tag will show beside MapReduce Server tag.

9 While Map Reduce is running, you can visit htt p: / /localhost:50030 / to view that Hadoop is dis patching jobs by Map Reduce. After finish, you can go to htt p: / /localhost:50060 / to see the result.

10

11 5. Reference NCHC Cloud Technique Develop Group htt p: / / t rac.nchc.org.tw /cloud / IBM Map - Reduce htt p: / / / tech / m a preducetools Cloud9 htt p: / / miacs.u m d.edu / ~ ji m mylin /cloud9 / u m d - ha doop - dist /cloud9 - docs / howto / s tart.ht ml Runing Hadoop htt p: / / - noll.com / wiki /Running_Hadoop_On_Ubuntu_Linux_%28Single - Node_Cluster%29 Related Files : Hadoop htt p: / / a pache.ntu.edu.tw / hadoop /core / IBM m a p reduce tool : htt p: / / / tech / m a preducetools word sa m ple 1 : The Art of War by 6th cent. B.C. Sunzi htt p: / / /etext / 132 word sa m ple 2 : The Adventures of Sherlock Holmes by Sir Arthur Conan Doyle htt p: / / /etext / 1661