SettingupanAmazonWebServicesAccount DianaMacleanforCS448G2011 Firstofall,whatdoweneed? Forthisassignment,we llberunningmapreduceworkflowsonamazon selastic MapReduceframework.You llneedtodoacoupleofthingstogetsetup: 1. CreateanAWSaccount a. SignupforS3(thisiswhereyou llstoreyourdataonthecloud)& createyourmainandloggingbuckets. i. SignupforS3Fox(optional,butrecommended) b. SignupforEC2(themachineswe llusetorunourworkflowson) c. SignupforEMR(thepackagedframeworkwe llusetorunour MapReducejobs;EMRwillrunyourworkflowsonEC2foryou.) 2. Createakeypairandaccessyouraccountcredentials 3. Redeemyour(free)credit!(Andcheckhowmuchyouhaveleft.) 4. FamiliarizeyourselfwithS3,EMRandEC2 1:CreateanAWSaccount Gotohttp://aws.amazon.comandsignupforanaccount,ifyoudonothaveone already.fornow,pleaseentertherequireddetails,includingpaymentdetailsif required.amazonhasgenerouslyagreedtoprovideeachstudentwithcreditforthis class;moreonhowtoredeemthislater.onceyou vecreatedanaccount,youwill needtosignupfortheservicesthatwewillbeusingfortheassignment. 1a:SignupforS3 AmazonSimpleStorageService(S3)isjustastorageframework youcanupload anydatahereandpay(verysmallamounts,unlessyouhaveverylargedata)for storagespace.youcanreadmoreabouts3here: http://aws.amazon.com/s3 GototheAWSManagementConsole,andclickthe S3 tab.thereshouldbea messageinformingyouthatyouneedtosignupfors3.enteryourownpayment detailsfornow wewillloadcreditontotheaccountoncewe reallsetup.
WeneedS3fortworeasons:1.)anEMRworkflowrequirestheinputdatatobeon S3;2.)EMRworkflowoutputisalwayssavedtoS3.Data(orobjects)inS3arestored inwhatwecall buckets.youcanthinkofbucketsasfolders.forthisclass,we ve putthedatayou llneedforpart1ofassignment2inapublicbucketcalled: cs448g data You llseehowtoreferencethisforemrinputlateron.inthemeanwhile,you llneed somebucketsofyourownto1.)storeyouremroutputin,and2.)storeyourlog filesifyouwanttodebugyouremrruns.onceyou reallsignedup,youshouldseea messagepromptingyoutogetstartedbycreatingyourownbuckets.we llcreate thelogbucketfirst. 1.IntheS3console,clickon CreateBucket. 2.AllS3bucketshavetohaveuniquenames,socallyourloggingbucketcs448glastname logging.clickon Create (noton SetUpLogging>> ).
3.Yournewbucketwillappearinthesidebar,withaninformativemessagetelling youthatitisempty. 4.Nowwe llcreateourmainbucket.again,clickon CreateaBucket.Callthisone cs448g lastname.we lllinkourloggingbuckettotheonewe recreatingnow,so clickon SetUpLogging>>. 5. Enablelogging,andstarttypinginthenameofyourloggingbucket.Itshould appearinthedropdownmenu.selectit,and Create. You redonecreatingyourbuckets!awesome
1a i)signupfors3fox(optional,butrecommended) YoucanmanageyourbucketsthroughtheS3console,butthereisafantasticplugin forfirefoxthatmakesuploadinganddownloadingbetweens3andyourlocal machinereallyeasy.ofcourse,you llneedtobeusingfirefox. Fromhttp://www.s3fox.net,clickon Download.Youshouldgetapopupwindow askingyouifyouwanttoinstalltheplugin(examplehere: http://www.s3fox.net/downloadpage.aspx).onceyou veinstalled,restartfirefox. Youshouldbeabletoaccess S3Organizer byclickingonthe Tools tabinthe Firefoxtaskbar,asshownbelow. WhenyoufirstopenS3Fox,you llneedtoenteryouraccesskeyandyoursecret Key.Youcanfindthesebyclickingon Account intheawsmanagementconsole (http://aws.amazon.com/account),andthenonthe SecurityCredentials link.this willtakeyoutoapagewhereyoucanviewyouraccesskeys.
Onceyou vecorrectlyenteredyourcredentialsins3fox,youshouldseeaconsole thatlookssomethinglikethefollowing(onlyyourswon thavefilesinityet).s3fox willmakeitmucheasiertouploadanddownloadyourscriptsandoutputfilesonce westartrunningouremrjobs. 1b:SignupforEC2 IntheAWSconsole,clickonthe EC2 tab,andsignupforec2. Proceedasbefore;thistimeyouwillprobablybeaskedtoverifyyouridentityvia phone.thisshouldnottakemorethanacoupleofminutestocomplete.
Onceyou veverifiedyouridentity,youshouldgetanemailverificationstatingthat you rereadytouseec2.awesome! YoucanreadmoreaboutEC2here:http://aws.amazon.com/ec2. EC2isessentiallyabigmassofmachinesthatyoucan rent torunyourown workloadson.ec2isgreatbecauseyoucancustomizeyourec2instancesaccording toyourcomputerequirements(e.g.specifyram,softwardetc.).wewon tlaunch instancesdirectly,butwhenwerunmapreducejobsthroughemr,emrwilluse EC2instances. 1c:SignupforEMR Repeatstep1b,exceptunderthe ElasticMapReduce tabontheawsmanagement Console. YoucanreadmoreaboutEMRhere:http://aws.amazon.com/elasticmapreduce Inanutshell,EMRprovidesaframeworkforrunningMapReducejobsusingdata storedons3,andmachinesfromtheec2service.we llbeworkingmostlywithemr fromhereon. 2:Accessyouraccountcredentials,andcreateakeypair IfyousignedupforS3Fox,thenyoualreadyknowhowtoaccessyouraccount credentials.takealookatsection1a iofthistutorial,ifnot. WhenwerunjobsonEMR,you llneedtohaveavalidpublic/privatekeypair.you canalsousethesekeystosshintoec2instancemachinesthatyoulaunched,butwe won tgothroughthathere.(comeandfindmeifyouwanttodothis.)tocreate yourfirstkeypair,clickonthe EC2 tabintheawsmanagementconsole. Ontheright,youshouldseealinkstating 0KeyPairs.Clickonthis.
You llbegivenanoptionto CreateKeyPair.Nameyourkeypairasyouwish(try tobemorecreativethanme). Uponprovidinganameandclickingon Create,yourprivatekey(a.pemfile)will automaticallydownload.saveitinasafeplacewhereyou llbeabletofinditagain. Ifyouneedtoaccessyourpublickey,you llbeabletofinditinthesameplacewhere youfoundyouraccountcredentials.amazonkeepsnorecordofyourprivatekey, however,soifyouloseit,you llneedtogenerateanewset.(note:youwon treally needtoaccessyourprivatekeyifyouusetheawsmanagementconsole,butyou willbeaskedtonameyourkeypaireachtimeyourunanemrjob.) 3:AddCredittoyourAccount Finally,thepartyou vebeenholdingyourbreathfor!adding(free)credittoyour account!inordertobecredited,you llneedyouruniquecreditcouponcode.ifyou havenotreceivedthisyet,e mailmeandiwillissueyouone.
Onceyouhaveyourcode,gotoyouraccountpage (http://aws.amazon.com/account)andclickon PaymentMethod.You llseeyour billingdetails;ifyouscrolldown,youwillalsoseeanoptiontoredeemacredit Coupon. ClickonRedeem/ViewAWSCredits,andenteryourcodetogetyourcredit.Please emailthecs448gcoursestaffimmediatelyifthisdoesnotwork. Unfortunately,wecanonlygiveyousomuchfreecredit,sodon tgotoowild!you cancheckonhowmuchcredityouhaveleftbyclickingonthe AccountActivity linkfromyouraccountpage.sometimesthiscantakeawhiletoupdate,sodon tbe surprisedifrecentchangesaren timmediatelyapparent. 4:FamiliarizeyourselfwithS3,EMRandEC2 NowyoushouldbeallsetuptostartwithAssignment2.Themorefamiliaryouare withaws,however,theeasieritwillbe.browsearoundtheawssite thereare plentyofhelpfultutorialsandresourcesthere.partyon