IBM Smart Cloud guide started
|
|
- Caren Phillips
- 8 years ago
- Views:
Transcription
1 IBM Smart Cloud guide started 1. Overview Access link: We are going to work in the IBM Smart Cloud Enterprise. The first thing we are going to do is to sign in. So, the dashboard is composed in four parts: Overview. We can see and access the main options to be managed. Control panel. Add images, create instances and control the storage. Account. Manage users inside instances and permissions. Support. Help and guide you to use the IBM Cloud.
2 Overview panel This window gives us information about our instances and our recent activity in SCE Control Panel In this case we can do the things we want such as create instances, delete it, reboot etc. Also, we can establish predefined images and specifics storage for large data sets.
3 Account These tools allow us to manage fixed IPs and keys for connecting machines.
4 Support
5 2. Create and access to instances You can create different kind of instances such as predefined or your own images. In this document we can focus in preinstalled images and how to configure. First of all, we have to learn how to work and manage instances by command lines. A link with a couple of examples: These examples show us how to create and describe instances. Also we can delete it or ask for a lot of information about everything show in the dashboard told before. When we create an instance, we have to say which private key we want. I recommend that this key must the same for all the instances. So we can connect between instances by the same key. This key allows us to connect for the instance too by software. For that, we have to download the key by the dashboard when we create a private key in the account window. If we want to connect by ssh (i.e: by putty) we have to do these steps: Firstly, we have to generate a.ppk with the software Puttygen. Click on Load button and select de private key downloaded in the dashboard. After that, write the instance s IP.
6 Click in the option Allow attempted changes of username in SSH-2 and select de private key (*.ppk) previously created by putty.
7 3. Configure instances 3.1. Use MPI in the instances The only thing we need is (after installing in one machine in a usual way) copy the folder in other machines and establish the PATH environment. In the last page of the document we can describe the steps in each part Linux Command Line If we want to manage by command line we have to download the tools and establish the JAVA environment. The following link helps us: D883-1A26A43BF046}&v=1.4.1&fid=&tid=&post= To access your IBM account by command line you have to create a file with a passphrase. There is a command line to create that called: ic-create-password -u --username username -p --password password -w --passphrase passphrase -g --file-path filepath [-h --help] See the documentation about the command line necessary to manage the IBM Cloud. Pseudo code for managing instances using MPI: 1. Create Master instance:./ic-create-instance.sh by command line Or Add Instance by dashboard web interface 2. Configure Master instance: a. Copy MPI directory (mpich) and copy directory command line tools. b. Environment variable PATH for MPI and JAVA_HOME Export PATH=$PATH:/home/idcuser/mpich/bin Export JAVA_HOME=/usr/lib/jvm/jre openjdk Export PATH=$PATH:$JAVA_HOME/bin c. Copy Directory keys. Inside key.priv & pass.txt for accessing IBM. The key.priv allows us connect the Master with any worker the first time (by ssh). This file we already have it (previously downloaded in the dashboard) Pass.txt is a file generated by command line tool to access the IBM Cloud. See the documents for creating the file d. Stop IPtables: sudo /sbin/service iptables stop
8 e. Generate public key for ssh: ssh-keygen t rsa. This provides us a file called id_rsa.pub. We use that for allowing Master to connect workers. Permissions to authorize ssh and connect it: chmod 700.ssh chmod 600 key.priv cat.ssh/id_rsa.pub >.ssh/authorized_keys2 chmod 600.ssh/* 3. Create Workers instances: while nº PE < Max do./ic-create_instance.sh u user w passphrase g pass.txt t instancetype (i.e: COP32.1/2048/60 k imageid (see dashboard information) L region (i.e: 61 for Germany) n instance name d description > instance.txt./getid instance.txt newid.txt It is a software that create a file with only the ID for the new instance. This file generated help us later to configure each instance created by this master instance. end while Nº PE = Nº PE Configure workers by master instance. a. Copy mpich directory and establish the PATH environment. b. Copy id_rsa.pub key into the file authorized_keys2 to allow the master to connect this machine. We can do this with an scp with the parameter i. The key can be sent for the master machine. It is the key file downloaded in the dashboard. An example: scp -r -i key.priv masterip :/home/ /.ssh/id_rsa.pub /home/ /.ssh/authorized_keys2 Don t forget to change the key.priv permissions: chmod 600 key.priv c. Change permissions for ssh: chmod 700.ssh chmod 600.ssh/* d. Stop iptables: sudo /sbin/service iptables stop e. Create the directory for MrCirrus: software, data, etc. Also it is necessary to create a little script called sendresult.sh because MrCirrus-Worker when it finished, execute this script to send the partial result to the Master machine. If you use a previous version for MrCirrus, that step is not necessary.
9 3.2. Hadoop f. Create directory called the same name of the tools directory of the IBM command line tools. It is necessary because the mpirun has to change the execution directory (step e) and the worker machine has to have the same directories for that. Example: The Master has the following directory.: - cmdtools with the command line tolos and software necessary to prepare and send the configuration to workers. - testmpi with the MrCirrus software, sequential app and data. So, the worker machine has to have the same name of these directories. g. Once you have configured the machines, you can test if all is right. Try to connect via ssh from the Master machine: ssh IPworker and also test the data and directories are rights. In Hadoop (hadoop-streaming in this case) you have to create a script to upload and download the necessary files to run your application. There are some command lines to do all this things. The script depends on the software. There is an example in this document to prepare a Blast execution for hadoop. This example is necessary to send for all hadoop machines (slaves) before running the job Some of the command line to manage hadoop-streaming are: Manage files: Hadoop dfs get Hadoop-directory/file my machine directory Hadoop dfs put my machine directory Hadoop-directory/file Hadoop dfs rm Hadoop-directory/file Hadoop dfs rmr Hadoop-directory Execute hadoop-streaming: Hadoop jar /mnt/biginsights/opt/ibm/biginsights/ihc/contrib/streaming/hadoop streaming.jar mapper scriptcreated input script-map output Hadoop directory scriptcreated is the script showed in this document. script-map is a file with each execution line.
10 3. Scripts examples. Create instances: Example:./createInstances.sh 4 testibm /home/ /passfile nproc=$1 prefix=$2 #prefix for the names of the machines created. pass=$3 #directory of passfile created by command line while [ $nproc -ne 0 ] do #crear instancias segun numero de procesadores. echo "creando instancia..."./ic-create-instance.sh -u ots@bioinf.jku.at -w bitlab -g $pass -t COP32.1/2048/60 -k c keyrhel -L 61 -n "$prefix$nproc" -d "Instance created by command line" > instances.txt nproc=`expr $nproc - 1` echo "instancia creada...faltan $nproc." cat instances.txt >> first.txt./getids instances.txt IDsAux.txt cat IDsAux.txt >> newids.txt done Configure instances: First of all, I have sent this script to the worker and I execute it by ssh from the Master Node. Example:./scriptClient.sh IPMaster Directory of MrCirrus (only the name) # Copy Directory MPI from master miip=$2 dirmpi=$1 scp -r -i key2.priv -o StrictHostKeyChecking=no $miip:/home/idcuser/mpich/ /home/idcuser/ chmod 777 mpich/bin/* echo "export PATH=/home/idcuser/mpich/bin:$PATH" >>.bash_profile scp -i key2.priv -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $miip:/home/idcuser/.ssh/id_rsa.pub /home/idcuser/.ssh/authorized_keys2 chmod 700.ssh/ chmod 600.ssh/* sudo /sbin/service iptables stop #una vez preparado hay que copiar los datos mkdir cmdtools mkdir $dirmpi
11 scp -r -i key2.priv -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $miip:/home/idcuser/$dirmpi/soft /home/idcuser/$dirmpi scp -r -i key2.priv -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $miip:/home/idcuser/$dirmpi/data /home/idcuser/$dirmpi # When the Worker finish, MrCirrus execute a script called sendresult.sh. This script depends of the application. In this case AllFrag is the app used. echo "scp -i /home/idcuser/key2.priv -o StrictHostKeyChecking=no /home/idcuser/$dirmpi/bin* $miip:/home/idcuser/$dirmpi/" > /home/idcuser/$dirmpi/sendresult.sh echo "ls -l > ls.txt" >> /home/idcuser/$dirmpi/sendresult.sh echo "rm bin*" >> /home/idcuser/$dirmpi/sendresult.sh chmod 777 /home/idcuser/$dirmpi/sendresult.sh Hadoop script #!/usr/bin/env bash # map_hadoop_blast.sh: running a C program using Hadoop Streaming # Written by Oscar, 2011 read offset CMD echo "offset = $offset, CMD = $CMD" >&2 #bucket=jlc-uma folder=dirprueba app=blastall64 APP_PATH=`pwd` # Splits CMD into individual words (white space delimiter) set $CMD app=$offset algorithm=$1 db=$3 seq2=$5 output_file=$7 paramb=${9} paramv=${11} #S3_PREFIX='$bucket.s3.amazonaws.com' # Check directory contents ls >&2 if [! -f $app ] then echo "soft get $app" >&2 hadoop dfs -get $folder/$app $app fi if [! -f $db ]
12 then echo "hadoop dfs -get $folder/$db $db" >&2 hadoop dfs -get $folder/$db $db fi if [! -f $db.nhr ] then echo "hadoop dfs -get $folder/$db.nhr $db.nhr" >&2 hadoop dfs -get $folder/$db.nhr $db.nhr fi if [! -f $db.nin ] then echo "hadoop dfs -get $folder/$db.nin $db.nin" >&2 hadoop dfs -get $folder/$db.nin $db.nin fi if [! -f $db.nsq ] then echo "hadoop dfs -get $folder/$db.nsq $db.nsq" >&2 hadoop dfs -get $folder/$db.nsq $db.nsq fi if [! -f $seq2 ] then echo "hadoop dfs -get $folder/$seq2 $seq2" >&2 hadoop dfs -get $folder/$seq2 $seq2 fi echo "Launching: $app -p $algorithm -d $db -i $seq2 -o $output_file -b $paramb -v $paramv" >&2 chmod 777 $APP_PATH/$app $APP_PATH/$app -p $algorithm -d $db -i $seq2 -o $output_file -b $paramb -v $paramv ls >&2 echo "Uploading to IBM $output_file $folder" >&2 hadoop dfs -put $output_file $folder/$output_file # Writes something to standard output echo "File $seq2 processed over the DB $db. Output file is $output_file!!" # Script report status exit 0
Easily parallelize existing application with Hadoop framework Juan Lago, July 2011
Easily parallelize existing application with Hadoop framework Juan Lago, July 2011 There are three ways of installing Hadoop: Standalone (or local) mode: no deamons running. Nothing to configure after
More informationHADOOP - MULTI NODE CLUSTER
HADOOP - MULTI NODE CLUSTER http://www.tutorialspoint.com/hadoop/hadoop_multi_node_cluster.htm Copyright tutorialspoint.com This chapter explains the setup of the Hadoop Multi-Node cluster on a distributed
More informationSingle Node Hadoop Cluster Setup
Single Node Hadoop Cluster Setup This document describes how to create Hadoop Single Node cluster in just 30 Minutes on Amazon EC2 cloud. You will learn following topics. Click Here to watch these steps
More informationSetup Hadoop On Ubuntu Linux. ---Multi-Node Cluster
Setup Hadoop On Ubuntu Linux ---Multi-Node Cluster We have installed the JDK and Hadoop for you. The JAVA_HOME is /usr/lib/jvm/java/jdk1.6.0_22 The Hadoop home is /home/user/hadoop-0.20.2 1. Network Edit
More informationHow To Install Hadoop 1.2.1.1 From Apa Hadoop 1.3.2 To 1.4.2 (Hadoop)
Contents Download and install Java JDK... 1 Download the Hadoop tar ball... 1 Update $HOME/.bashrc... 3 Configuration of Hadoop in Pseudo Distributed Mode... 4 Format the newly created cluster to create
More informationHSearch Installation
To configure HSearch you need to install Hadoop, Hbase, Zookeeper, HSearch and Tomcat. 1. Add the machines ip address in the /etc/hosts to access all the servers using name as shown below. 2. Allow all
More information1.0. User Manual For HPC Cluster at GIKI. Volume. Ghulam Ishaq Khan Institute of Engineering Sciences & Technology
Volume 1.0 FACULTY OF CUMPUTER SCIENCE & ENGINEERING Ghulam Ishaq Khan Institute of Engineering Sciences & Technology User Manual For HPC Cluster at GIKI Designed and prepared by Faculty of Computer Science
More informationRunning Knn Spark on EC2 Documentation
Pseudo code Running Knn Spark on EC2 Documentation Preparing to use Amazon AWS First, open a Spark launcher instance. Open a m3.medium account with all default settings. Step 1: Login to the AWS console.
More informationLecture 2 (08/31, 09/02, 09/09): Hadoop. Decisions, Operations & Information Technologies Robert H. Smith School of Business Fall, 2015
Lecture 2 (08/31, 09/02, 09/09): Hadoop Decisions, Operations & Information Technologies Robert H. Smith School of Business Fall, 2015 K. Zhang BUDT 758 What we ll cover Overview Architecture o Hadoop
More informationExtending Remote Desktop for Large Installations. Distributed Package Installs
Extending Remote Desktop for Large Installations This article describes four ways Remote Desktop can be extended for large installations. The four ways are: Distributed Package Installs, List Sharing,
More informationRunning Kmeans Mapreduce code on Amazon AWS
Running Kmeans Mapreduce code on Amazon AWS Pseudo Code Input: Dataset D, Number of clusters k Output: Data points with cluster memberships Step 1: for iteration = 1 to MaxIterations do Step 2: Mapper:
More informationSecure Shell. The Protocol
Usually referred to as ssh The name is used for both the program and the protocol ssh is an extremely versatile network program data encryption and compression terminal access to remote host file transfer
More informationThe objective of this lab is to learn how to set up an environment for running distributed Hadoop applications.
Lab 9: Hadoop Development The objective of this lab is to learn how to set up an environment for running distributed Hadoop applications. Introduction Hadoop can be run in one of three modes: Standalone
More informationCISE Research Infrastructure: Mid-Scale Infrastructure - NSFCloud (CRI: NSFCloud)
Chameleon Cloud Tutorial National Science Foundation Program Solicitation # NSF 13-602 CISE Research Infrastructure: Mid-Scale Infrastructure - NSFCloud (CRI: NSFCloud) Cloud - DevStack Sandbox Objectives
More informationHadoop (pseudo-distributed) installation and configuration
Hadoop (pseudo-distributed) installation and configuration 1. Operating systems. Linux-based systems are preferred, e.g., Ubuntu or Mac OS X. 2. Install Java. For Linux, you should download JDK 8 under
More informationHADOOP. Installation and Deployment of a Single Node on a Linux System. Presented by: Liv Nguekap And Garrett Poppe
HADOOP Installation and Deployment of a Single Node on a Linux System Presented by: Liv Nguekap And Garrett Poppe Topics Create hadoopuser and group Edit sudoers Set up SSH Install JDK Install Hadoop Editting
More informationHadoop Hands-On Exercises
Hadoop Hands-On Exercises Lawrence Berkeley National Lab Oct 2011 We will Training accounts/user Agreement forms Test access to carver HDFS commands Monitoring Run the word count example Simple streaming
More information研 發 專 案 原 始 程 式 碼 安 裝 及 操 作 手 冊. Version 0.1
102 年 度 國 科 會 雲 端 計 算 與 資 訊 安 全 技 術 研 發 專 案 原 始 程 式 碼 安 裝 及 操 作 手 冊 Version 0.1 總 計 畫 名 稱 : 行 動 雲 端 環 境 動 態 群 組 服 務 研 究 與 創 新 應 用 子 計 畫 一 : 行 動 雲 端 群 組 服 務 架 構 與 動 態 群 組 管 理 (NSC 102-2218-E-259-003) 計
More informationTP1: Getting Started with Hadoop
TP1: Getting Started with Hadoop Alexandru Costan MapReduce has emerged as a leading programming model for data-intensive computing. It was originally proposed by Google to simplify development of web
More informationHadoop Installation MapReduce Examples Jake Karnes
Big Data Management Hadoop Installation MapReduce Examples Jake Karnes These slides are based on materials / slides from Cloudera.com Amazon.com Prof. P. Zadrozny's Slides Prerequistes You must have an
More informationDeploying MongoDB and Hadoop to Amazon Web Services
SGT WHITE PAPER Deploying MongoDB and Hadoop to Amazon Web Services HCCP Big Data Lab 2015 SGT, Inc. All Rights Reserved 7701 Greenbelt Road, Suite 400, Greenbelt, MD 20770 Tel: (301) 614-8600 Fax: (301)
More informationTS-800. Configuring SSH Client Software in UNIX and Windows Environments for Use with the SFTP Access Method in SAS 9.2, SAS 9.3, and SAS 9.
TS-800 Configuring SSH Client Software in UNIX and Windows Environments for Use with the SFTP Access Method in SAS 9.2, SAS 9.3, and SAS 9.4 dsas Table of Contents Overview... 1 Configuring OpenSSH Software
More informationWinSCP PuTTY as an alternative to F-Secure July 11, 2006
WinSCP PuTTY as an alternative to F-Secure July 11, 2006 Brief Summary of this Document F-Secure SSH Client 5.4 Build 34 is currently the Berkeley Lab s standard SSH client. It consists of three integrated
More informationTutorial- Counting Words in File(s) using MapReduce
Tutorial- Counting Words in File(s) using MapReduce 1 Overview This document serves as a tutorial to setup and run a simple application in Hadoop MapReduce framework. A job in Hadoop MapReduce usually
More informationCS380 Final Project Evaluating the Scalability of Hadoop in a Real and Virtual Environment
CS380 Final Project Evaluating the Scalability of Hadoop in a Real and Virtual Environment James Devine December 15, 2008 Abstract Mapreduce has been a very successful computational technique that has
More informationExtreme computing lab exercises Session one
Extreme computing lab exercises Session one Michail Basios (m.basios@sms.ed.ac.uk) Stratis Viglas (sviglas@inf.ed.ac.uk) 1 Getting started First you need to access the machine where you will be doing all
More informationSet JAVA PATH in Linux Environment. Edit.bashrc and add below 2 lines $vi.bashrc export JAVA_HOME=/usr/lib/jvm/java-7-oracle/
Download the Hadoop tar. Download the Java from Oracle - Unpack the Comparisons -- $tar -zxvf hadoop-2.6.0.tar.gz $tar -zxf jdk1.7.0_60.tar.gz Set JAVA PATH in Linux Environment. Edit.bashrc and add below
More informationIntroduction to analyzing big data using Amazon Web Services
Introduction to analyzing big data using Amazon Web Services This tutorial accompanies the BARC seminar given at Whitehead on January 31, 2013. It contains instructions for: 1. Getting started with Amazon
More informationInstalling Hadoop. Hortonworks Hadoop. April 29, 2015. Mogulla, Deepak Reddy VERSION 1.0
April 29, 2015 Installing Hadoop Hortonworks Hadoop VERSION 1.0 Mogulla, Deepak Reddy Table of Contents Get Linux platform ready...2 Update Linux...2 Update/install Java:...2 Setup SSH Certificates...3
More informationSetting up your virtual infrastructure using FIWARE Lab Cloud
Setting up your virtual infrastructure using FIWARE Lab Cloud Fernando López Telefónica I+D Cloud Architects, FIWARE fernando.lopezaguilar@telefonica.com, @flopezaguilar (Slides: http://tinyurl.com/fiwarelab-cloud)
More informationInstalling Hadoop. You need a *nix system (Linux, Mac OS X, ) with a working installation of Java 1.7, either OpenJDK or the Oracle JDK. See, e.g.
Big Data Computing Instructor: Prof. Irene Finocchi Master's Degree in Computer Science Academic Year 2013-2014, spring semester Installing Hadoop Emanuele Fusco (fusco@di.uniroma1.it) Prerequisites You
More informationUsing The Hortonworks Virtual Sandbox
Using The Hortonworks Virtual Sandbox Powered By Apache Hadoop This work by Hortonworks, Inc. is licensed under a Creative Commons Attribution- ShareAlike3.0 Unported License. Legal Notice Copyright 2012
More informationSSH! Keep it secret. Keep it safe
SSH! Keep it secret. Keep it safe Using Secure Shell to Help Manage Multiple Servers Don Prezioso Ashland University Why use SSH? Proliferation of servers Physical servers now Virtual / Hosted System management
More informationHow to Backup XenServer VM with VirtualIQ
How to Backup XenServer VM with VirtualIQ 1. Using Live Backup of VM option: Live Backup: This option can be used, if user does not want to power off the VM during the backup operation. This approach takes
More informationManual for using Super Computing Resources
Manual for using Super Computing Resources Super Computing Research and Education Centre at Research Centre for Modeling and Simulation National University of Science and Technology H-12 Campus, Islamabad
More informationHadoop Hands-On Exercises
Hadoop Hands-On Exercises Lawrence Berkeley National Lab July 2011 We will Training accounts/user Agreement forms Test access to carver HDFS commands Monitoring Run the word count example Simple streaming
More informationComsol Multiphysics. Running COMSOL on the Amazon Cloud. VERSION 4.3a
Comsol Multiphysics Running COMSOL on the Amazon Cloud VERSION 4.3a Running COMSOL on the Amazon Cloud 1998 2012 COMSOL Protected by U.S. Patents 7,519,518; 7,596,474; and 7,623,991. Patents pending. This
More informationKognitio Technote Kognitio v8.x Hadoop Connector Setup
Kognitio Technote Kognitio v8.x Hadoop Connector Setup For External Release Kognitio Document No Authors Reviewed By Authorised By Document Version Stuart Watt Date Table Of Contents Document Control...
More informationCONFIGURING ECLIPSE FOR AWS EMR DEVELOPMENT
CONFIGURING ECLIPSE FOR AWS EMR DEVELOPMENT With this post we thought of sharing a tutorial for configuring Eclipse IDE (Intergrated Development Environment) for Amazon AWS EMR scripting and development.
More informationUsing Google Compute Engine
Using Google Compute Engine Chris Paciorek January 30, 2014 WARNING: This document is now out-of-date (January 2014) as Google has updated various aspects of Google Compute Engine. But it may still be
More informationConfiguring for SFTP March 2013
Configuring for SFTP March 2013 Overview You can upload files to and download files from Optimal Payments securely via SFTP. In order to be configured to upload and download files via SFTP, contact Technical
More informationSource Code Management for Continuous Integration and Deployment. Version 1.0 DO NOT DISTRIBUTE
Source Code Management for Continuous Integration and Deployment Version 1.0 Copyright 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. This work may not be reproduced or redistributed,
More informationCreating a DUO MFA Service in AWS
Amazon AWS is a cloud based development environment with a goal to provide many options to companies wishing to leverage the power and convenience of cloud computing within their organisation. In 2013
More informationAdobe Marketing Cloud Using FTP and sftp with the Adobe Marketing Cloud
Adobe Marketing Cloud Using FTP and sftp with the Adobe Marketing Cloud Contents File Transfer Protocol...3 Setting Up and Using FTP Accounts Hosted by Adobe...3 SAINT...3 Data Sources...4 Data Connectors...5
More informationInstall and configure SSH server
Copyright IBM Corporation 2009 All rights reserved Install and configure SSH server What this exercise is about... 1 What you should be able to do... 1 Introduction... 1 Part 1: Install and configure freesshd
More informationBack Up Linux And Windows Systems With BackupPC
By Falko Timme Published: 2007-01-25 14:33 Version 1.0 Author: Falko Timme Last edited 01/19/2007 This tutorial shows how you can back up Linux and Windows systems with BackupPC.
More informationHadoop Distributed File System and Map Reduce Processing on Multi-Node Cluster
Hadoop Distributed File System and Map Reduce Processing on Multi-Node Cluster Dr. G. Venkata Rami Reddy 1, CH. V. V. N. Srikanth Kumar 2 1 Assistant Professor, Department of SE, School Of Information
More informationDistributed convex Belief Propagation Amazon EC2 Tutorial
6/8/2011 Distributed convex Belief Propagation Amazon EC2 Tutorial Alexander G. Schwing, Tamir Hazan, Marc Pollefeys and Raquel Urtasun Distributed convex Belief Propagation Amazon EC2 Tutorial Introduction
More informationData Analytics. CloudSuite1.0 Benchmark Suite Copyright (c) 2011, Parallel Systems Architecture Lab, EPFL. All rights reserved.
Data Analytics CloudSuite1.0 Benchmark Suite Copyright (c) 2011, Parallel Systems Architecture Lab, EPFL All rights reserved. The data analytics benchmark relies on using the Hadoop MapReduce framework
More informationHadoop Training Hands On Exercise
Hadoop Training Hands On Exercise 1. Getting started: Step 1: Download and Install the Vmware player - Download the VMware- player- 5.0.1-894247.zip and unzip it on your windows machine - Click the exe
More informationA SHORT INTRODUCTION TO BITNAMI WITH CLOUD & HEAT. Version 1.12 2014-07-01
A SHORT INTRODUCTION TO BITNAMI WITH CLOUD & HEAT Version 1.12 2014-07-01 PAGE _ 2 TABLE OF CONTENTS 1. Introduction.... 3 2. Logging in to Cloud&Heat Dashboard... 4 2.1 Overview of Cloud&Heat Dashboard....
More informationBuilding a Private Cloud Cloud Infrastructure Using Opensource
Cloud Infrastructure Using Opensource with Ubuntu Server 10.04 Enterprise Cloud (Eucalyptus) OSCON (Note: Special thanks to Jim Beasley, my lead Cloud Ninja, for putting this document together!) Introduction
More informationInstallation Guide Setting Up and Testing Hadoop on Mac By Ryan Tabora, Think Big Analytics
Installation Guide Setting Up and Testing Hadoop on Mac By Ryan Tabora, Think Big Analytics www.thinkbiganalytics.com 520 San Antonio Rd, Suite 210 Mt. View, CA 94040 (650) 949-2350 Table of Contents OVERVIEW
More informationHADOOP CLUSTER SETUP GUIDE:
HADOOP CLUSTER SETUP GUIDE: Passwordless SSH Sessions: Before we start our installation, we have to ensure that passwordless SSH Login is possible to any of the Linux machines of CS120. In order to do
More informationRecommended File System Ownership and Privileges
FOR MAGENTO COMMUNITY EDITION Whenever a patch is released to fix an issue in the code, a notice is sent directly to your Admin Inbox. If the update is security related, the incoming message is colorcoded
More informationHOW TO CREATE A SLICE IN GENI?
HOW TO CREATE A SLICE IN GENI? Ahmet Can Babaoglu Computer Science, North Carolina State University July, 2012 *Notice that all the steps are tested on Ubuntu 10.04 platform only. Step by step: How to
More informationRHadoop and MapR. Accessing Enterprise- Grade Hadoop from R. Version 2.0 (14.March.2014)
RHadoop and MapR Accessing Enterprise- Grade Hadoop from R Version 2.0 (14.March.2014) Table of Contents Introduction... 3 Environment... 3 R... 3 Special Installation Notes... 4 Install R... 5 Install
More informationMobile Labs Plugin for IBM Urban Code Deploy
Mobile Labs Plugin for IBM Urban Code Deploy Thank you for deciding to use the Mobile Labs plugin to IBM Urban Code Deploy. With the plugin, you will be able to automate the processes of installing or
More informationComsol Multiphysics. Running COMSOL on the Amazon Cloud. VERSION 4.3b
Comsol Multiphysics Running COMSOL on the Amazon Cloud VERSION 4.3b Running COMSOL on the Amazon Cloud 1998 2013 COMSOL Protected by U.S. Patents 7,519,518; 7,596,474; and 7,623,991. Patents pending. This
More informationSparkLab May 2015 An Introduction to
SparkLab May 2015 An Introduction to & Apostolos N. Papadopoulos Assistant Professor Data Engineering Lab, Department of Informatics, Aristotle University of Thessaloniki Abstract Welcome to SparkLab!
More informationSingle Node Setup. Table of contents
Table of contents 1 Purpose... 2 2 Prerequisites...2 2.1 Supported Platforms...2 2.2 Required Software... 2 2.3 Installing Software...2 3 Download...2 4 Prepare to Start the Hadoop Cluster... 3 5 Standalone
More informationWorking with Docker on Microsoft Azure
Working with Docker on Microsoft Azure Lab Overview In this lab you will create a Docker enabled virtual machine from the Azure Marketplace. You will then go through basic Docker commands. After that,
More informationCASHNet Secure File Transfer Instructions
CASHNet Secure File Transfer Instructions Copyright 2009, 2010 Higher One Payments, Inc. CASHNet, CASHNet Business Office, CASHNet Commerce Center, CASHNet SMARTPAY and all related logos and designs are
More informationNavigating the Rescue Mode for Linux
Navigating the Rescue Mode for Linux SUPPORT GUIDE DEDICATED SERVERS ABOUT THIS GUIDE This document will take you through the process of booting your Linux server into rescue mode to identify and fix the
More informationHadoop Data Warehouse Manual
Ruben Vervaeke & Jonas Lesy 1 Hadoop Data Warehouse Manual To start off, we d like to advise you to read the thesis written about this project before applying any changes to the setup! The thesis can be
More informationDeploy Apache Hadoop with Emulex OneConnect OCe14000 Ethernet Network Adapters
CONNECT - Lab Guide Deploy Apache Hadoop with Emulex OneConnect OCe14000 Ethernet Network Adapters Hardware, software and configuration steps needed to deploy Apache Hadoop 2.4.1 with the Emulex family
More informationSSH. Introduction. SSH Private Key
SSH 1. Introduction 2. SSH Private Key 2.1. How can I download my private SSH key 2.2. How can I get my private SSH key in PPK format? 3. How to connect to my server 4. How can I access my server as the
More informationConnectivity using ssh, rsync & vsftpd
Connectivity using ssh, rsync & vsftpd A Presentation for the 2005 Linux Server Boot Camp by David Brown David has 15 years of systems development experience with EDS, and has been writing Linux based
More informationBig Business, Big Data, Industrialized Workload
Big Business, Big Data, Industrialized Workload Big Data Big Data 4 Billion 600TB London - NYC 1 Billion by 2020 100 Million Giga Bytes Copyright 3/20/2014 BMC Software, Inc 2 Copyright 3/20/2014 BMC Software,
More informationIDS 561 Big data analytics Assignment 1
IDS 561 Big data analytics Assignment 1 Due Midnight, October 4th, 2015 General Instructions The purpose of this tutorial is (1) to get you started with Hadoop and (2) to get you acquainted with the code
More informationCloud Storage Quick Start Guide
Cloud Storage Quick Start Guide Copyright - GoGrid Cloud Hosting. All rights reserved Table of Contents 1. About Cloud Storage...3 2. Configuring RHEL and CentOS Servers to Access Cloud Storage...3 3.
More informationIntroduction to Running Hadoop on the High Performance Clusters at the Center for Computational Research
Introduction to Running Hadoop on the High Performance Clusters at the Center for Computational Research Cynthia Cornelius Center for Computational Research University at Buffalo, SUNY 701 Ellicott St
More informationSecuring Windows Remote Desktop with CopSSH
Securing Windows Remote Desktop with CopSSH Presented by DrNathan@teamhackaday.com If you enjoyed this article, please consider joining our Folding@Home team I like having the ability to remotely access
More informationZend Server Amazon AMI Quick Start Guide
Zend Server Amazon AMI Quick Start Guide By Zend Technologies www.zend.com Disclaimer This is the Quick Start Guide for The Zend Server Zend Server Amazon Machine Image The information in this document
More informationEZcast Installation guide
EZcast Installation guide Document written by > Michel JANSENS > Arnaud WIJNS from ULB PODCAST team http://podcast.ulb.ac.be http://ezcast.ulb.ac.be podcast@ulb.ac.be SOMMAIRE SOMMAIRE... 2 1. INSTALLATION
More informationApache Hadoop 2.0 Installation and Single Node Cluster Configuration on Ubuntu A guide to install and setup Single-Node Apache Hadoop 2.
EDUREKA Apache Hadoop 2.0 Installation and Single Node Cluster Configuration on Ubuntu A guide to install and setup Single-Node Apache Hadoop 2.0 Cluster edureka! 11/12/2013 A guide to Install and Configure
More informationProject 5 Twitter Analyzer Due: Fri. 2015-12-11 11:59:59 pm
Project 5 Twitter Analyzer Due: Fri. 2015-12-11 11:59:59 pm Goal. In this project you will use Hadoop to build a tool for processing sets of Twitter posts (i.e. tweets) and determining which people, tweets,
More informationHow to Run Spark Application
How to Run Spark Application Junghoon Kang Contents 1 Intro 2 2 How to Install Spark on a Local Machine? 2 2.1 On Ubuntu 14.04.................................... 2 3 How to Run Spark Application on a
More informationcloud-kepler Documentation
cloud-kepler Documentation Release 1.2 Scott Fleming, Andrea Zonca, Jack Flowers, Peter McCullough, El July 31, 2014 Contents 1 System configuration 3 1.1 Python and Virtualenv setup.......................................
More informationRstudio Server on Amazon EC2
Rstudio Server on Amazon EC2 Liad Shekel liad.shekel@gmail.com June 2015 Liad Shekel Rstudio Server on Amazon EC2 1 / 72 Rstudio Server on Amazon EC2 Outline 1 Amazon Web Services (AWS) History Services
More informationIBM Pure Application Create Custom Virtual Image Guide - Part 1 Virtual Image by extending
PERFICIENT IBM Pure Application Create Custom Virtual Image Guide - Part 1 Virtual Image by extending How to guide on Virtual Image creation for Pure Application Technical Architect: Chuck Misuraca Change
More informationWindows Intune Walkthrough: Windows Phone 8 Management
Windows Intune Walkthrough: Windows Phone 8 Management This document will review all the necessary steps to setup and manage Windows Phone 8 using the Windows Intune service. Note: If you want to test
More informationIntroduction to Cloud Computing
Introduction to Cloud Computing Qloud Demonstration 15 319, spring 2010 3 rd Lecture, Jan 19 th Suhail Rehman Time to check out the Qloud! Enough Talk! Time for some Action! Finally you can have your own
More informationAdafruit's Raspberry Pi Lesson 6. Using SSH
Adafruit's Raspberry Pi Lesson 6. Using SSH Created by Simon Monk Last updated on 2015-04-09 03:47:50 PM EDT Guide Contents Guide Contents Overview Enabling SSH Using SSH on a Mac or Linux SSH under Windows
More informationAbout this Tutorial. Audience. Prerequisites. Copyright & Disclaimer
About this Tutorial Apache Mahout is an open source project that is primarily used in producing scalable machine learning algorithms. This brief tutorial provides a quick introduction to Apache Mahout
More informationFile transfer clients manual File Delivery Services
File transfer clients manual File Delivery Services Publisher Post CH Ltd Information Technology Webergutstrasse 12 CH-3030 Berne (Zollikofen) Contact Post CH Ltd Information Technology Webergutstrasse
More informationINASP: Effective Network Management Workshops
INASP: Effective Network Management Workshops Linux Familiarization and Commands (Exercises) Based on the materials developed by NSRC for AfNOG 2013, and reused with thanks. Adapted for the INASP Network
More informationHigh-Performance Reservoir Risk Assessment (Jacta Cluster)
High-Performance Reservoir Risk Assessment (Jacta Cluster) SKUA-GOCAD 2013.1 Paradigm 2011.3 With Epos 4.1 Data Management Configuration Guide 2008 2013 Paradigm Ltd. or its affiliates and subsidiaries.
More informationDeploying Cloudera CDH (Cloudera Distribution Including Apache Hadoop) with Emulex OneConnect OCe14000 Network Adapters
Deploying Cloudera CDH (Cloudera Distribution Including Apache Hadoop) with Emulex OneConnect OCe14000 Network Adapters Table of Contents Introduction... Hardware requirements... Recommended Hadoop cluster
More informationRed Hat Enterprise Linux OpenStack Platform 7 OpenStack Data Processing
Red Hat Enterprise Linux OpenStack Platform 7 OpenStack Data Processing Manually provisioning and scaling Hadoop clusters in Red Hat OpenStack OpenStack Documentation Team Red Hat Enterprise Linux OpenStack
More informationHow to connect to the University of Exeter VPN service
How to connect to the University of Exeter VPN service *****Important Part of the process of using the VPN service involves the automatic download and installation of Juniper Network Connect software,
More informationHadoop Tutorial. General Instructions
CS246: Mining Massive Datasets Winter 2016 Hadoop Tutorial Due 11:59pm January 12, 2016 General Instructions The purpose of this tutorial is (1) to get you started with Hadoop and (2) to get you acquainted
More informationHadoop 2.6 Configuration and More Examples
Hadoop 2.6 Configuration and More Examples Big Data 2015 Apache Hadoop & YARN Apache Hadoop (1.X)! De facto Big Data open source platform Running for about 5 years in production at hundreds of companies
More informationhadoop Running hadoop on Grid'5000 Vinicius Cogo vielmo@lasige.di.fc.ul.pt Marcelo Pasin pasin@di.fc.ul.pt Andrea Charão andrea@inf.ufsm.
hadoop Running hadoop on Grid'5000 Vinicius Cogo vielmo@lasige.di.fc.ul.pt Marcelo Pasin pasin@di.fc.ul.pt Andrea Charão andrea@inf.ufsm.br Outline 1 Introduction 2 MapReduce 3 Hadoop 4 How to Install
More informationMapReduce, Hadoop and Amazon AWS
MapReduce, Hadoop and Amazon AWS Yasser Ganjisaffar http://www.ics.uci.edu/~yganjisa February 2011 What is Hadoop? A software framework that supports data-intensive distributed applications. It enables
More informationCactoScale Guide User Guide. Athanasios Tsitsipas (UULM), Papazachos Zafeirios (QUB), Sakil Barbhuiya (QUB)
CactoScale Guide User Guide Athanasios Tsitsipas (UULM), Papazachos Zafeirios (QUB), Sakil Barbhuiya (QUB) Version History Version Date Change Author 0.1 12/10/2014 Initial version Athanasios Tsitsipas(UULM)
More informationHow to install PowerChute Network Shutdown on VMware ESXi 3.5, 4.0 and 4.1
How to install PowerChute Network Shutdown on VMware ESXi 3.5, 4.0 and 4.1 Basic knowledge of Linux commands and Linux administration is needed before user should attempt the installation of the software.
More informationUsing Network Attached Storage with Linux. by Andy Pepperdine
Using Network Attached Storage with Linux by Andy Pepperdine I acquired a WD My Cloud device to act as a demonstration, and decide whether to use it myself later. This paper is my experience of how to
More informationA SHORT INTRODUCTION TO DUPLICITY WITH CLOUD OBJECT STORAGE. Version 1.12 2014-07-01
A SHORT INTRODUCTION TO DUPLICITY WITH CLOUD OBJECT STORAGE Version 1.12 2014-07-01 PAGE _ 2 TABLE OF CONTENTS 1. Introduction....Page 03 2. System Configuration....Page 04 3. Create Backup Script....Page
More informationHow To Enable A Websphere To Communicate With Ssl On An Ipad From Aaya One X Portal 1.1.3 On A Pc Or Macbook Or Ipad (For Acedo) On A Network With A Password Protected (
Avaya one X Portal 1.1.3 Lightweight Directory Access Protocol (LDAP) over Secure Socket Layer (SSL) Configuration This document provides configuration steps for Avaya one X Portal s 1.1.3 communication
More information