EPAM Cloud. Auto Configuration Services. Quick Reference Guide. August 2015 CIRG-4. Version 1.5

Transcription

1 EPAM Cloud Auto Configuration Services Quick Reference Guide August 2015 CIRG-4 Version 1.5

2 Contents Contents... 2 Preface... 4 About this Guide... 4 Audience... 4 The structure of the Guide... 4 Documentation References EPAM Cloud Services. General Notion Cloud Monitoring Service (CMS)... 7 Service Activation... 7 Service Manipulations... 7 Viewing the Results... 8 Web UI... 8 Pricing Auto Configuration Service (ACS) Service Activation Manipulating Chef Server Reconfiguring Your VM with Chef Roles Web UI Pricing Log Aggregation Service (LAS) Service Activation Service Manipulations Web UI and Viewing Logs Pricing Load Balancer Service (LBS) Starting the Load Balancer Configuring the Load Balancer Web UI Pricing Docker Service (DOS) Starting the Docker Service EPAM SYSTEMS

3 Working with Containers Docker Info Web UI and Rest API Working with Docker Images via Docker Registry Pricing Hadoop Data Platform Service (HDP) Starting the Hadoop Service Retrieving Hadoop Info Running Jobs Manipulating Slaves and Clients Web UI Pricing Ambari as a Service Starting a One-Node Ambari Server and Configuring the Cluster Configuring and Starting Cluster Nodes Retrieving Information on Ambari resources Web UI Pricing Default Blueprint Configuration Example Cloudify as a Service (CAS) Cloudify as a Service Cloudify Commands Cloudify UI Changing Credentials Pricing Infrastructure Scheduling Service (CRON) Scheduling Activation Describing Schedules Deleting Schedules Cron Reference Pricing Version History... 45

4 Preface About this Guide This guide is designed for EPAM Cloud users who resort to EPAM Private Cloud Services. The document contains brief descriptions of the available auto-configuration based services and provides the information on their typical flow. NB: This manual provides the most common cases of the usage of commands engaged in services manipulation. The full descriptions of the mentioned commands are given in Maestro CLI Reference Guide (Section 13: Services) Audience This guide is designed for EPAM Private Cloud users who manage their own infrastructure via Maestro CLI. The structure of the Guide The guide consists of ten chapters. The first one describes the general idea of a service, the others give the services descriptions. Please your comments and feedback to EPAM Cloud Consulting at SpecialEPM-CSUPConsulting@epam.com to help us provide you with documentation that is as clear, correct and readable as possible. 4 EPAM SYSTEMS

5 Documentation References You might also want to check the following EPAM Cloud Orchestrator documents: Document Title Maestro CLI Developer Guide Maestro CLI Setup Guide Maestro CLI Quick Start Guide Auto Configuration: Box Solutions Maestro CLI Reference Guide Maestro API Reference Guide Project Management Guide Resource Utilization Quotas Guide Checkpoint Utilization Guide Notification Subscriptions Guide EPAM Private Cloud Billing Guide Contains Information on Maestro CLI client use and customization Installation of Maestro CLI Client Setting up a common cloud environment Auto Configuration Service concept and usage EPAM Cloud Orchestrator Command Line Interface and the list of CLI commands, their parameters and response examples EPAM Cloud Orchestrator Application Programming Interface Hints and guidelines on customization of EPAM Orchestrator for the specific project needs Resource Utilization Quotas and respective guidelines Concepts of checkpoints, including their utilization within Orchestrator, limitations and best practices Notifications description, usage and customization guidelines and default values Current billing model implemented for EPAM Private Cloud AWS Utilization Guide Setup and utilization of Amazon Web Services-based infrastructure through EPAM Cloud Orchestrator Graphical User Interface Guide Maestro Stacks Guide Maestro CLI Quick Reference Card EPAM Cloud Orchestration graphical user interface Maestro Stacks creation and utilization A set of all CLI commands Please your comments and feedback to EPAM Cloud Consulting at SpecialEPM-CSUPConsulting@epam.comto help us provide you with documentation that is as clear, correct and readable as possible. EPAM SYSTEMS 5

6 1 EPAM Cloud Services. General Notion EPAM Private Cloud offers you a set of services, related to Cloud utilization.the main one is Cloud Computing Service that allows you to create and manage your own infrastructure in Cloud. The other services complement Cloud Computing and make it easier and faster in use. These services are software solutions that allow you to install and manipulate a component, necessary for your project, with a simple set of Maestro CLI commands. This functionality is meant to simplify your work with Orchestrator and give you the additional infrastructure monitoring and management tools. Below, you can see the list of the common commands used for services manipulation, and examples of their usage: Starting a service: Most services are started with the or2-manage-service command with the corresponding --service flag that identifies which service is to be started: or2-manage-services -p project -r region -s service --activate Retrieving information on active services: The or2-decribe-services command describes the services activated for given project in the given region: or2-describe-services -p project -r region Getting information on a specific service: The corresponding or2-describe- [service] commands can provide you with the necessary details on the instances involved into the service. For example, the following command will return information about the Log Aggregation service: or2-describe-logging -p project -r region Getting information on service-related instances: To see the list of all the instances involved into a specific service, or into all services, you can use the or2din command with -S/--service parameter followed by a service name or any keyword (to get instances engaged in all services) or2-describe-instances -p project -r region -S any Monitoring service performance: You can monitor the state and the performance of each service using the or2-audit command with --group PROJECT flag. This command returns the list of the service events that take place in the specified project during the current day. or2-audit -p project -r region --group PROJECT It is impossible to start services in a project/region, if its Chef mode is set to USER or EPC. 6 EPAM SYSTEMS

7 2 Cloud Monitoring Service (CMS) EPAM Orchestrator provides the users with a set of monitoring tools that allow to get information on different aspects of the service performance. Cloud Monitoring service includes two components: Chef monitoring and Zabbix monitoring services. Chef Server monitoring service provides you with the Chef Server statistics that is available on UI Monitoring page. This is a default service that does not need to be set up manually. Chef Server Monitoring is a passive service, which means that it does not load the system and uses standard tools of your operation system. Zabbix monitoring service allows the user to get Zabbix statistics on their infrastructure. The service needs to be activated by the user. The table below provides the list of service-related commands and their descriptions. Command or2-manage-service. -s monitoring -a or2-start-monitoring (or2mon) or2-describe-monitoring (or2dmon) or2-stop-monitoring (or2sm) Description Starts the service in the specified project and region Adds an instance to the monitoring list Retrieves information about the instances monitored by Zabbix server, Zabbix monitoring templates and Zabbix agent availability Removes an instance from the monitoring list Service Activation To start Zabbix monitoring service, run or2-manage-service (or2ms) command with the following flags: or2-manage-service -p project -r region -s monitoring -- activate Each project can have only one Zabbix server activated for it. If the server is already activated, you will get the respective error message. If there is no Zabbix server activated for your project, a special stack will be run to launch a VM with all the corresponding Zabbix Server settings. the ID of the executed stack. The command response will give Service Manipulations To start monitoring an instance, run: or2-start-monitoring -p project -r region -i instanceid EPAM SYSTEMS 7

8 For the correct Zabbix Monitoring service performance, it is recommended to add the custom image based instances to the monitoring list only after they come to the running state. Otherwise, the custom image can be indicated incorrectly and will be monitored as a Linux image (regardless of its real type). To retrieve the information about the instances monitored by Zabbix server, Zabbix monitoring templates and Zabbix agent availability, run: or2-describe-monitoring -p project -r region To remove the instance from the Monitoring list, use: or2-stop-monitoring -p project -r region -i instanceid To stop the Zabbix Monitoring Service and remove the Server VM, run the following command: or2-manage-service -p project -r region -s monitoring --deactivate Viewing the Results When the Monitoring Service is set up, you can see the details about the server performance on the UI Monitoring page. Figure 1 - Zabbix Statistics Web UI It is possible to connect to Zabbix server via HTTP connection and to get the detailed information about the server. To get the URL to be used for connection, use the or2-describe-services --full command that returns the list of the services activated for the project, their IP and DNS names and the web UI URL addresses. Copy the webuiurl value and paste it to the address line of your browser. To login, use the following credentials: Login: user 8 EPAM SYSTEMS

9 Password: zabbix After the successful authorization, you will get access to Zabbix Server data: Figure 2 - Zabbix UI Please remember that Zabbix statistics for an instance is shown only after the or2-startmonitoring command adds this instance to the monitoring list. Pricing The service usage price is defined by the price of the Zabbix Server VM. The default parameters of a Zabbix Server VM are: Shape: MEDIUM Image: Ubuntu12.04_64-bit Therefore, the approximate monthly cost of a Zabbix Server usage in case of 100% and 24/7 load is about $57.06 in EPAM-BY1 region (as to 07/28/2014). The price can vary depending on the region. To get more detailed estimations, please, use our Cost Estimator tool. EPAM SYSTEMS 9

10 3 Auto Configuration Service (ACS) Auto configuration service allows Cloud users to run instances with pre-installed sets of software, effectively eliminating the need to install and configure software manually. The table below provides the list of service-related commands and their descriptions. Command Description or2-manage-service. -s chef -a Starts the service in the specified project and region or2-chef-mode (or2cm) Sets one of the existing chef modes to the project or2-describe-chef (or2dchef) Describes the project s Chef Server mode or2-set-instance-properties (or2setp) Used with c/--chefattribute and -h/-- chefrole flag sets the desired chef attribute to be used and the role to be set to the instance Service Activation Auto Configuration Service is activated by default as soon as the project gets activated in Cloud. However, you can change the Chef Server used at your project by switching the Chef mode. To do it, you need to activate the Chef Server Service that is based on EPAM Private Cloud auto-configuration service, and can be deployed in few Maestro CLI commands. To set up the Chef Server Service, run the or2-manage-service (or2ms) command with the following flags: or2-manage-service -p project -r region -s chef --activate Manipulating Chef Server When the service is run, you can use the or2-chef-mode (or2cm) command to switch between the following ACS modes: Default mode (-m default) - the default mode for all projects in the EPAM Private Cloud. In this case, a common Chef server is used for all production environment machines. EPC mode (-m epc) - use project-specific Chef server, created by EPAM Orchestrator for the specified project. User mode (-m user) - use project-specific Chef server, created and properly configured by the user. When switching to this mode, the user should provide Chef server's instance ID (or instance IP) and manually upload validation.pem file to the 10 EPAM SYSTEMS

11 Orchestrator s file storage. The user should also provide the path to validation.pem file during the command invocation. The command call can look as follows: or2-chef-mode -p demopro -r demoreg -m epc NB: 1. The service is released in Beta mode, and some performance issues are possible. 2. It is impossible to start other services in a project/region, if its Chef mode is set to USER or EPC. To get the current Chef mode, Chef Server DNS and Chef Server state, you can use the or2-describe-chef (or2dchef) command: or2-describe-chef -p demopro -r demoreg to stop the Chef Server Service, run the following command: or2-manage-service -p project -r region -s chef --deactivate This command does not remove any resources created during the service performance Reconfiguring Your VM with Chef Roles In order to set a chef role or several roles for an instance, you can use the or2-set-instanceproperties command with alias that allow to simplify the role assignment. The -c/-- chefattribute flag is used to specify the desired chef attribute to be used, and -h/--chefrole - the role to be set to the instance: or2-set--instance-properties -i EVBYMINSD26C6T1 -h role1 -h role2 -c value1 -c "recipename1.attribute1=value2" -p epmc- 2chef Web UI It is possible to connect to Chef Server, created for EPC and USER modes, via HTTP connection and to get the detailed information about the server. NB: The UI for the Default Chef server is not accessible To get the URL to be used for connection, use the or2-describe-chef command. Use the server DNS name to connect. To login to an EPC Chef Server, use the following login and password: Login: user Password: chef-server For initial login to a User Chef Server, use the login and password provided in the left corner of the login page. It is highly advisable that you change these default credentials to custom ones. EPAM SYSTEMS 11

12 When you successfully login, you will get access to Chef Server information: Figure 3 - Chef UI Pricing The service usage influences the project cost in case the Chef mode is switched to EPC. In this case, a special VM for project Chef Server is created. This VM has the following parameters: Shape: MEDIUM Image: Ubuntu12.04_64-bit It is billed just as any other VM of such type. Therefore, the approximate monthly cost of such server in case of 100% and 24/7 load is about $57.06 in EPAM-BY1 region (as to 07/28/2014). The price can vary depending on the region. To get more detailed estimations, please, use our Cost Estimator tool. When the Chef mode is switched from EPC to any other one, the VM launched for EPC mode, is stopped, and is not billed any more. 4 Log Aggregation Service (LAS) EPAM Cloud Orchestrator provides the users with a GrayLog-based Log Aggregator Service that collects the logs from the specified instances and gives an easy and visual access to them via a web-interface. The table below provides the list of service-related commands and their descriptions. Command or2-manage-service. -s log -a or2-start-logging (or2log) or2-describe-logging (or2dlog) Description Starts the service in the specified project and region Start collecting logs from the specified instance Describes the project s Chef Server mode 12 EPAM SYSTEMS

13 or2-stop-logging (or2stoplog) Stops collecting logs from the specified instance Service Activation To start the service, use the or2-manage-service command with the following parameters: or2-manage-service -p project -r region -s log --activate During the service activation, a VM with is started and a GrayLog server is configured on it. The server setup is a complicated process and can take up to an hour. If the project already has a GrayLog Server, you will get the respective error message. Service Manipulations EPAM Orchestrator supports the following commands for Log service manipulations: To add an instance to the logging list, and start collecting logs from these instances, run: or2-start-logging -p project -r region -i instanceid To get the list of the logged instances and the DNS name of the GrayLog Server, use: or2-describe-logging -p project -r region To remove the instance from the Logging Service list, use: or2-stop-logging -p project -r region -i instanceid The server stops collecting the instance log data, but all the logs, previously aggregated, are kept. To stop the log aggregation service and remove the GrayLog Server VM, run the following command: or2-manage-service -p project -r region -s log --deactivate Web UI and Viewing Logs You can view the collected logs by connecting to the Log Server via HTTP. The typical address template for connecting to Log Server is: You can see the Log Server instance DNS in the response of the or2-describe-logging command, as well as of the or2-describe-services command. The login and password that are to be used during the login to the server are returned at or2-describe-services command call: EPAM SYSTEMS 13

14 Figure 4 - Log service user/password details When you log in, you will get to GrayLog Web Interface that provides you with the full access to the gathered data: Figure 5 - GrayLog UI Pricing The service usage price is defined by the price of the GrayLog Server VM. The default parameters of a GrayLog Server VM are: Shape: MEDIUM Image: Ubuntu12.04_64-bit Therefore, the approximate monthly cost of a GrayLog Server usage in case of 100% and 24/7 load is about $57.06 in EPAM-BY1 region (as to 07/28/2014). The price can vary depending on the region. To get more detailed estimations, please, use our Cost Estimator tool. 5 Load Balancer Service (LBS) EPAM Cloud Orchestrator provides the users with a Nginx-based Load Balancing Service that allows the user to arrange load balancing with a set of CLI commands. 14 EPAM SYSTEMS

15 The table below provides the list of service-related commands and their descriptions. Command or2-manage-service. -s load-balancer -a or2-load-balancer-config (or2lbconf) or2-describe-load-balancing (or2dlb) Description Starts the service in the specified project and region Configures load balancing Returns the list of the hosts manipulated by the Load Balancer and the information on their availability Starting the Load Balancer To start the Load Balancer Service, use the or2-manage-service command with the following flags: or2-manage-service -p project -r region -s load-balancer -- activate Only one Load Balancer can be created for a project. If your project already has a Load Balancer, the command will return a corresponding error message. To stop the Load Balancer Service and to remove the Load Balancer Server, perform the following command: or2-manage-service -p project -r region -s load-balancer -- deactivate This command does not remove any resources created during the service performance. Configuring the Load Balancer All the Load Balancer configuration actions are performed with or2-load-balancer-config (or2lbconf) command. The command deals with four configuration areas, each having its own set of properties or properties behaviour: Balancing - this area deals with Load Balancer Server, balancing members, the details on the balancer connections. Limits - this area deals with the user connections and requests limits Bans - this area allows to ban specific URLs or user IPs Cache - this are allows to set up Load Balancer caching When used with --project and --region parameters only, the command displays the list of the configuration settings applied to the Load Balancer. EPAM SYSTEMS 15

16 CLI Parameters Parameter Name Description General -p, --project Project ID -r, --region -P, --plain-output --full --help Virtualization region Use plain output view Show full command output Display command help Balancing -b, --balancing Specifies the configuration area as "balancing" --balancername -i, --instance FQDN for the Load Balancer The ID of the instance to be added or removed from the balancing members list --port The port to route to. Default: 80 --permanent --iphash -d, --remove -d, --removeall Enable/Disable "keep alive" feature: Nginx will open the specified number of connections with each host [0?]. Default 0. Enable/disable "sticky session" [on/off] Default: on Remove a host from the balancing members list Remove all hosts from the balancing members list Limits -l, --limit Specifies the configuration area as "limit" --connections --requests --perip --total Specifies that the number of allowed simultaneous user connections should be set Specifies that the limit of requests per second should be set Sets the connections or requests limit for a given IP (set 0 to remove the limit) Sets the connections or requests limit for the whole balancer Bans --ban Specifies the configuration area to "ban" --ip Adds IP address to the ban list. Format: xxx.xxx.xxx.xxx --url Sets the URL to return failure status code 16 EPAM SYSTEMS

17 --status Set status code for URL to return (Default: 403 Forbidden) -d, --remove -d, --removeall Removes ban from IP or URL Removes all bans Cache --cache Specifies the configuration area to "cahce" --url --extension --expiration -d, --remove -d, --removeall Sets the URL to cache Sets the extensions of files to be cached Sets the cache expiration time (in minutes) Removes caching files or URLs Disables caching To get the list of the hosts manipulated by the Load Balancer and see their availability, use the or2-describe-load-balancing (or2dlb) command: or2-describe-load-balancing -p project -r region Below, you can see some examples of Load Balancer configuration commands: Add an instance to the balancing group: or2lbconf --balancing --instance instance1 Change the port of instance1 to 2752: or2lbconf --balancing --instance instance1 --port 2752 Set maximum connections from a unique IP to 10 and the overall maximum connections to balancer - to 50: or2lbconf --balancing --instance instance1 Set /maestro/ to return 404 Not found: or2lbconf --ban --url /maestro/ --status 404 Enable caching on /images/ with 10 minutes expiration: or2lbconf --cache --url /images/ --expiration 10 Web UI Load Balancer service does not have any web UI. All the manipulations are to be performed via CLI. EPAM SYSTEMS 17

18 Pricing The service usage price is defined by the price of the Load Balancer Server VM. The default parameters of a Load Balancer Server VM are: Shape: MEDIUM Image: Ubuntu12.04_64-bit Therefore, the approximate monthly cost of a Load Balancer Server usage in case of 100% and 24/7 load is about $57.06 in EPAM-BY1 region (as to 07/28/2014). The price can vary depending on the region. To get more detailed estimations, please, use our Cost Estimator tool. 18 EPAM SYSTEMS

19 6 Docker Service (DOS) Docker is a new approach to virtual resources provisioning. It allows you to get virtual environments (containers) within minutes, as they are created based on shared kernel (hypervisor). Meanwhile Docker Containers remain independent in manipulation. EPAM Private Cloud allows you to have Docker as a Service for your project needs. Below, you can see the list of the main Docker concepts: Master - a Docker host with a Docker manager (swarm) that performs Docker cluster health check, load balancing on containers creation, and collects information on existing images, containers, configuration, statuses etc. Hypervisor - A VM that plays a role of a base for containers. A Hypervisor VM is created by Orchestrator on Docker Service activation Container - A Hypervisor-hosted resource that uses a part of Hypervisor s capacities and can be used as a typical virtual server. Container Image - An image stored on Hypervisor and used for containers creation Docker Registry - a repository service that allows you to share VM images between hypervisors. Registry is hosted on a separate VM that is used as a storage for container images. Repository - an entity within a registry, in which the images are grouped. Typically, repository names are taken according to the OS family used on the images that will be stored in this or that repository (e.g., CentOS). Tag - Images in repositories are referenced by tags, which are typically given according to image OS version (e.g., 6, 7, etc.). The service performs in cluster mode (based on Docker Swarm). We use When a user activates the Docker service in a region for their project, Orchestration starts a VM which hosts a Docker Master that becomes the entry point to the cluster. Mater VM is also the first hypervisor where you can run containers. After the Master VM is started, you can create new hypervisors (nodes) to increase your Docker cluster s capacity. To find out more details on Docker as a product, please visit the Official Docker Web-Site. The table below provides the list of service-related commands and their descriptions. Command Description or2-manage-service. -s docker -a Starts the service in the specified project and region or2-docker-container (or2dc) Manipulates Docker containers or2-docker-image (or2di) Manipulates images available for containers creation or2-describe-docker (or2dd) Gives the list of existing Docker elements and their details or2-docker-registry (or2dr) Manipulates Docker registries EPAM SYSTEMS 19

20 or2-docker-registry-image (or2dri) Manipulates images in a registry Starting the Docker Service To start the Docker Service, use the or2-manage-service (or2ms) command with the following flags: or2-manage-service -p project -r region -s docker --activate The command runs a maestro stack that creates and configures a Docker Master VM. If you need a new hypervisor node to be added to your Docker cluster, just repeat the or2- manage-service command. By default, Docker nodes are MEDIUM-shaped VM with Ubuntu operating system. If needed, you can use the --shape parameter with the or2-manage-service command to run a node with a non-default shape. To stop the service, use --deactivate and -i hypervisor_instance_id to deactivate hypervisors one by one, with the Docker Master being deactivated the last. Working with Containers The Hypervisor plays a role of a base for containers that use the hypervisor resources. To run, terminate, stop and start the containers, the or2-docker-container (or2dc) command is used. To run a new container, call the or2dc command with the following parameters: or2-docker-container -p project -r region -a run -i image_id Here, the -a parameter specifies the action to be performed (run), and -i parameter specifies the container image to be used. Docker manager will automatically select the hypervisor with the lowest load and run the container there. However, you can use the -h/--hypervisor parameter to set a specific hypervisor to be used. To see the list of available images, run the or2di command with the -a describe flag: or2-docker-image -p project -r region -a describe While running a new container, you can specify an entry point command by setting it in the --cmd parameter: or2-docker-container -p project -r region -a run -i image_id --cmd "/bin/bash" To stop or start a container, use the or2dc command with the following parameters: or2-docker-container -p project -r region -a stop[or start] -c container_name 20 EPAM SYSTEMS

21 The stop command shuts down the container and releases some of the resources occupied by it. The start command re-launches the stopped container. To remove a container, call the or2dc command with the following parameters: or2-docker-container -p project -r region -a terminate -c container_name Here, the -a parameter specifies the action to be performed (terminate) and -c specifies the container name or ID Docker Info The or2-describe-docker (or2dd) command allows to get the list of the existing Docker elements and their details. Using this command with different parameters, you can get the following information: General overview of the Docker service resources with their details and roles: or2dd -p project -r region The detailed information on a Docker cluster: or2dd -p project -r region -t cluster Web UI and Rest API The Docker service does not have any web UI. All the manipulations are to be performed via CLI. However, there is REST API you can use for your needs The connection details can be found in or2dser command response. Working with Docker Images via Docker Registry Docker service is provided together with the Docker Registry facilities - a repository service that allows you to share images within the Docker service. A Registry is created on a separate VM and is used as a storage for images. Images in a registry are organized as a catalog: they are grouped in repositories and tagged. EPAM SYSTEMS 21

22 REGISTRY Repository 2 Repository 3 Repository REGION Image_2 (tag_2) Image (tag) Image_3 (tag_3) Hypervisor 1 Hypervisor 2 Container Image Image Container Figure 6 - Docker Registry flow We typically recommend to name repositories after the OS family (e.g.centos), and give images tags according to the OS version (6,7, etc). Docker service and registry manipulations are performed with a set of the following Maestro CLI commands: The typical Docker images manipulation flow can be described in the following steps: 1. Create a new Docker Registry: or2dr -p project -r region -a create On the command call, Orchestrator launches a SMALL VM based on Ubuntu12.04_64-bit image. The command response will return the ID of the stack, used to create the new registry: 2. Find the DNS of the Registry VM, using the or2-describe-sevices (or2dser) command: or2dser -p project -r region The command output will give the list of active project services, where you can find the details: 22 EPAM SYSTEMS

23 3. Create a new image from an existing container, using or2-docker-image (or2di) command with -a commit property: Where: or2di -p project -r region -a commit -h hypervisor_dns -c container_id -n image_name -g image_tag -a : the action to be taken -h : Docker hypervisor instance ID -c : the ID of the container that will be used to create an image -i : the ID of the image to be created -g : the tag of the image to be created When the command is executed, you will find the following response: 4. Push the new image to an existing Registry, using or2-docker-image (or2di) command with -a push property: or2di -p project -r region -a push -R target_repo_name -i image_name -g image_tag -dr dockerregistryinstanceid You can check whether the image was uploaded to the registry by calling the or2- describe-registry-image (or2dri) command: or2di -p project -r region -i registry_instance_id Please note, that on the registry, the image name is treated as a repository name: 5. Pull the image from the Registry to all existing hypervisors, using or2-docker-image (or2di) command with -a pull property: or2dri -p project -r region -a pull -i image_name -g image_tag -dr dockerregistryinstanceid EPAM SYSTEMS 23

24 After this, you can check the images on other hypervisors and find the imported one there. Please note that the image name includes the hostname of the registry where it is taken from: After this, the new image can be used to run containers in the Docker cluster. You can also pull images from public Docker Registry ( To do this, run the or2di -a pull command without the -dr parameter. 6. To delete a repository with all included images (tags) from a registry, run the or2dri command with the -a delete parameter: or2dri -p project -r region -a delete -dr dockerregistryinstanceid -R repository_name You can delete only the registry with all included tags. It is impossible to remove a single tag (image) Pricing The service usage price is defined by the price of the Docker hypervisor VM. The default parameters of a Docker hypervisor VM are: Shape: MEDIUM Image: Ubuntu14.04_64-bit Therefore, the approximate monthly cost of a Docker Server usage in case of 100% and 24/7 load is about $57.06 in EPAM-BY1 region (as to 07/28/2014). The price can vary depending on the region and the shape you select for the Docker hypervisor. To get more detailed estimations, please, use our Cost Estimator tool. 24 EPAM SYSTEMS

25 7 Hadoop Data Platform Service (HDP) Hadoop is an open-source solution that provides storage and computing facilities for processing large-scale data in clusters. EPAM Private Cloud provides Hadoop capacities that can be used by developers to test and debug their Hadoop jobs before running them on production. EPAM Private Cloud Hadoop Data Platform Service is based on Hortonworks Hadoop solution. The table below provides the list of service-related commands and their descriptions. Command or2-manage-service. -s hadoop -a -l slaves -h shape or2-describe-hadoop (or2dh) or2-manage-hadoop (or2mh) Description Starts the service in the specified project and region Gives the list, the states, and the DNS names of existing Hadoop resources. Creates and removes Hadoop resources Starting the Hadoop Service To start the Hadoop service, run the or2-manage-service (or2ms) command with --activate (-a), --service (-s) hadoop and other necessary flags: where: or2ms -p project -r region -a -s hadoop -l slaves -h shape -l (--hadoop-slave-count) specifies the number of Hadoop slave machines that will be run (1 by default, if the property is not specified) -h (--shape) specifies the instance shape for Hadoop slave machines (MEDIUM by default, if the property is not specified) You can also use the -k (--key-name) option to specify the SSH key for all the created resources. EPAM SYSTEMS 25

26 The service when activated, by default starts the following virtual machines: VM Role OS Shape Number Hadoop Client Ubuntu x64 SMALL 1 Hadoop Resource Manager Ubuntu x64 MEDIUM 1 Hadoop Name Node Ubuntu x64 MEDIUM 1 Hadoop Slave Ubuntu x64 MEDIUM (alterable with -h parameter 1 (alterable with -l parameter To check whether Hadoop has started properly, you should login to the client and run a test job. The cluster is ready to work if the test job is performed without issues. Each project can have only one Hadoop service activated for the region, but the service can include several Hadoop clients, each responsible for its own job. For more details on new Hadoop resources creation, see the Manipulating Slaves and Clients section. Retrieving Hadoop Info To see the list, the states and the DNS names of Hadoop resources, run the or2-describehadoop (or2dh) command: or2dh -p project -r region You can also use the or2-describe-services (or2dser) command with the -s Hadoop flag to find the Hadoop client DNS. Running Jobs To run a Hadoop job, you have to connect to the Hadoop Client you want to use. To connect, use the DNS name retrieved by the or2dh command and the following credentials: User: hdfs Password: user When connected to the Client, run the following command on it: Where: hadoop jar job_path [job parameters] job_path stands for the path to the.jar file describing the Hadoop job job_parameters stands for the list of the parameters that can be accepted by the current job Each Hadoop client has a set of pre-installed demo jobs you can run to check how the service works. For example, the command below will call the job with a program that is intended to calculate the Pi number: 26 EPAM SYSTEMS

27 hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduceexamples jar pi Here, pi specifies the name of a valid program available within the specified job 10 stands for an integer specifying the number of mappings 20 stands for an integer specifying the number of samples per map. You can also see the other examples in the /usr/lib/hadoop-mapreduce/ folder. Manipulating Slaves and Clients It is possible to change the number of existing Hadoop Clients and Slaves after the service activation. This can be done with the or2-manage-hadoop (or2mh) command: or2mh -p project -r region -a -c number -s resource_type where: -a (--add) - a flag identifying that a resource should be added resource_type is a type of the Hadoop resource (client, slave) number identifies the number of the new resources to be created (1 by default) You can also specify the resource shape with the -h / --shape parameter. To remove a slave, use the or2mh command with the -m /--remove parameter: or2mh -p project -r region -c number -s resource_type --remove When you remove a slave, it will still be detected as active on Resource Manager and Name Nod UIs for some time. Web UI To monitor the Hadoop Service performance, you can login to the Hadoop Resource Manager and Hadoop Name Node. For Resource Manager, use its DNS name and port 8088: When you connect, you can see the list of the performed jobs (called Applications here), their status, and get the access to other details: EPAM SYSTEMS 27

28 Figure 7 - Hadoop RM Web UI To login to Hadoop Name Node UI, use the Name Node DNS and the port: Figure 8 - Hadoop Name Node Web UI You don t have to specify any credentials to connect to Hadoop Resource Manager and Name Node web interfaces. Pricing The service usage price is defined by the price of the created Hadoop Resources. The default parameters of a minimum set of Hadoop VMs are: Image: Ubuntu12.04_64-bit Shapes: MEDIUM (three VMs) SMALL (one VM) 28 EPAM SYSTEMS

29 Therefore, the approximate monthly cost of minimum set of Hadoop Data Platform Service in case of 100% and 24/7 load is about $ in EPAM-BY1 region (as to 09/17/2014). The price can vary depending on the region, the number of clients and slaves and their shapes. To get more detailed estimations, please, use our Cost Estimator tool. EPAM SYSTEMS 29

30 8 Ambari as a Service Apache Ambari is an open-source software that provides a web based UI and APIs for provisioning, management and monitoring of Hadoop clusters. As EPAM Cloud Service, Ambari creates an Ambari-Server for your project, sets up a cluster of Ambari-Agents that deploy a chosen stack of Hadoop bundle for you and provide the existing Ambari facilities for managing the Hadoop cluster of yours. Ambari as a Service is based on Hortonworks Ambari You can create Ambari resources in both AWS and standard EPAM Cloud regions. Please, keep in mind the following service performance specifics: If you need to create at least one Cluster resource in Amazon, Ambari server should also be hosted in AWS, in the same region. All AWS-hosted Ambari resources should be hosted in the same region. You can run Cluster resources in several regions simultaneously, including both EPAM and AWS regions. In this case, the server should be hosted in AWS. When an AWS-based Ambari resource is stopped and started again, it changes DNS and loses connection to other Ambai resources. Please, keep your AWS Ambari resources running. Ambari service has been released recently. If you encounter any issues with it, please do not hesitate to contact our Consulting Team. The table below provides the list of service-related commands and their descriptions. Command or2-manage-service. -s ambari -a -c cluster_name -k key or2-ambari-cluster (or2ac) or2-describe-maestro-stack-resources -p project -r region -s cluster_stack_name or2-describe-instances -p project -r region -l cluster_name=your_cluster_name Description Starts the service in the specified project and region Sets up an Ambari cluster for the project Retrieves the information on the resources created in the Ambari cluster Starting a One-Node Ambari Server and Configuring the Cluster To activate the service and create an Ambari server, run the or2-manage-service (or2ms) command with --activate (-a), --service (-s) ambari and other necessary flags: or2ms -p project -r region -a -s ambari -c cluster_name -k key -b blueprint 30 EPAM SYSTEMS

31 where: -c (--cluster-name) specifies the name for the newly created Ambari cluster. -b (--blueprint) specifies the name of a valid Ambari blueprint file you want to use with your Ambari Server. If you do not specify the blueprint file, the default one will be used (see the details below). Ambari Blueprint is a declarative definition of a cluster. With a Blueprint, you specify a Stack, the Component layout and the Configurations to materialize a Hadoop cluster instance (via a REST API) without having to use the Ambari Cluster Install Wizard. -k (--key-name) the SSH key name that will be used for all Ambari resources. The service, when activated, starts an Ambari server (CentOS6_64-bit, MEDIUM) and reserves a cluster with the specified name and configuration parameters. Configuring and Starting Cluster Nodes When the Server is up and running, you can start setting up the Ambari-Agents and describe the cluster configuration parameters. All the settings are taken from two files: The blueprint file you specified at the Ambari-Serveractivation step. It provides the information on the software stack to be installed on the cluster s VMs. The configuration file you specify with the or2-ambari-cluster command describes the cluster s hardware stack (the number of VMs in each group, additional volumes attached to those, etc.). The command is called as follows: or2ac -p project -r region --cluster-name name --action run config-file json where: --cluster-name is the name of the Ambari cluster you want to set up --config-file is the full path to a locally stored.json file with cluster configurations (see Configuration Example). To make sure that the cluster was started successfully, use the or2-describe-services Maestro CLI command: EPAM SYSTEMS 31

32 Before you start creating an Ambari cluster, please, make sure that the action won t exceed the project resource quota (either daily or monthly). If it does, the cluster creation will fail, but some of the resources may remain active and billed. To see your monthly quota usage info, please, call the command: or2report -p project -r region -y year -m month -t quota To see your project daily quota limitation, use: or2dpro -p project --full and see the volumecreationinterval_count, instancecreationinterval_count and maxvolumesizw_gb values for resource creation and volumes size limits. For more details on quotas in EPAM Cloud, see Quotas Utilization Guide. 32 EPAM SYSTEMS

33 Retrieving Information on Ambari resources There are several ways to get the information on the resources engaged in Ambari Service: To get the service availability, the name of the stack that launched the service and the details on the Ambari Server, use: or2-describe-services -p project -r region To get the information on the resources created in the Ambari cluster, use the following commands: or2-describe-maestro-stack-resources -p project -r region -s cluster_stack_name or or2-describe-instances -p project -r region -l cluster_name=your_cluster_name Web UI When you start the Ambari service and the server is run, you can loin to it using its DNS name and port 8080: The server URL is also returned as a part of or2-describe-services command output. Use the following credentials to login to the Ambari server in EPAM regions: Login: admin Password: admin To login to the Ambari Server hosted in AWS, please, use the ec2-user login and the SSH key you specified at service start. However, there is no information on the server until the Ambari cluster resources are up and running. After the or2-ambari-cluster command execution, the information on the cluster resources appear on the Server. You will see an operation on maestro on the main toolbar, and if you click it, you will get the statistics on the cluster resources setup: EPAM SYSTEMS 33

34 Figure 9 - Cluster resources setup It is strongly recommended not to perform any operations on the server till the setup is finished. When the setup is completed, the Ambari service is ready to use. Pricing The service usage price is defined by the price of the created Ambari Resources and will depend on the number and parameters of cluster VMs. To get more detailed estimations, please, use our Cost Estimator tool. 34 EPAM SYSTEMS

35 Default Blueprint As it has been mentioned before, the blueprint file contains the information on the software that should be installed on Ambari Cluster VMs. In case you don t specify a Blueprint, the default one will be used. It contains the following settings: hgroup1 hgroup2 hgroup3 ZOOKEEPER_SERVER ZOOKEEPER_CLIENT PIG HISTORYSERVER NAMENODE HCAT TEZ_CLIENT AMBARI_SERVER APP_TIMELINE_SERVER SQOOP HDFS_CLIENT HIVE_CLIENT NODEMANAGER YARN_CLIENT MAPREDUCE2_CLIENT DATANODE GANGLIA_MONITOR ZOOKEEPER_SERVER ZOOKEEPER_CLIENT PIG HIVE_SERVER HCAT NAGIOS_SERVER SECONDARY_NAMENODE TEZ_CLIENT HIVE_METASTORE GANGLIA_SERVER SQOOP HDFS_CLIENT HIVE_CLIENT NODEMANAGER YARN_CLIENT MAPREDUCE2_CLIENT DATANODE MYSQL_SERVER GANGLIA_MONITOR WEBHCAT_SERVER RESOURCEMANAGER ZOOKEEPER_CLIENT PIG SQOOP HIVE_CLIENT HDFS_CLIENT YARN_CLIENT HCAT MAPREDUCE2_CLIENT TEZ_CLIENT GANGLIA_MONITOR EPAM SYSTEMS 35

36 Configuration Example The code below provides an example of a configuration file: { instances: [ { zone: "AWS-USEAST", group: "hgroup1", shape: "small", count: 2, volumes: [ { size: 1, mount: "/storage/hadoop1", fs: "ext4" }, { size: 2, mount: "/storage/hadoop2", fs: "ext3"}, { size: 1, mount: "/storage/hadoop3", fs: "ext4"} ]}, { zone: "EPAM-BY1", group: "hgroup2", shape: "small", count: 1}, { zone: "EPAM-RU1", group: "hgroup3", shape: "small", count: 1, volumes: [ { size: 3, mount: "/storage/hadoopcsa1", fs: "ext4"}, { size: 3, mount: "/storage/hadoopcsa2", fs: "ext3"}] } ] } 36 EPAM SYSTEMS

37 9 Cloudify as a Service (CAS) The Cloudify Servcie is a PaaS solution that allows you to create your own Private Cloud inside EPAM Private Cloud. In contrast to DevOps tools that simply assist in installing your applications, the Cloudify Service, built on proven technology (our mission-critical high-performance elastic application platform GigaSpace XAP), provides a mechanism for managing the entire life cycle of your application. For the detailed description of the Cloudify service, please, see the Cloudify Service Guide. Cloudify as a Service To start using the Cloudify Service, you should perform the following steps: Install EPAM Private Cloud Orchestrator CLI Activate your project in EPAM Orchestration Service Access the EPAM Private Cloud via EPAM Cloud Orchestrator CLI Using EPAM Private Cloud Orchestrator CLI, create a key for your project in the selected region. The Cloudify Service can be started with only one command: or2-manage-service -s Cloudify -a -p project -r region -k key name When executed, this command will: Download Cloudify package with default recipes Unpack it Set your properties (user, project, region, key, EPAM Private Cloud Orchestrator url - all of them will be taken from EPAM Private Cloud Orchestrator properties) Run Cloudify CLI (only for Windows) EPAM SYSTEMS 37

38 Figure 10 - Cloudify Shell After that, all you need to do is just to start working with Cloudify as with an independent application. NB: It is recommended to use the Cloudify Service with the following images: CentOS6_64-bit Ubuntu12.04_64-bit Cloudify Commands When the Cloudify service is activated, the Cloudify CLI is launched. The Service can be easily operated by the following commands: Command bootstrap-cloud <cloud name> connect <Cloudify manager IP> Description Create a new Cloudify cloud. The command returns the IP of the created Cloudify Manager and connects the current CLI session to this manager. To work with EPAM Private Cloud you should use epam? as a cloud name. Note: Only one Cloudify cloud per project in a region can be created inside EPAM Private Cloud If a Cloudify cloud has already been created for the project in the given region, the command returns the corresponding Cloudify Manager IP, but CLI session is not connected to the manager. Connect to an existing Cloudify manager install-application name> <application Install an application from your local receips folder 38 EPAM SYSTEMS

39 install-service <service name> Install a service from your local receips folder uninstall-service <service name> Uninstall the specified service uninstall-application <application name> shutdown-managers teardown-cloud Uninstall the specified application Remove all Cloudify data from Manager VM correctly Delete the specified cloud Cloudify UI Cloudify Service provides you with a web user interface that allows to obtain the data on your cloud performance. The web UI URL is returned as a part of the response displayed by the bootstrap-cloud command used to run the Cloudify Manager machine. Here, you can get the information on the Cloudify Cloud: Figure 11 - Cloudify UI EPAM SYSTEMS 39