Introduction to CloudScript

Transcription

1 Introduction to CloudScript A NephoScale Whitepaper Authors: Nick Peterson, Alan Meadows Date: CloudScript is a build language for the cloud. It is a simple Domain Specific Language (DSL) that is similar in purpose to make, ant, or rake. Yet, the underlying distributed state machine architecture of the CloudScript interpreter opens up powerful new possibilities for advanced workflow automation. You'll find the CloudScript DSL is business readable. Its self documenting syntax encourages transparency and reproducibility. The ease with which infrastructure can be provisioned enables you to quickly experiment with different environments for your applications. This tutorial isn't going to focus on developing CloudScript line-by-line but rather on the larger picture of programming Infrastructure-as-a-Service (IaaS). The goal of this tutorial is to give you the conceptual background to understand the purpose and function of CloudScript. We'll provide plenty of examples of CloudScript in subsequent guides. Design Philosophies Our approach to cloud computing is based on our experience as service providers and from what we have learned helping organizations to use IaaS as an enabling technology for the implementation of Developer Operations (DevOps) practices and Agile software development methodologies. We want to make it easy for you to treat the infrastructure as code for your application like any other artifact in your project lifecycle. Using CloudScript you will be able to quickly develop, use less code, keep it simple, don't repeat yourself, and be explicit all through a simple and consistent framework. The consistency of design between CloudScript and the RESTful API make it easy for you to use a combination of thin client or thick client programming models depending on your needs. With CloudScript you can orchestrate any element of the NephoScale platform, and you might choose it in a thin client approach to encapsulate all the operational intelligence about your application in a single location. You might prefer that your management application resides elsewhere in a thick client approach making simple RESTful API calls. Or, you may mix and match thin client and thick client programming. Its up to you to decide on the best architecture for your application. More than anything else our goal was to take the best of programming IaaS and improve it. The execution model is simple and easy to understand. The syntax is inspired by make and rake. Embedded code can be included in its original form without any mangling or escaping of characters. It is easy to read. Principles of CloudScript The CloudScript build language is characterized by four primary design principles. First, its dependency based programming provides a computational model of tasks and their dependencies where the execution order can be explicitly specified. Second, because most steps in a build process are idempotent, CloudScript supports granular control over idempotency of operations. Third, it provides an enveloping capability for delivering any embedded code you want such as Bash, Perl, Ruby, etc. to be run on your servers. Finally, its distributed state machine architecture allows you to synchronize partially ordered events and pass data between resources to enable advanced workflow automation. Dependency Based Programming As with most build languages CloudScript computes using a dependency based model where we have tasks with specific prerequisites as their dependencies. With CloudScript you can explicitly declare all pre-requisites so you have maximum control, or you can allow the interpreter to define the dependencies based on what it chooses as the best execution order. Idempotency It's natural to think of a build in terms of tasks and dependencies and most steps in a build are idempotent. Idempotency is a very useful property because it means that we can repeat or retry an operation as often as necessary without causing unintended effects. Further, when provisioning or autoscaling an environment we really don't want unnecessary work to slow down the build and if a resource already exists then we should not go to the trouble of repeating or retrying an operation. Enveloping and Embedded Languages Enveloping languages work well for simple control of other programs, and much of the inspiration for CloudScript comes from make, Unix pipes and rake. It provides a limited set of abstractions and data structures for provisioning resources and for managing the dependencies between infrastructure resources required to create full fledged applications. By embedding other more powerful languages within CloudScript to be run on the provisioned servers you can program all kinds of infrastructure and application configurations in a single location. Distributed State Machine

2 In a distributed system the order in which events occur is only a partial ordering, and it's sometimes impossible to say which of two events happened first. Because the CloudScript interpreter creates an event loop for each resource provisioning thread you can use it to synchronize advanced configurations of distributed systems such as clustered software applications or data processing applications to ensure that the environment configuration or the processing job is completed before moving on to the next step. By allowing data to be piped from one resource to another CloudScript allows you chain together processing steps much like the Unix pipes allow for redirection of standard output from one process to another. Organization of CloudScript CloudScript is organized into one or more sections. There is a header area and an area to define globals. The threads section is where you will define one or independent threads of execution, which in turn may be specified by one or more tasks. Tasks encapsulate all the infrastructure dependencies of what we typically think of as a business service, such as a high availability web application or an unstructured text search application. Embedded languages can be included as text templates and can be templatized and populated with values in the CloudScript interpreter before they are run on the server. Header You will begin each CloudScript with a header where you define the version of the interpreter and the name of the result template to render when the run is complete. cloudscript joomla_single_stack version = ' ' result_template = joomla_result_template Version The version string is a date for your CloudScript and the interpreter will find the nearest version to run. Result Template The result template is a special text template that is interpreted at the end of the CloudScript run. Like all text templates, the template strings will be populated with the value of the referenced variable. text_template joomla_result_template You can now access your Joomla administration site at joomla_server.ipaddress_public }}/administration _eof Globals You may specify any string variables in the globals section. globals mysql_root_password joomla_admin_password joomla_db_name joomla_db_username joomla_db_pasword = 'gaw33wse' = 'kig19dur' = 'joomla' = 'joomla' = 'def27yam' Threads A thread is a distinct event loop in the CloudScript interpreter's distributed state machine, and it can be comprised of one or more tasks that are executed in sequence. All globals are available to each thread, but variables that are local to a thread cannot be referenced by another thread. thread joomla_setup_and_load_test tasks = [joomla_setup, joomla_load_test] Tasks A task represents the infrastructure elements that we normally associate with a service. All the elements in NephoScale that you can provision using the Portal or RESTful API are available in CloudScript. Within a task, a resource to be provisioned is specified with a resource-command tuple that is comprised of resource type, resource name (and optional dependencies), and resource action, followed by the list of attributes for the resource type. Each resource name must be unique for the resource type. Dependencies are specified as a list after the resource name in the resource-command tuple, or they may be directly referenced in an attribute. task joomla_setup # create joomla server keys # create joomla server root password key /key/password joomla_server_password_key read_or_create key_group = _SERVER # create joomla server console key /key/password joomla_server_console_key read_or_create key_group = _CONSOLE

3 # create storage slice keys /key/token joomla_slice_key read_or_create username = 'joomla' # create joomla bootstrap # script and recipe # create slice to store script in cloud storage /storage/slice joomla_slice read_or_create keys = [joomla_slice_key] # create slice container to store script in cloud storage /storage/container joomla_container => [joomla_slice] read_or_create slice = joomla_slice # place script data in cloud storage /storage/object joomla_bootstrap_object => [joomla_slice] read_or_create container_name = 'joomla_container' file_name = 'bootstrap_joomla.sh' slice = joomla_slice content_data = joomla_bootstrap_data # associate the cloudstorage object with the joomla script /orchestration/script joomla_bootstrap_script => [joomla_slice] read_or_create data_uri = 'cloudstorage://joomla_slice/joomla_container/bootstrap_joomla.sh' type = _SHELL encoding = _STORAGE # create the recipe and associate the script /orchestration/recipe joomla_bootstrap_recipe => [joomla_slice, joomla_bootstrap_script] read_or_create scripts = [joomla_bootstrap_script] # create the joomla server /server/cloud joomla_server => [joomla_bootstrap_recipe] read_or_create hostname = 'joomla-single' image = 'Linux Ubuntu Server LTS 64-bit' type = 'CS1' keys = [joomla_server_password_key, joomla_server_console_key] recipes = [joomla_bootstrap_recipe] recipe_timeout = 900 Text Template A text template is a block of embedded code or a result template. You may substitute variables from the enveloping CloudScript into the template by specifying a template string. text_template joomla_bootstrap_data #!/bin/sh # # Install packages # # get latest package list apt-get update # install apache2 apt-get install -y apache2 # let the user know joomla is coming echo "Joomla is installing, please wait..." > /var/www/index.html # prepare mysql preseed echo "mysql-server-5.1 mysql-server/root_password password {{ mysql_root_password }}" > mysql.preseed echo "mysql-server-5.1 mysql-server/root_password_again password {{ mysql_root_password }}" >> mysql.preseed echo "mysql-server-5.1 mysql-server/start_on_boot boolean true" >> mysql.preseed cat mysql.preseed sudo debconf-set-selections rm mysql.preseed # install php/mysql packages apt-get -y install mysql-server apt-get -y install libapache2-mod-php5 apt-get -y install php5-mysql # restart apache to use the modules /etc/init.d/apache2 restart # download wordpress wget -O /tmp/latest.tgz # extract it and fix permissions cd /var/www && tar xvfz /tmp/latest.tgz chown -R www-data:www-data /var/www

4 #... _eof Template Strings Anything that is inside a text template that is delimited by opening and closing double curly braces is presumed to be a template string that will be substituted when the CloudScript interpreter is run. {{ mysql_root_password }} Elements of CloudScript Programming We find that the most difficult aspect of learning CloudScript is where you make the conceptual leap from dependency based programming to the distributed state machine architecture. You'll need to understand two key concepts to learn how the CloudScript interpreter runs as a distributed system. First, to understand how this all works, you need to be aware of what is going on under the hood with the NephoScale Job Manager. A job is the fundamental unit of work on the NephoScale platform. Every resource that is provisioned is the result of a job, and a resource might require more than one job to be provisioned. Jobs are distributed workers. When you declare a /server/cloud myapp create resource-command tuple you will end up with a job request to provision that resource on the system. The Job Manager is responsible for coordinating all of these distributed jobs and ensuring that they are run reliably to completion. Next, you'll need to grasp that your embedded code is run on the server as a script that is part of a recipe. This is accomplished by having the nephoscale-admin CLI that is installed on all server images make a series of calls to the RESTful API from the server. The first call is made to retrieve the list of recipes to run on the server. Once the list of recipes is received the scripts that comprise the recipe are executed on the system. The standard output of each script is marshalled into JSON format and a callback is made with the result of the recipe again to the RESTful API. If the JSON formatted output from your script is correctly formed, then object member pairs can be referenced within the CloudScript interpreter and used by further steps in your CloudScript. The power of CloudScript programming comes from this distributed programming model and the capability it provides for managing complex infrastructure dependencies. Data Types The set of data types in CloudScript is quite small and limited to Strings, Integers, and Lists. Constants are written in all Upper- Case and prefixed with an '_' character. Resource-Command Tuple At the heart of provisioning infrastructure resources with CloudScript is the resource-command tuple. You can think of this data structure as everything you would include in a call to the RESTful API with a few extra elements. The resource type specifies the thing you want to provision on NephoScale. The resource name must be unique to the type, and identifies your instance uniquely. Optionally, you may specify dependencies on resources that must be created as pre-requisites to creating the resource. The resource command is the action you would like to apply to the resource. The resource attributes are the keyvalue pairs for the instance you will create. Resource Types NephoScale has a well defined set of types for every resource that can be provisioned. Included in the following are some of the more common types, but you'll need to see the CloudScript Developer's Guide for the complete listing. /server/cloud /server/dedicated /storage/slice /storage/container /storage/object /orchestration/script /orchestration/recipe /key/password /key/ssh-rsa /key/token Resource Names A resource name must be unique for the resource type. You cannot have two cloud servers both named 'myapp'. We enforce this constraint because it makes for more readable code and allows you to reference instances by the natural names you give them, however, it can be confusing and frustrating if you are not aware of the constraint in advance. Resource Dependencies When creating the execution order for resource provisioning in CloudScript you may specify dependencies for the pre-requisite resources to exist.

5 /orchestration/recipe joomla_bootstrap_recipe => [joomla_slice, joomla_bootstrap_script] read_or_create Resource Commands CloudScript supports three commands, and this is where the idempotency granular controls come into play. The create command will always try to create the resource that you specify. If you use the create command then you need to ensure that there is not already a resource with this name and type otherwise you will get a namespace error. The read_or_create command will only create the resource if there is not already instance with the name and type, otherwise it will skip over the resource-command tuple. The delete command will remove the resource. Resource Attributes Much like an HTTP POST to the RESTful API, when creating a new resource-command tuple you need to specify the attributes of the resource you are creating. /server/cloud joomla_server => [joomla_bootstrap_recipe] read_or_create hostname = 'joomla-single' image = 'Linux Ubuntu Server LTS 64-bit' type = 'CS1' keys = [joomla_server_password_key, joomla_server_console_key] recipes = [joomla_bootstrap_recipe] recipe_timeout = 900 Orchestration Once you have mastered the simple power of CloudScript you are ready to move on to where you will do the majority of your work. In order for your CloudScript to create a complex environment or autoscale an existing environment you need to orchestrate the code that is run on servers and coordinate the execution order of the CloudScript, including data that is passed using the callbacks and how you want to run the nephoscale-admin CLI. We support multiple resources to facilitate this orchestration process and to make the debugging of your CloudScript as simple as possible. To understand NephoScale orchestration you need to understand the resources that you will create, including projects, scripts, recipes, and callbacks, and you will also need to master the nephoscale-admin CLI which is a command line interface to the RESTful API that resides on every server. Projects A project is an organizing element for all your scripts and recipes including your CloudScript and embedded code. A project will contain the history of results each time a CloudScript is run. You can think of a CloudScript project much as you would any other coding project you would have in Eclipse for example. Scripts A script is a pointer to a file on Cloud Storage, and it has a name that can be referenced within a CloudScript. The code that is in the file on Cloud Storage will be downloaded by nephoscale-admin CLI that resides on a server and executed. Recipes A recipe is a collection of one or more scripts. The scripts that comprise a recipe can be given an execution order, and this organization will help you to modularized and reuse your configuration management code. nephoscale-admin CLI The nephoscale-admin CLI is a small python command line interface to the NephoScale RESTful API. The nephoscale-admin CLI makes it easy to make calls from you servers to the RESTful API and to do things like run recipes on your server for configuration management code. Callbacks When the nephoscale-admin CLI is run on your server and commanded to retrieve recipes from the RESTful API it will marshal the output of each script that is part of a recipe and return the output in a JSON formatted result. If your script generates data that you would like to use further on in your CloudScript then simply output a JSON object with member pairs, and those reference those object member pairs as variables in your CloudScript. Callback Variables You can pass variables from your embedded code using callbacks into the CloudScript interpreter by echoing JSON objects with member pairs to standard output. # only echo JSON so it can be processed by CloudScript as <*>.results.server_key echo "{ \"server_key\":\"$server_key\" }" CloudScript Artifacts CloudScript is a thin orchestration layer on top of the NephoScale platform. To fully understand what is happening under the

6 hood, you will need to be aware of how CloudScript uses the Cloud Storage service and the Job Manager system. Cloud Storage Cloud Storage is an object based storage service with a simple file system like model comprised of slices, containers, and objects. Slices are similar to the drives on your computer, likewise containers are like folders, and objects are like files. While Cloud Storage is analogous to your familiar local file system there are some differences based on how it is accessed and the level of data redundancy it provides. For the purpose of CloudScript, integration with Cloud Storage is seamless, and it is merely a convenient location to manage the files associated with CloudScript. CloudScript Slice There is a default slice called 'cloudscript' that will be created to store all your CloudScript orchestration projects. However, you are not limited to organizing your files here, you can reference other scripts in any slice. Project Containers Within the default slice, for each project you create there will be a container where all the artifacts of the project will be stored. Script Objects Script objects are the files in your project, including your CloudScript file which has a extension of.cs and any other files you will like to manage here such as configuration management code, configuration files, etc. Job Manager The Job Manager does all the heavy lifting in the NephoScale system. Your CloudScript is interpreted into a set of jobs and control of the execution of jobs is managed by the CloudScript interpreter. However, underlying it all, the most basic units of work like creating a cloud server are accomplished using the Job Manager system. Conclusion In this guide we have given you an introduction to CloudScript. It is a simple DSL that is similar in purpose to make, ant, or rake, and it is a build language for the cloud. We also covered many of the design principles and organization of the underlying distributed state machine architecture of the CloudScript interpreter, and we covered the conceptual background material to give you an understand of the possibilities for advanced workflow automation. Our hope is that you find the syntax of CloudScript to be simple and easy to read, and that you find the language powerful and simple, so that you may get quickly to the goals for your project, which are automating the provisioning and management of IT and to treat the infrastructure as code like any other artifact in your project lifecycle. In this tutorial we presented the larger picture of programming IaaS. We provide plenty of examples of CloudScript in subsequent guides, and they are easy to try out on the NephoScale platform. Happy Coding! :) References