Project Full Title: Cloud based Simulation platform for Manufacturing and Engineering. Project Acronym: CloudSME Project Number:

Transcription

1 Project Full Title: Cloud based Simulation platform for Manufacturing and Engineering Project Acronym: CloudSME Project Number: Programme: Cooperation Themes: Information and Communication Technologies; Nanosciences, Nanotechnologies, Materials and new Production Technologies - NMP Call Identifier: FP NMP-ICT-FOF ( Factories of the Future ) Funding Scheme: Collaborative Project Start date of project: 01/07/2013 Duration: 30 months Deliverable: D9.1: CloudSME Simulation Platform as a prototype service Due date of deliverable: 30/06/2014 Actual submission date: 30/06/2014 WPL: SZTAKI Dissemination Level: PU Version: 1.0 Workpackage WP9 Page 1

2 Contents 1. List of Figures and Tables Status and Change History Glossary Introduction CloudBroker-related improvements WS-PGRADE/gUSE Bug fixes, other improvements Permanently running instances Accessing cloud resources directly Direct cloud plugin and its related improvements WS-PGRADE/gUSE WS-PGRADE/gUSE Accessing the EGI Federated Cloud Single-job Submission Wizard demo for the EGI FedCloud Accessing S3 storage resources from Data Avenue Quality improvement of WS-PGRADE/gUSE Logging improvements Bug fixes Automatic build system in the LPDS cloud Support for Java Improvements supporting application development WS-PGRADE/gUSE on Amazon EC ASM releases ASM ASM ASM Single-job wizard Podoactiva use-case Conclusions Workpackage WP9 Page 2

3 1. List of Figures and Tables Figures Figure 1: Example WS-PGRADE/gUSE workflow... 7 Figure 2: Architecture of WS-PGRADE/gUSE... 7 Figure 3: CloudBroker job configuration in WS-PGRADE Figure 4: Overview of the EC2-based direct cloud access process Figure 5: Cloud settings in DCI Bridge Add new menu Figure 6: Cloud settings in DCI Bridge Middleware settings menu Figure 7: Cloud settings in DCI Bridge Edit menu Figure 8: User authentication in WS-PGRADE Figure 9: Job configuration in WS-PGRADE Figure 10: The main change in architecture (On top the old version is, the new version is down): the use of Amazon Java client instead of installing Eucalyptus software Figure 11: The two options to edit cloud resource in Edit menu within DCI Bridge Figure 12: Improved user authentication in WS-PGRADE (in case of SZTAKI cloud access) Figure 13: Improved job configuration in WS-PGRADE Figure 14: The architectural concept of EGI Federated Cloud access Figure 15: The communication between master and slave DCI Bridges within EGI Federated Cloud access Figure 16: Single-job Submission Wizard Figure 17: The Data Avenue architecture with the S3 adaptor and storage Figure 18: Single-Job Wizard main interface Figure 19: Drag and drop a stand-alone application Figure 20: Application uploaded Figure 21: Drag and drop the input files Figure 22: Inputs uploaded Figure 23: type command line arguments (optional) Figure 24: Specify the expected output file names (optional) Figure 25: The job has been submitted Figure 26: When the job is finished, the outputs can be downloaded Figure 27: Configuration panel Figure 28: Execution panel Figure 29: validation in progress Figure 30: Results can be downloaded Workpackage WP9 Page 3

4 Tables Table 1 - Status Change History... 5 Table 2 - Deliverable Change History... 5 Table 3 Glossary... 6 Workpackage WP9 Page 4

5 2. Status and Change History Status: Name: Date: Signature: Draft: Zoltan Farkas 17/06/2014 n.n. electronically Reviewed: Shane Kite 30/06/2014 n.n. electronically Approved: Tamas Kiss 30/06/2014 n.n. electronically Table 1 - Status Change History Version Date Pages Author Modification /06/ /06/2014 All sections ZF First draft version All sections ZF, TG Added materials /06/2014 Section 9 AB Added ASM, Single-job wizard and Podoactiva use-case /06/ /06/2014 All sections ZF Minor corrections Section 10 PK Added Conclusion /06/2014 Section 5 ZF Added Introduction /06/2014 All sections SK, ZF Updated based on internal review Table 2 - Deliverable Change History Workpackage WP9 Page 5

6 3. Glossary ASM DCI EC2 EGI IaaS ID JAX-WS JDL JSDL OCCI PaaS RSL Application-Specific Module Distributed Computing Infrastructure Elastic Compute Cloud European Grid Infrastructure Infrastructure as a Service Identifier Java API for XML Web Services Job Description Language Job Submission Description Language Open Cloud Computing Interface Platform as a Service The Globus Resource Specification Language S3 Simple Storage Service SaaS VM WFI Software as a Service Virtual Machine Workflow Interpreter Table 3 Glossary Workpackage WP9 Page 6

7 4. Introduction WS-PGRADE/gUSE is used as the gateway technology behind the CloudSME Simulation Platform. As such, it will be used by companies to provide user interfaces for their simulation applications, so having functionalities matching their requirements, interaction with cloud providers, stability and robustness are key requirements towards this software. Additionally to these trivial requirements, the work described in this deliverable tries to match the requirements collected from and through WP4, WP6 and WP8 as well. There were multiple platform through which these requirements were collected: CloudSME CodeCamp at Budapest: during this event all the companies were represented, where the applications and requirements were discussed in small groups, with representatives from both CB/ST and MTA SZTAKI. CloudSME Ticketing system: any question, requirement was collected beyond the above event through this forum. Bi-weekly WP6 Skype call: was an excellent platform to monitor the progress of usecase implementation, and to receive feedback and requirements from the simulation software providers. The features of WS-PGRADE/gUSE are really rich: it enables to create, run and manage workflows (a typical workflow is shown in Figure 1) on different distributed computing infrastructures, including cloud providers as well. The very detailed interface can be customized using different methodologies (either the Application Specific Module or the Remote API). Figure 1: Example WS-PGRADE/gUSE workflow WS-PGRADE/gUSE is built up using different services being responsible for different functionalities. The high-level overview architecture is shown in Figure 2. Workpackage WP9 Page 7

8 Figure 2: Architecture of WS-PGRADE/gUSE In this deliverable we are going to present the work performed in WP9 to match the above requirements. First, we present any new development that is related to interaction with the CloudBroker Platform. These developments focus on usability and robustness. Next, we present an additional approach for accessing cloud providers beside the CloudBroker Platform. This is important in case of using some free or local cloud service for developing the application. Workflow can easily be created to run on such services without the need to register the workflow s applications in a software database. Of course once the workflow development phase finished, the application should be migrated over the CloudBroker Platform so that its advanced features like billing and invoicing or detailed logging can be made use of. Next, we present quality improvement-related developments. This includes fixing bugs reported by the community, improving WS-PGRADE/gUSE s build and release process, and improved logging features. In the last section we present all the developments that were performed to improve or extend the customizability and usability of the gateway, including the development of a software provider use-case. Finally, we conclude the contents of the deliverable. Workpackage WP9 Page 8

9 5. CloudBroker-related improvements In this section we describe all the developments targeting the improvements of the already existing CloudBroker integration in WS-PGRADE/gUSE. 5.1 WS-PGRADE/gUSE Prior to WS-PGRADE/gUSE version 3.6.4, CloudBroker entities (like Resources, Software, Executable or Instance types) were identified by their name, not their unique identifier in WS-PGRADE/gUSE. Thus, this reference by name hinders the use of entities of a given type having the same name. For example, let us assume that two organizations (A and B) are registered in the CloudBroker Platform, and both of these organizations define Software called C (at this point, C belonging to A will have the unique identifier ID C_A, and C belonging to B will have the unique identifier ID C_B in the CloudBroker Platform). Also let us assume that user U belongs to organization B, and Software C of organization A is made public. This means, that once the user U accesses the CloudBroker Platform, he or she will see two Software entities called C, one belonging to organization A (and having ID ID C_A ), and one belonging to the user s home organization, B (and having ID ID C_B ). By using the referencing by name method in the portal, in the above case it is impossible to tell if the user has selected Software C of organization A or organization B. Thus, a major improvement of WS-PGRADE/gUSE in terms of interfacing with the CloudBroker Platform is the introduction of referencing by identifier. Of course the identifiers are hidden from the users, they still see the name of the Software they have selected, but internally, the identifiers of the entities referenced are stored. This improvement required change in the following components of WS-PGRADE/gUSE: the DCI Bridge, the web frontend (WS-PGRADE), and the Workflow Interpreter (WFI). DCI Bridge is the job submission component of WS-PGRADE/gUSE. It has a pluggable architecture, where different plugins are responsible for interfacing with different compute infrastructures. A dedicated CloudBroker Plugin is responsible for arranging job submissions to the CloudBroker Platform. Upon job submission, the DCI Bridge receives a JSDL (Job Submission Description Language) document. This JSDL contains all the necessary entity identifiers to have the job submitted to the CloudBroker Platform: Executable, Software, Resource, Region and Instance type (see Figure 3 for a job s configuration in WS-PGRADE). Using the received identifiers, the DCI Bridge s CloudBroker plugin creates a new job, and arranges its execution. Once the job has finished, the results are fetched, and stored on the portal. The necessary change in WS-PGRADE is invisible for the users. However, when the user selects the Software and Executable to run on a selected Resource, Region and Instance Type (as shown in Figure 3), then in the background, the identifiers belonging to these selected entities is saved in the workflow s configuration. Workpackage WP9 Page 9

10 Figure 3: CloudBroker job configuration in WS-PGRADE The CloudBroker-specific parts of the above job s configuration as stored on the portal <execute key="gridtype" value="cloudbroker"/> <execute key="grid" value="platform"/> <execute key="softwarename" value="acc06e8d a a1c95a40c"/> <execute key="executablename" value="c75db6fa-d6eb-47e3-a b31"/> <execute key="resourcename" value="409b7b6b-2bd0-476c-9fcc-a98ed21780ae"/> <execute key="regionname" value="d43f115f-d9da-4277-a989-0c1f9fb8997d"/> <execute key="instancetypename" value=" f-7e35-4d54-a2bf-a885f8c38064"/> server are as follows: As it can be seen, Software, Executable, Resource, Region and Instance types are not stored using their names, but their IDs. Finally, the necessary change in WFI is really simple. It only has to parse the above shown workflow description, and create a proper JSDL document out of that for the DCI Bridge. The relevant part of the JSDL document is as follows: <Application> <ns2:posixapplication_type> <ns2:executable>c75db6fa-d6eb-47e3-a b31</ns2:executable> </ns2:posixapplication_type> </Application> <Resources regionname="d43f115f-d9da-4277-a989-0c1f9fb8997d" instancetypename=" f-7e35-4d54-a2bf-a885f8c38064" resourcename="409b7b6b-2bd0-476c-9fcc-a98ed21780ae"> </Resources> As it can be seen, the identifier of the Software is missing. This is so, because the identifier of the Executable unambiguously identifies the Software as well Bug fixes, other improvements Beside the above-described development, this release also introduces some bug fixes and enhancements. An important enhancement is the usage of the latest CloudBroker API. This enables to use new resources offered by resource providers in the CloudSME project. A bug related to handling output files of non-wrapped job has also been fixed. 5.2 Permanently running instances Permanently running instances is a new feature of the CloudBroker Platform, enabling the set up of virtual machine instances dedicated for the execution of jobs of a given Executable. For any job submitted of a given Executable afterwards, the already running virtual machine Workpackage WP9 Page 10

11 instances will be used to run the jobs immediately, thus permanently running instances help to eliminate the time-consuming virtual machine startup times. Exploiting this feature from WS-PGRADE/gUSE is transparent. If users need to have such instances ready, they have to start as many null jobs from the CloudBroker platform with the permanently running instance flag set, as many virtual machines as they are willing to have permanently running. Afterwards, jobs submitted from WS-PGRADE/gUSE will be able to use the permanently running instances for executing the jobs. Beside this automatic feature, MTA SZTAKI has started to investigate how to create a userfriendly framework above this feature that enables to start up and shut down permanently running instances for different CloudBroker Platform application through a user interface. Workpackage WP9 Page 11

12 6. Accessing cloud resources directly In this section we describe developments related to accessing clouds directly, by using their EC2 or OCCI interface. Accessing cloud services through these interfaces is important for example in case of private cloud infrastructures, where the CloudBroker Platform cannot directly connect to the cloud services interfaces, but a locally deployed WS-PGRADE/gUSE service can. However, this feature should only be used in the development phase, as it doesn t provide as detailed logging and accounting information as the CloudBroker Platform, and doesn t provide any billing or invoicing services either. Thus, running an application on cloud services at production level is recommended with the help of the CloudBroker Platform (and highly recommended when commercial cloud providers are being used). 6.1 Direct cloud plugin and its related improvements The EC2-based Direct Cloud Access solution doesn t use any commercial brokering platform for job submission to cloud. Any clouds that implements the Amazon EC2 interface is accessible by using this development. This solution enables you to connect your gateway directly with your private cloud. Within the guse release process the release introduced the main functionalities of direct cloud access solution, and then in release the additional changes and improvements were developed. Figure 4: Overview of the EC2-based direct cloud access process Workpackage WP9 Page 12

13 6.1.1 WS-PGRADE/gUSE Within the process of direct cloud access (Figure 4) the Master DCI Bridge that directly connects to guse, forwards jobs through EC2-based frontend cloud service to the Slave DCI Bridge located in the cloud. Technically, the master DCI Bridge starts virtual machine(s) via EC2-based service (the master DCI Bridge can start more VMs of different cloud services where EC2-based frontends are integrated). The started VM contains the properly configured slave DCI Bridge that was previously created and saved as an image in the cloud repository. The process contains the next three tasks: Task 1: The DCI Bridge administrator downloads a public base image containing a properly configured DCI Bridge (this will be the slave DCI Bridge) from a corresponding repository. This image will be saved in the target cloud environment (the used cloud service provided by the Cloud Provider must expose an Amazon EC2 interface). Task 2: The DCI Bridge administrator properly configures the master DCI Bridge (which connects to guse). Task 3: The WS-PGRADE user gets account from a Cloud Provider to a cloud where the image was imported from the repository (the Cloud Provider can provide information about the exact way to get cloud account). From this point the user can use the WS-PGRADE portal for job submission to cloud The process in detail Prerequisites: The ready-for-use, public base image is created by the guse Developer Team. It contains the following software components: Java 6 Apache Tomcat DCI Bridge configured to Cloud resource by the DCI Bridge administrator (this will be the slave DCI Bridge) However, beyond this mandatory image configuration the image content can be freely exceeded. Task 1: Image Downloading and saving in the target cloud(s) - performed by DCI Bridge administrator: DCI Bridge administrator downloads a public base image containing a properly configured DCI Bridge (this will be the slave DCI Bridge) from a corresponding repository. This image will be saved in the target cloud environment (the used cloud service provided by the Cloud Provider must expose an Amazon EC2 interface). Task 2: Master DCI Bridge configuration - performed by DCI Bridge administrator Preparation before master DCI Bridge configurations: Installing euca2ools: To start VM, you need an installed euca2ools on your master DCI Bridge machine. Euca2ools is included with package installations of Eucalyptus. Check with your administrator to confirm that euca2ools is installed properly on your master DCI Bridge machine. Settings in master DCI Bridge (Figure 5, Figure 6, and Figure 7): The steps: 1. Add new Resource name to Cloud middleware in the Cloud/Add new window. 2. Set middleware in the Cloud/Middleware settings window within the DCI Bridge administration interface. 3. Set to BASIC_ATHENTICATION the Supported proxy types property. 4. Keep empty the checkbox of This service running property within Cloud/Edit window and define the location of service WSDL at the WSDL of the other service to access the corresponding cloud service. The form of this setting must be: Workpackage WP9 Page 13

14 <Exact URL of the corresponding Amazon EC2-based cloud service>(space)<the inner ID of image>(space)<maximum number of enabled VMs for a user> Figure 5: Cloud settings in DCI Bridge Add new menu Figure 6: Cloud settings in DCI Bridge Middleware settings menu Figure 7: Cloud settings in DCI Bridge Edit menu Workpackage WP9 Page 14

15 Task 3: Authentication and Job Configuration - performed by WS-PGRADE portal user: If the User already has an account to access the corresponding cloud resource, then she/he can easy use this to authenticate. Otherwise, the user previously needs to register at the Cloud Provider and after registration she/he can authenticate (the Cloud Provider can give information about the exact way to get cloud account). (The authentication type is EC2 API-based Basic Authentication.) Once the User gets authentication data, then you can use it after WS-PGRADE portal login in the Security/Cloud window (Figure 8). (The used password must be not in plaintext form but it must be encrypted by MD5/SHA1.) Figure 8: User authentication in WS-PGRADE Job configuration: The elements of cloud-job configuration in WS-PGRADE (in the Job Executable tab) are transparently shown in Figure 9. The job submission process does not differ from the routine way. To use of robot certification for job submission, you need to do some preliminary tasks: 1. Configure your corresponding job(s) with robot certificate by adding robot permission association. 2. Copy the content of directory apache-tomcat /temp/dci_bridge/robotcert to the same directory of the image that contains the slave DCI Bridge (see description of task 1). 3. Submit your workflow. Workpackage WP9 Page 15

16 Figure 9: Job configuration in WS-PGRADE WS-PGRADE/gUSE The improved version of EC2-based direct cloud access is more comfortable to use, the administrators don t need to install additional Euca2ools software to prepare master DCI Bridge and the users can use more cloud infrastructures simultaneously from one portal by this solution. Figure 10: The main change in architecture (On top the old version is, the new version is down): the use of Amazon Java client instead of installing Eucalyptus software Workpackage WP9 Page 16

17 Improvements Improvements in DCI Bridge configuration: There is a new user-friendly interface for more comfortable configuration. You have two options for changing resource settings in the Cloud/Edit menu (Figure 11): If you want to redirect your job submission to other DCI objects, select Service and parameters tab. If you want to execute your jobs in the cloud (where EC2-based frontend is located) select EC2 cloud frontend tab. (If you keep empty both options, then the old settings will remain.) Figure 11: The two options to edit cloud resource in Edit menu within DCI Bridge Improvements in user authentication: The used password can be in plain text form or can be encrypted by SHA1. Workpackage WP9 Page 17

18 Figure 12: Improved user authentication in WS-PGRADE (in case of SZTAKI cloud access) Improvements in job configuration: You can use the SZTAKI cloud or the LPDS cloud infrastructure for job submission into the cloud (you need valid authentication data to use them). For job configuration you need to sign the Binary icon on the top of the window in the Job Executable tab and then select the cloud at Type setting and SZTAKI cloud or LPDS cloud at Grid setting. The other parts of job configuration and job submission are not differing from the routine way. Figure 13: Improved job configuration in WS-PGRADE 6.2 Accessing the EGI Federated Cloud The generic goal of the concept of EGI Federated cloud access is to enable running any services/jobs from WS-PGRADE/gUSE workflows on EGI Federated Cloud infrastructures. The solution (Figure 14 and Figure 15) is based on the EC2 direct cloud access discussed in the previous section: the main development task is to create a master/slave DCI Bridge arrangement for direct job submission to EGI Federated Cloud. Workpackage WP9 Page 18

19 Figure 14: The architectural concept of EGI Federated Cloud access Within the cloud access process the Master DCI Bridge that directly connects to guse, forwards jobs through rocci (ruby Open Cloud Computing Interface)-based frontend cloud service to the Slave DCI Bridge located in the EGI Federated Cloud-based infrastructure. Technically, the Master DCI Bridge starts the Virtual Machine (VM) via roccibased service (the Master DCI Bridge can start more VMs of different cloud services where rocci-based frontends are integrated). The started VM contains the properly configured Slave DCI Bridge that was previously created and saved as image in the cloud repository. This development is compatible to all cloud services that provide rocci interface. Figure 15: The communication between master and slave DCI Bridges within EGI Federated Cloud access Workpackage WP9 Page 19

20 6.2.1 Single-job Submission Wizard demo for the EGI FedCloud The Single-job Wizard (described in Section 9.3 in detail) enables new users to execute their application on a compute infrastructure really quickly. The typical view of the interface is shown in Figure 16. Figure 16: Single-job Submission Wizard All the users have to do is: first, upload their executable, next upload input files for the executable, next specify the command-line arguments, and finally define the name of the output files their executable is going to produce. This interface is located at the left hand side of Figure 16. Once an experiment has been submitted, users can monitor their progress, and download any results produced, or they can remove experiments. This interface is located at the right hand side of Figure 16. This Single-job Submission Wizard has been presented during the EGI Community Forum held in Helsinki, Finland between 19 th -23 rd of May The workflow behind the interface was set up to run on the EGI Federated Cloud infrastructure. With the demo MTA SZTAKI has presented that it is really simple to use a complex infrastructure like the EGI FedCloud from a convenient user interface backed by WS-PGRADE/gUSE. 6.3 Accessing S3 storage resources from Data Avenue In version the Data Avenue functions were integrated into a new portlet of WS- PGRADE. In version this solution was improved - among others - with the support for accessing S3 storage resources. The implementation process of accessing S3 storage resources is contains the following elements: To access cloud storages an adaptor called S3 Adaptor has been developed for Data Avenue, and connected to the Blacktop through the adaptor interface. The S3 Adaptor relies on Amazon s AWS SDK to access S3 storage resources (such as Amazon s S3 or Ceph Object Gateways). S3 Adaptor implements the adaptor interface of Data Avenue Blacktop, which consists of a set of methods such as list, mkdir, rmdir, rename, delete, copy, move, getinputstream, getoutputstream, permissions. They main challenge at implementing these functions was that S3 basically assumes a single-level hierarchy of files, namely, objects can be organized into buckets, but no deeper hierarchy is explicitly supported. Still, following the quasi-standard way of emulating a deeper hierarchy (which are well recognized by other tools as well), object names can contain slash ( / ) symbol that can be rendered as if files would reside within subdirectories. Workpackage WP9 Page 20

21 Figure 17: The Data Avenue architecture with the S3 adaptor and storage Using this technique, Data Avenue can automatically perform and emulate operation such as directory creation (which creates 0 size objects with specific names), directory- or file deletion (which removes all objects with name starting the specified prefix), renaming (which changes object name suffixes). Such operations are made completely transparent by the Blacktop for the users, and they behave as if a hierarchical file system would be available on the underlying storage resource. Getting input stream of S3 objects are directly provided by Amazon s SDK, however writing large-size S3 objects requires so called multi-part upload. The S3 Adaptor automatically manages multipart uploads by buffering contents to be written, and flushing, when a chunk of appropriate data size is available, providing a plain output stream for the Blacktop. When Blacktop detects that a copy operation on the same server is requested, transfer is done using S3 API calls, which is performed on the server (e.g., copying objects between buckets). This lowers network load on of the Blacktop, and data are not streamed through the Blacktop. Permissions are also special in the case of S3. Owner, group, and other permissions are automatically mapped to S3 access control lists (ACLs), owner, authenticated users or all users grantees respectively. Workpackage WP9 Page 21

22 7. Quality improvement of WS-PGRADE/gUSE In this section we overview developments that improve the quality or usage of WS- PGRADE/gUSE. 7.1 Logging improvements For better troubleshooting and debugging, WS-PGRADE/gUSE uses the log4j-based solution from guse version The Apache log4j is a logging library for Java. With log4j it is possible to enable logging at runtime without modifying the application binary. The log4j package is designed so that these statements can remain in shipped code without incurring a heavy performance cost. Editing a configuration file, without touching the application binary, can control logging behavior. The practical advantage of log4j-logging is to set the level of logging. Therefore you can avoid the logging of unnecessary events. (Technically, the target file of logging is not changed with the introducing of log4j: the catalina.out file.) In guse you can use the INFO, ERROR, and DEBUG properties to set the desired logging level to the given component classes. A use case: set a guse component to enable detailed debugging: 1. Go to the corresponding guse component (to the <Tomcat_Home>/webapps/ <Component_Name>/WEB-INF/classes directory) 2. Open the log4j.properties file 3. Modify the next row: log4j.category.hu.sztaki=info to this: log4j.category.hu.sztaki=debug 4. Restart Tomcat. Note: log4j provides many other configuration settings. About this you find details here: Bug fixes The next errors and bugs were fixed in the WS-PGRADE/gUSE releases from to Bug fixes and minor changes in version 3.6.4: The DCI Bridge handles HTTPS-based inputs and outputs beside HTTP-based I/O. Additional minor bug fixes are in CloudBroker portlet development, in error log handling (SourceForge bug report: #218) and in Data Avenue portlet development, in download function. Improved Install Wizard tool: fixes a number of class loading issues, enables using already available databases for the deployment. Bug fixes and minor changes in version 3.6.2: Solving the job saving problem in case of CloudBroker (SourceForge bug report: #121). Adding missing messages in case of CloudBroker (SourceForge bug report: #123). Solving the lost arguments problem in case of CloudBroker job configuration (SourceForge bug report: #134). Workpackage WP9 Page 22

23 7.3 Automatic build system in the LPDS cloud In order to speed up the release process, MTA SZTAKI has created a virtual machine image and a template that can be used to automatically build and start WS-PGRADE/gUSE from the source code repository, using the Git commit hash. This eases the life of testers, as they simply have to modify at most two parameters in the template, start a virtual machine, and examine the results of the build. If the build fails, they can immediately report towards the developers. If the build succeeds, additional tests can be performed to check new functionalities. The automatic build template can be used as follows on the MTA SZTAKI LPDS cloud infrastructure: 1. One has to log in to the OpenNebula web interface at 2. The build template has to be cloned, 3. The template copy has to be updated: a. The template should be opened in the advanced mode, b. The Git repository URL and the Git commit hash have to be modified to the one sent by the developer. 4. A new VM can be started based on the modified template. The template looks like the following: NIC=[NETWORK_UNAME="oneadmin",NETWORK="cloud-local"] GRAPHICS=[LISTEN=" ",TYPE="VNC"] VCPU="2" CPU="0.001" MEMORY="2048" DISK=[IMAGE_UNAME="zfarkas@sztaki.hu",IMAGE="[APP] guse AutoBuild v2"] CONTEXT=[USER_DATA="#cloud-config bootcmd: - [ '/usr/bin/deploy_guse.sh', 'git://git.code.sf.net/u/zfarkas/guse_git', '52fc8c31ae8eea7b53bb7cd8e6bea0a1766ee210', ' ] "] 7.4 Support for Java 7 In order to keep up with recent versions of Java, WS-PGRADE/gUSE has been modified so it can be started using Java 7, instead of the old, unsupported Java 6. This was achieved by upgrading to JAX-WS version 2.2 and by dropping or upgrading some Java library dependencies (mostly related to managing grid middlewares). Workpackage WP9 Page 23

24 8. Improvements supporting application development 8.1 WS-PGRADE/gUSE on Amazon EC2 In order to help trying out WS-PGRADE/gUSE on a clean, dedicated deployment, MTA SZTAKI has decided to make WS-PGRADE/gUSE available on Amazon EC2. An image has been made public, which offers a stock WS-PGRADE/gUSE deployment that is automatically started up and initialized when the virtual machine boots up. The documentation on how to use the image is really simple, and can be accessed through ASM releases In this period three version of ASM has been released providing the following improvements ASM ASM is technically equivalent to 3.4.5, as no new functions were introduced. Nevertheless this version serves full JavaDoc documentation encapsulated to the API itself. In addition it contains both web-service implementations for CredentialProvider and for WorkflowStatusHandlerService that makes the upgrade process more convenient than it was for upgrading to ASM ASM For increasing performance of workflow configuration, a brand new class called ASMWorkflowConfiguration was introduced. It enables to configure the workflow locally, without getting and saving it in the WFS each time. Hence the visibility of the getworkflow and the saveworkflow methods of ASMService are replaced to be public. Related issue was reported in SCI-BUS User Forum. In ASMService class CheckWorkflowConsistency method is implemented that checks that all inputs, executables, and credentials are available. Related issue reported in SourceForge: Logging is introduced by setting up an LSF4J logging environment ASM The following issues were resolved in this release: Modifying JDL/RSL properties in ASM. Two new methods are introduced for getting/setting a JDL attribute for a given job: getjobattribute(string userid, String workflowname, String jobname, String attribute) / setjobattribute(string userid, String workflowname, String jobname, String attribute, String expression) where, for instance to set the number of the nodes in the JDL, the argument "attribute" must be the proper JDL attribue ("NodeNumber" in this example), and "expression" should be the number of the nodes required. Other example could be if the resource should be modified programmatically, in this case the value of "attribute" variable has to be "Requirements" and the expression must be "other.glueceinfohostname\"myresource\"". This request is reported here: A bug reported here: about getworkflowoutputs method was resolved in this release. Additionally a method called downloadworkflow is introduced in ASMService class to download the whole ASM workflow (what is imported through ASM). Workpackage WP9 Page 24

25 Getting/Storing SubmissionTexts typed by the users right before the workflow submission is implemented in ASMWorkflow object. Technically a new variable called submissiontext is introduced in where the submission method stores the text and its getter function can be used to get it. This issue is reported in this thread: Single-job wizard Single-job wizard is developed to support those users, who are not experienced in workflow development in WS-PGRADE/gUSE system, to be able to execute their own application on grid or cloud environment without having a long learning curve. Before this development, the new users must download and run the Graph Editor, create a node and the proper input and output ports, save it, then must create a Concrete from the graph in the portal interface. Then the workflow must be configured, meanwhile all inputs and binaries must be uploaded and the execution resource must be set. Finally a valid credential must be downloaded or set in the Credential portlet as well. These steps are necessary in case of development of a complex workflow, but many of them can be saved for the inexperienced users, who would like to get acquainted with the system. Moreover following these steps are simply too long for the users concerning the fact that they would like to try the underlying system by executing one job. We were focusing on simplicity, therefore the users must perform at most four steps: 1. Drag n drop their applications 2. Drag n drop the inputs required 3. Optionally add command line arguments 4. Optionally specify the names of the output files that will be generated by the application. Assume a simple script as application requiring 4 input files named inputval1.txt to inputval4.txt (containing just a number), and one number as a command line argument. The application adds all the numbers and returns an output file called outputval.txt. The following figures illustrate these simplified wizard-like steps to be followed by the users. Figure 18 shows the main panel of the wizard; its left side shows the steps and instructs the user what to do (blue step denotes the current one, the greys are the further ones while the light ones denote the previous steps; users are free to navigate among them by clicking next and previous button), while the right hand-side of the portlet shows those applications that have already been uploaded and submitted. Currently it s empty. Figure 18: Single-Job Wizard main interface Workpackage WP9 Page 25

26 First of all, as it is shown in Figure 19, the users must upload the application by dragging it from a folder of the desktop. Figure 20 shows the result, the green tick denotes that the application has been uploaded successfully to the portal server (to a temporary directory). Figure 19: Drag and drop a stand-alone application Figure 20: Application uploaded Next the inputs must be uploaded using the same drag-n-drop concept as it is followed before. Figure 21 and Figure 22 show this process. Workpackage WP9 Page 26

27 Figure 21: Drag and drop the input files Figure 22: Inputs uploaded The third step (show in Figure 23) is to type some command line arguments if needed. Figure 23: type command line arguments (optional) The last step to perform is to specify names of the output files, which are going to be generated by the application. Finally by clicking Start execution button users can trigger the import process resulting a new one-job workflow in their user space containing their inputs Workpackage WP9 Page 27

28 and settings. This workflow is submitted immediately (its status changes are illustrated in Figure 25 and Figure 26) and finally the outputs can be downloaded. Moreover users have option to download all the workflow as well, or remove the experiment. Figure 24: Specify the expected output file names (optional) Figure 25: The job has been submitted Figure 26: When the job is finished, the outputs can be downloaded Behind the curtains To achieve a general solution and to be able to utilize the features provided by ASM, a predeveloped one-job workflow is set up and parameterized for the users. The job is executed on those resource what is prepared by the portal administrator, so if mechanism of the Robot Certificates are utilized, then the users do not have to take care about authentication neither. To follow the concept of ASM the portal administrator must export a pre-developed workflow to the local repository. It contains only one wrapper job covering a simple script for managing input decompression, maintaining the execution of the real application, managing error handling and output decompression, so to cover the whole lifecycle of the execution. Workpackage WP9 Page 28

29 The job contains 4 inputs (the application, inputs compressed, command line arguments, output names descriptor file) and 1 output (all the required outputs compressed). As it is a normal workflow in aspect of WS-PGRADE/gUSE, the portal administrators can use the core configuration possibilities for set it up to be executed in a certain resources and, to associate Robot Permission to the application. 8.4 Podoactiva use-case Using Podoactiva interface the podiatrist will be able to validate the scanned images that will be the input of the insole design process, and he will not need the ability and knowledge of using a CAD program. The interface is developed using ASM utilizing workflow creation and execution features of SZTAKI Gateway. The applications are executed on cloud infrastructure operated by BiFi via CloudBroker. The validation will take place in a few easy steps displayed in the next screenshots. The portlet is separated in two main panels: A configuration panel(shown in Figure 27), that enables the users to upload their scanned images to be validated, and the Manage panel, which contains all the validation jobs submitted showing their current status and letting the users to download their results. Once users have uploaded an image file in the configuration panel and have clicked to Validate button, the whole workflow import and submission process is triggered, they will then be redirected to the Manage panel automatically. In this panel the newly generated validation (which is technically a one-job workflow in aspect of the core SZTAKI gateway) is shown together with its current status (illustrated in Figure 29). When the validation is finished its result is shown by a green tick if the image was validated successfully, or by a red cross otherwise. Of course, the results can be downloaded as well (shown in Figure 30). Figure 27: Configuration panel Figure 28: Execution panel Workpackage WP9 Page 29

30 Figure 29: validation in progress Figure 30: Results can be downloaded Workpackage WP9 Page 30

31 9. Conclusions WS-PGRADE/gUSE is intensively used in many scientific projects both in Europe and outside Europe before the CloudSME project. However, CloudSME is the first project where companies use WS-PGRADE/gUSE and this raised many issues to be solved. The biggest challenge in a company environment is that the platform should be robust, reliable and well documented. In the first project year we have put many efforts to improve these quality requirements of the platform and particularly when it works together with the CloudBroker Platform in an integrated way to serve the needs of the simulation software developer and user companies. As a result of this effort the platform became much more robust and easier to deploy and operate. In order to use WS-PGRADE/gUSE as a generic platform to develop cloud applications another important previously not investigated feature is the capability of accessing cloud storages both directly from a portlet and inside simulation workflows. Therefore we have developed a new S3 storage plugin for the Data Avenue service. This new plugin enables the use of cloud storages compatible with Amazon s S3 storage access protocol. In order to provide an example how to use the platform by the partner companies we have supported Podoactiva to port their application to cloud via the integrated WS- PGRADE/gUSE and CB platform. To facilitate this kind of simple, 1-job workflow applications we have developed a new Single-job wizard by which even a novice user without learning anything about the platform can use it immediately in an intuitive way. This wizard solution is based on the ASM API that was also significantly improved during the project year and several new releases have been issued. In order to enable the immediate use of the WS-PGRADE/gUSE platform in a cloud environment we have created its Virtual Machine Image that can be deployed at any time by anyone on the Amazon cloud. This will be later incorporated into the one stop shop marketplace developed for CloudSME. Finally, we wanted to enable the direct access of clouds from the WS-PGRADE/gUSE platform both for companies and academic institutions. This is important when companies and academics work in joint projects since usually academics prefer to use their own private (free) cloud instead of the commercial clouds. Therefore we released new versions of the WS-PGRADE/gUSE platform that enable the direct access to clouds. This work is particularly important for accessing the largest European academic cloud federation called as EGI FedCloud that is used by a large set of academic research communities all over Europe. Workpackage WP9 Page 31