Collaborative Open Market to Place Objects at your Service

Transcription

1 Collaborative Open Market to Place Objects at your Service D4.1.2 Basic implementation of the COMPOSE runtime infrastructure Project Acronym Project Title COMPOSE Project Number Work Package WP4 Infrastructure Development and Prototyping Lead Beneficiary IBM Editor Yacov Manevich IBM Reviewer Alvaro Villalba BSC Reviewer Juan David Parra PASSAU Dissemination Level PU Contractual Delivery Date 30/04/2014 Actual Delivery Date 30/04/2014 Contractual resubmission Delivery Date Actual resubmission Delivery Date Version V1.1 15/01/ /01/2015 D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 1 of 19

2 Abstract The demonstrated run-time is a highly scalable cloud Runtime Environment hosting applications, services, and service objects in the COMPOSE ecosystem. This is a corner stone of the COMPOSE architecture, such that all components are to be architected in a manner that should be deployable by the provided COMPOSE cloud run-time platform. The COMPOSE cloud is responsible for the management of services accessible through the COMPOSE marketplace, including their core functionalities (e.g., search and discovery, assisted composition, etc.). We started our quest by surveying existing technologies and picking up the one most suitable for COMPOSE to use as a starting point. The conclusion was that Cloud Foundry, an open and extensible PaaS technology is the best basis for the COMPOSE run-time. The present prototype is a customization of the Cloud Foundry PaaS cloud to the specific requirements of the IoT world, as manifested by COMPOSE. A Universal Service Broker is introduced which helps incorporating into the OCMPOSE run-time newly developed services. This is demonstrated by the incorporation of the scalable communication infrastructure as a service to the COMPOSE platform. In addition, we demonstrate the manner in which COMPOSE applications are pushed and hosted by the cloud, and the manner in which they bind to the services they need, as well as being able to exchange information between them. Finally, an external application connects to a COMPOSE application and interacts with it to present the COMPOSE application internal data and capabilities. This document aims to accompany the initial prototype of the COMPOSE run-time platform rather than provide a detailed design document of the demonstrated system. A full-fledged detailed design document is scheduled for the end of the project. D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 2 of 19

3 Document History Version Date Comments V0.1 16/04/2014 Initial version V0.2 20/04/2014 Integrated universal Service Broker API V0.3 20/04/2014 First version ready for review V0.4 27/04/2014 Updated after internal IBM review V0.5 29/04/2014 Incorporated comments from PASSAU review V0.6 29/04/2014 Incorporated comments from BSC review V1.0 30/04/2014 Final Version V1.1 05/01/2015 Added sub-section on Criteria for selection of technologies, Important features and Fit with the project requirements, at the request of the reviewers D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 3 of 19

4 Table of Contents Abstract... 2 Document History... 3 List of Figures... 5 List of Tables... 5 Acronyms Introduction Criteria for selection of technologies, Important features and Fit with the project requirements Prototype overview - short description Architectural responsibilities and interactions High level design Main functionalities introduced The Universal Service Broker COMPOSE Customized Services Pushing applications and binding to services Data Services The run-time environment: What is being demonstrated External API - for use by 3rd parties Installation & configuration Future directions D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 4 of 19

5 List of Figures Figure 1: COMPOSE components within the cloud platform... 7 Figure 2: The road to an IoT PaaS Figure 3: Demo outline Figure 4: Universal Service Broker Architecture Figure 5: Binding an application using cf commands Figure 6: Binding an application to a service at "push" time List of Tables Table 1: Acronyms table... 6 D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 5 of 19

6 Acronyms Table 1: Acronyms table Acronym COMPOSE API CF IaaS JS JSON JVM PaaS PID REST RMI UAA URL USBB Meaning Application Programming Interface Cloud Foundry Infrastructure as a Service JavaScript JavaScript Object Notation Java virtual Machine Platform as a Service Process ID Representational state transfer Remote Method Invocation User Account and Authentication Universal Service Locator Universal Service Broker Back-end D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 6 of 19

7 1 Introduction The COMPOSE run-time environment provides hosting and management services to all COMPOSE entities. It is a central piece of the COMPOSE architecture and all additional components are constructed to fit within this platform. The COMPOSE run-time is manifested as a customization of the Cloud Foundry (CF) environment to better serve the IoT world. COMPOSE components are created and hosted as CF services or applications. CF Services are state-full and provide general capabilities to CF applications, and are accessible only to CF applications. CF applications on the other hand are stateless, and may be bound to CF services to get additional capabilities, such as storage, communication, and access to the COMPOSE data management component. Figure 1: COMPOSE components within the cloud platform The generic CF environment is enhanced by COMPOSE additional capabilities. In Figure 1 COMPOSE specific additions which are state-full and accessible only to COMPOSE internal entities are denoted as COMPOSE customized services. The COMPOSE additional capabilities which are state-less and should serve external clients as well are denoted as COMPSE internal components. Finally, COMPOSE hosts also user applications, such as the pilots, which can make use of the added COMPOSE specific capabilities. In order to ease the introduction of the various COMPOSE customized services into the platform we have developed a Universal Service Broker, which eases the task of the new D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 7 of 19

8 COMPOSE customized service provider, as well as easing and mainstreaming the life of the platform provider. All COMPOSE components, such as services and service objects, are hosted or exposed via the platform, either as services or as applications. In addition, external developers applications are hosted as well on the COMPOSE platform, and can bind to COMPOSE provided services. Once the general architecture has been agreed upon and the choice was made to make use of Cloud Foundry as the underlying cloud infrastructure the main design choices refer to the proper break-down of all the rest of the COMPOSE components to fit the COMPOSE platform internal architecture. The criteria for making these decisions depends on the accessibility required from the component, since COMPOSE customized services can be accessed only within the platform, while COMPOSE internal components hosted as CF applications can be accessed by external entities. Additional considerations relate to state information, and the amount of management required, namely whether an existing installation is to be used or whether COMPSE should install and manage the component for the provider. 1.1 Criteria for selection of technologies, Important features and Fit with the project requirements This task is about the design and implementation of the run-time component within the overall COMPOSE architecture. This component is designed to host all the platform ingredients developed in COMPOSE. Both internal components, such as data management, service management, and security, as well as user / developer artefacts, such as service objects, and applications. In addition this component supports the overarching goal of COMPOSE which is to ease the task of IoT application developers, while making sure that the resulting platform is scalable. The COMPOSE runtime infrastructure consists of a customization of an open PaaS cloud platform for the IoT domain. COMPOSE has to deal with a large number of connected objects, applications, and users. The natural choice for hosting such an environment is a cloud based environment. In addition, COMPOSE aims to help developers when creating new IoT applications. Thus, the natural choice for COMPOSE is based on a PaaS environment. In a PaaS environment the developer is relieved from most of the systems aspects of deploying and hosting his applications on the cloud, such as monitoring, disaster recovery and deployment / lifecycle aspects. COMPOSE run-time, as an IoT PaaS, proceeds one step further and provides developers with specific services which are required for developing, running, and hosting IoT applications. In addition the COMPOSE run-time provides collaboration services that enable applications to collaborate and application workflows to be deployed based on individual standalone applications. When looking for an existing cloud PaaS to base the COMPOSE run-time upon we were looking for the following criteria: Ease the task of a developer deploying an IoT application; support for popular web based programming languages such as JS; middleware available with the platform, and ease of extending available middleware; IaaS neutral, open, with strong D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 8 of 19

9 governance and a vibrant community. For all the reasons mentioned above we chose Cloud Foundry as the basis for the COMPOSE platform run-time environment. Other platforms we looked at include Amazon, Azure, Google Apps, and Heroku. Moreover we needed an open source platform that can be modified for our purposes, and CF fulfils this criterion as well. Figure 2: The road to an IoT PaaS 1. Within Cloud Foundry there are two major kinds of entities, namely services and applications. COMPOSE components need to be architected accordingly. CF applications provide Web access, but reside on stateless VMs, thus having no permanent file system; their state storage is supported through cloud storage services (such as DB, NoSQL) bound to the applications. On the other hand CF services reside on statefull VMs, but have no direct web access. Only CF application can access these services through the internal cloud binding mechanism. This division fits well with the COMPOSE architecture, and all COMPOSE components were designed to fit this model. The resulting architecture can be seen in Figure 1. Developers interact with the COMPOSE platform via the developers portal, which is external to the cloud, and communicates with the COMPOSE controller. The developers portal is connected to backend components hosted in the cloud that manage the applications created by the developers, while augmenting and extending it to facilitate the consumption of IoT services. The COMPOSE controller is an adaptation layer that resides in front of the CF cloud controller. It delegates most actions to the CF-controller, but also mediates COMPOSE specific actions, like the management of Service Objects, additional COMPOSE Security functions, etc. In addition it also registers every service object and application within the COMPOSE service registry. Related requirements which are met by the current choice include: Scalability Choice of a scalable cloud run-time solution 1 Adapted from Accelerating your Journey to Application Transformation, EMC World 2012 D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 9 of 19

10 Usability - reduce the complexity of writing and deploying IoT applications the developers portal is integrated with the cloud environment and with a click of a button an application is deployed to the cloud. o Ease of development applications created in the Developers portal are not written from scratch but rather use scaffolding and pre-built components. o Autonomous deployment- language support and environment is automatically created by the cloud. Semantic support the run-time platform hosts the service discovery component which serves as the basis for semantics support t throughout the platform. This component is hosted as a service and enables the COMPOSE controller and the developers portal to communicate with it for inserting new information regarding COMPOSE entities, as well as query for available relevant information. This serves as a basis for the recommendation and composition components as well which are served by the cloud run-time. Support the hosting of applications written in popular web programming languages the first language supported by COMPOSE is JavaScript. Provision platform middleware services (e.g. DBMS, NoSQL, Messaging), and enable easy integration of apps with them A plethora of middleware is provided with the COMPOSE platform for applications to bind to them at run-time. Allow platform services to be extended (i.e. add a custom data-store, messaging solution). The COMPOSE platform adds specific IoT related services to the plain vanilla cloud platform, and enables applications to bind to them at run-time. These services include customized data management, service discovery, and security. Be IaaS neutral, i.e. support hosting on multiple cloud IaaS vendors, or on bare metal, to prevent vendor lock-in. Open with a favourable license. Security the COMPOSE run-time provides ample support for an all engulfing security solution: o UAA re/use account and authentication module o Identity management o UAA services for application to - application security o Code analysis before deployment o Custom buildpack for data provenance & security, information flow control D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 10 of 19

11 Monitoring The COMPOSE run-time platform presents its own monitoring infrastructure to collect and disseminate information on all its relevant entities, using built-in cloud monitoring information as well. 2 Prototype overview - short description The prototype demonstrates the various capabilities of the COMPOSE run-time platform. The cover story is an application that detects impending and actual collisions of airplanes. The application marks these on its real-time interactive map, and stores collision events in an internal DB that can be presented to the user as well. An outline can be viewed in Figure 3. Figure 3: Demo outline The following components are highlighted in the COMPOSE run-time demo: Universal Service Broker which is used to easily add customized services to the platform. Customized Services the COMPOSE communication infrastructure is added as a service to the COMPOSE platform. It is introduced by the universal Service Broker, and is being bound by COMPOSE applications using it to communicate information between them. COMPOSE applications being run and hosted by the platform. Bound to standard and COMPOSE specific services. D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 11 of 19

12 o Collision detector - Collects sensor coordinates, detects imminent collisions, and publishes collision warnings and sensor coordinates via the COMPOSE communication service. The collision locations are saved in a MySQL database. This application binds to the communication service and to a MySQL DB instance o Demo frontend application - Subscribed to updates from the communication services. Receives requests from the demo applet for sensor coordinates and collision history. This application binds to the same communication service and to a MySQL DB instance o Demo simulator - Performs REST calls to the Service Object API endpoint to update sensor s coordinates that simulate planes locations. External applet that connects to the COMPOSE frontend application and presents the current coordinates on a map, including an indication of near or real collisions. It can also query the COMPOSE frontend application for a history of collisions and presents them to the user. 2.1 Architectural responsibilities and interactions As can be seen in Figure 1 the COMPOSE run-time component lies at the heart of the COMPOSE platform, and all other COMPOSE components need to interact with it. Some of the COMPOSE components will be connected to the platform as COMPOSE specific services, and will be connected via the universal Service Broker. An example of such a component is the COMPOSE communication infrastructure. Some COMPOSE components will be broken down into a back-end component running as a CF service and a front-end component running as a CF application. Examples of this include the COMPOSE data management layer, and the COMPOSE discovery capability. Some components will have to reside and interact with the underlying cloud infrastructure itself. Examples of that include the COMPOSE controller and some of the Security components. Finally some components will be hosted as COMPOSE applications along with external developer provided applications. 2.2 High level design Main functionalities introduced Internally a standard Cloud Foundry installation was enhanced to accommodate COMPOSE specific requirements, which represent IoT requirements in general. These enhancements include a Universal Service Broker to add specific COMPOSE services to the cloud run-time. For this demonstration the scalable communication infrastructure was incorporated as a service to the run-time environment. D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 12 of 19

13 2.2.1 The Universal Service Broker The service broker is a mediator between service providers that wish to offer new capabilities to CF applications on the one hand, and Cloud Foundry on the other hand that hosts the applications, and mediates between the applications and the services. In most cases each service provider implements its own service broker, which takes a toll on the service provider and on the cloud foundry environment management team in the form of integration overhead. Since COMPOSE aims to ease the life of developers, customized service providers, and platform managers, we opted for the creation of this Universal Service Broker which is controlled by the platform provider. The COMPOSE run-time cloud based installation comes with a universal service broker preinstalled, which requires from the service provider only a couple of scripts that interact directly with the service provider itself. This solution enables to integrate into the COMPOSE platform cloud any kind of service, while minimizing the demands from the service provider and removing as much customization work as possible. The universal service broker consists of two parts, namely a servlet as the front-end and a backend. The front-end is a thin Tomcat servlet that is stateless, and thus can be hosted as a Cloud Foundry application. The servlet receives the standard REST requests from the Cloud Controller, parses them and forwards them to the Service Broker s backend (USBB) component which is a java application that uses a Derby database as its data store (embedded by default, but can also run as a network server). The servlet communicates with the backend using RMI. The backend receives the request, and executes the relevant script provided by the service provider, then reads the output of the script s execution (whether the execution was successful or failed, etc.) and sends the response back to the servlet, which passes it back to the Cloud Controller. Figure 4: Universal Service Broker Architecture The universal service broker also removes management overhead from the cloud administrator. When a new service is added, it automatically detects this and updates the cloud controller. D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 13 of 19

14 It is also very easy to deploy. The front end is a tomcat servlet which is deployed in a standard fashion, and the back end is a native java application that installs its database upon first launch COMPOSE Customized Services The cloud run-time environment comes with two kinds of services: Generic services, such as MySQL database (shared databases and private ones alike), apache Derby database, etc. Custom built services, that are unique to COMPOSE, such as the COMPOSE communication infrastructure, a service that provides easy and scalable communication between applications. The COMPOSE communication service is a highly scalable, light-weight, fully-distributed, network-aware communication middleware that has self-organizing structured overlay properties. It provides publish/subscribe messaging to applications, such that applications don t need to know the host and port of fellow peers, they only need to know the name of the topic\group they send\receive the information to\from, and the infrastructure takes care of the low level communication tasks such as routing the messages between the peers of the overlay, and forming a fault-tolerant, strongly connected topology. In the current prototype the pub / sub infrastructure is used to communicate events of interest between 2 COMPOSE applications running within the cloud Pushing applications and binding to services When a developer wants to upload an application to the COMPOSE cloud platform, it registers the new application and then uploads the binaries\code and relevant resources the application needs to the Cloud Controller, where the data is saved in a central blob store. This procedure is called pushing an application. After the binaries are uploaded to the COMPOSE run-time, it detects the type of application (ruby, node.js, java, etc.), and installs a matching environment that launches the application (this is called a buildpack ). An application can utilize external software in the form of services. In Cloud Foundry, binding is being done between service instances and applications, and when a user binds an application, the Cloud Controller contacts the appropriate service broker for credentials (host, port, username & password, etc.) and injects the credentials in an environment variable inside the application. This procedure provides the application with the required credentials in order to interact with the service while maintaining data confidentiality Data Services In addition, the current prototype makes use of data services provided by BSC, in the form of a Service Object endpoint to which an application feeds simulated coordinates, and COMPOSE D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 14 of 19

15 applications extract and use that data. The COMPOSE service object endpoint exposes a JSON and REST based API for creating, updating, and obtaining date from service objects The run-time environment: Ubuntu LTS Cloud Foundry build 2222, single instance using bosh_nise MySQL and Service Broker backend running on the same machine, while frontend running as a Cloud Foundry application. Compose Communication service running on 3 Linux blades Cloud Foundry instance managed from a win7 machine with cf-cli installed Service Objects API running in BSC serviced at 3 What is being demonstrated The COMPOSE run-time demo simulates real time coordinates of flying objects like planes, and sends them to the COMPOSE service REST API endpoint. Each sensor has a related Service Object in COMPOSE. The Service object is queried by the collision detector for updated coordinates, and if a collision is detected a record is inserted into a MySQL database. The updated coordinates are passed on to the COMPOSE communication infrastructure, which passes the data to the COMPOSE frontend-demo application. This application caches the updated coordinates and collision history taken from the database, and presents the information to the user. Both the collision detector and the COMPOSE frontend-demo application are hosted as Cloud Foundry applications which are bound to the same service instances (COMPOSE communication and a MySQL database) before their deployment to the cloud. 4 External API - for use by 3rd parties The Universal service broker executes scripts supplied by the service provider and returns their output to the Cloud Controller. For each service, its service configuration file (named like the service name and suffixed with.yml) states where to find the scripts that are executed by the USBB upon provisioning\unprovisioning\bind\unbind requests passed by the servlet (originated from the Cloud Controller). The scripts need to comply with a minimal set of rules: All scripts need to output whether their invocation was successful by outputting the tagged string: <BROKER>result=SUCCESS</BROKER> to standard output. The service broker ignores all output not surrounded by <BROKER> </BROKER> tags. D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 15 of 19

16 The provision script also needs to output <BROKER>instanceid=somevalue</BROKER>. This value is the value the service broker uses in order to translate the CF provided unique instance ID to the service provider's instance upon unprovisioning. The bind script needs to output a similar field: <BROKER>bindid=somevalue</BROKER> for the same reason. Besides that, since the bind operation is used to connect the CF application to the new service being provided, the bind operation in CF actually simply sets the VCAP_SERVICES environment variable that contains all of the credentials and connection information an application needs in order to connect to the service. These credentials (if there are any) should be supplied by the bind script, and the service broker simply passes them on to cloud foundry in the bind response. Each credential should be in its own line, for example: <BROKER>username=testuser</BROKER> <BROKER>password=P@$$W0RD</BROKER> Every key=value combination whose key is not instanceid or bindid will be passed to the VCAP_SERVICES as a credential. Supported value types are strings and integers only. 4.1 Installation & configuration In order to deploy the backend, simply launch the backend application with the command: java -Dconf.init=true -Djava.rmi.server.hostname=server-ip -jar broker.jar conf.xml &> broker.log with the server s ip address. This automatically creates an embedded derby database instance, a configuration file (conf.xml), a services directory and a template service with a configuration file (test_service.yml). Use this template service to create new services. Upon start-up, the service broker tries to update the Cloud Controller with the services it possesses. After the first launch, the backend configuration file needs to be edited to suit the environment and the backend process needs to be re-run in order to reload the configuration. In order to add a new service all that is needed is simply to create a directory with the same name as the new service (under the main CF services directory) and create the YAML configuration file and the appropriate scripts (and of course- have the scripts listed in the configuration file). There is no need to restart the service broker in order to add new services. It is important to deploy the broker backend first and the servlet only later, since if the servlet starts up and fails to connect to the backend, it won t try again(but will try again if the first attempt of connection since start-up was successful), so make sure the standard output log file has the lines: Connecting to rmi:// :5611/broker Connected, before continuing with further steps. When you re done, it s time to introduce the service broker to the Cloud Controller. This is done by invoking cf add-service-broker and giving it the URL of the servlet (depends on where and how you deployed the Tomcat server). After adding the service broker, you should be able to provision and bind your new services. D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 16 of 19

17 Binding applications to a service can be done at application creation time, as can be seen in Figure 5, or after the application has been created, using the cf-bind-service command, as can be seen in Figure 6. D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 17 of 19

18 Figure 6: Binding an application to a service at "push" time Figure 5: Binding an application using cf commands D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 18 of 19

19 5 Future directions Incorporation of the COMPOSE controller and service management component into the COMPOSE platform. This will streamline the management of user provided COMPOSE applications within the COMPOSE platform. Deployment of additional services. First and foremost, the data management component. Next in line is the discovery mechanism. Introduction of the COMPOSE security components into the platform. Integration of the COMPOSE pilots to be run within the COMPOSE platform, as a validation point that all necessary capabilities are in place and functioning well. D4.1.2-Basic implementation of the COMPOSE runtime infrastructure Page 19 of 19