Industry Collaboration: Remote Monitoring of a Cloud-Based System Using Open Source Tools. Abstract

Transcription

1 Industry Collaboration: Remote Monitoring of a Cloud-Based System Using Open Source Tools Ana E. Goulart, Nishanth Prabhu, Tyler Covington Electronic Systems Engineering Technology Program Texas A&M University goulart@tamu.edu, nishanth.prabhu791@gmail.com, covi500@tamu.edu Chris Corti IP Communications Business Unit Cisco Systems, Inc. ccorti@cisco.com Abstract Internet of Things (IoT) is often associated with remote monitoring. There are three parts to remote monitoring: sensors, data collection, and visualization. The sensors, or probes, perform measurements. For engineering undergraduates, temperature and motion detector sensors are familiar examples of such measuring devices. Data collection requires a communication network to transmit the measurements and a database to store them. Students have used short-range wireless links and computers connected to the Internet to store and organize the data. Last but not least, visualization informs the system status in a user-friendly way, using a graphical interface. Industries in the energy, environmental, logistics, and surveillance fields have widely employed remote monitoring. However, this past semester, we worked on an industry-sponsored research project to monitor the status of a cloud-based system. This system has several virtual machines, each responsible for applications such as databases, web services, and authentication. The virtual machines are hosted in an Internet cloud, away from the service provider. Our challenge was to monitor these applications. It was also an opportunity to expose our undergraduate students to a new type of remote monitoring system, one that is closely related to the Internet. This paper presents how we implemented this system using open source tools. The sensors were implemented by our customer, who provided us with an application programming interface (API). The data collection and visualization tools include Java script object notation (JSON), representational state transfer (REST) interface, a model/view/controller web application using Groovy/Grails, and HTML5. This paper provides a case study of how we combined these tools to monitor a cloud-based system. Lessons learned from the perspective of student and industry are also discussed. Introduction A big picture of the Internet of Things (IoT) was given in [1]. This big picture includes three components: sensors, data collection, and visualization, as can be seen in Figure 1. The data comes from the sensors or any other monitoring device, such as temperature sensors or accelerometers. The volume of data may be large, depending on the number of sensors and the sampling frequency.

2 Therefore, the data collection component can be quite complex, for it includes data processing, management, and storage. The data processing converts a large volume of raw data into a more meaningful metadata. Finally, the visualization means the user interfaces that present the processed data to the users. This last component is essential, as it can timely display the collected data in a visual and clear format. Thus, visualization allows quick detection of problems or alarms in the monitored system. Figure 1. Internet of Things (IoT) big picture [1]. The idea of Internet of Things (IoT) is not new, if we consider that some type of remote monitoring has been done in many industries, such as surveillance, logistics, oil and gas. Maybe the new idea is that the sensors are connected to the Internet, and use the Internet Protocol (IP) to send data packets to the data collection components, such as servers and databases. Nonetheless, it is important to expose undergraduate engineering students to this concept, to the different systems that apply this idea, and to the tools that are used to build such remote monitoring, or IoT, systems. This paper describes a project in which engineering students developed the data collection and visualization components that monitor an IP telephony system that is hosted in a cloud [2]. The novelty of this project was that the system to be monitored was a not a physical device, but a group of virtual machines hosted in a remote location - a cloud - providing multiple components, such as databases, web services, call processing, and authorization services for the telephony application. This project, named SystemHealthDisplay, was part of an industry-academia collaboration between Texas A&M University and Cisco Systems. Four undergraduate students and one graduate student contributed to the project, under the advice of a faculty member and a software engineer from Cisco. The next sections describe the problem statement given by Cisco, the solution implemented by students and the tools they used, with some retrospective discussions about the project at the end of the paper. Problem Statement As applications and services are moved into the cloud, engineers are responsible for monitoring their health and fixing problems as they occur in real time. One alternative for monitoring a cloud system is similar to the way a production factory control room monitors the flow of output from station to station as raw material is processed from beginning to finished product. The control room may display the stations on a monitor as boxes, each containing relevant information, such as temperature, and activity state. If any station reports a problem, the display in the control room might indicate that by coloring the box red. This allows personnel in the control room to quickly

3 notice the change and take action. Similarly, the cloud model has numerous components that need to be monitored for health status. For instance, a database may have a primary and secondary component on separate virtual machines (VMs), or nodes. Likewise, the Infrastructure as a Service (IaaS) layer also has primary and secondary VM nodes. Eventually, we want to display the health status on selected monitors across Cisco sites, except that the layout would need to be color coded according to current process health state and probably should embed some critical state information. The presentation should also be available on a web browser, and provide a way to interact with it, for instance, by clicking a box to drill deeper into its subcomponents to investigate a root cause of any detected problem. Design Approach Our solution was to implement a simple display (i.e., a TV monitor) to indicate each cloud component s health. It contains a list of critical components with a color-coded background identifying the current state, as indicated in Table 1. Table 1. Legend for color-coded states. STATE COLOR Online Fault Offline Unknown Green Yellow Red Grey The state of each component is captured by sending pings, i.e., by polling each component through a representational state transfer (REST) interface. Cisco Systems provided an Application Programming Interface (API) called ping API which returns the status of the cloud components. The ping API determines if a component s state is fault or offline by monitoring its service status. For instance, assuming a database component, if there is no database process running on the virtual machine, then that component is offline. If there is a database process, the ping API does a simple select on the table. If the select returns a row, it is healthy ( online ), otherwise it is faulty. If there is no response from the ping API (e.g., the ping API has not been implemented), then the state is unknown.

4 In other words, the sensor part of our remote monitoring system is provided by the ping API. The ping API returns Java Script object notation (JSON) output for a given ping to a web address (i.e., Uniform Resource Identifier (URI)). The JSON response can be parsed, stored in a database, and displayed as a web page (i.e., in Hypertext Transfer Markup Language (HTML) format). Figure 2 provides an overview of this process. At the top of Figure 2, we can see a ping request and a sample of the JSON response that is received from the Cisco cloud (shown as steps 1, 2 and 3). Note that the client application pings the cloud periodically. In the current design, it pings each cloud component every five seconds. Figure 2. HealthSystemDisplay architecture. The state of each system is stored in a database, with its corresponding timestamp. The information in the database needs to be displayed in a display (i.e., TV monitor or web page) that can be accessed from any Cisco site. Our design approach was to use Groovy/Grails [3, 4] to create a web- based application. It uses the concept of Model/View/Controller (MVC), where the model or domain is the database, the view is the HTML page that is displayed to the user, and the controller makes most of the decisions and sends commands (or actions ). For instance, for each view there is an action in the controller. Note that the view is also refreshed periodically to display the latest data saved in the database (currently set to refresh every 15 seconds).

5 SystemHealthDisplay To illustrate how the SystemHealthDisplay application was implemented in Grails, this section provides an overview of the Model, Controller, and View modules, including the Job scheduler to send the ping requests periodically to the cloud. Model A Java class called SystemHealthDisplay was created in Grails. It defines the model or domain. Once this class is created, Grails creates the database schema for each specified domain. In other words, for each cloud component, the database contains the following columns: service name, time stamp, state, ping URI, as shown in the class SystemHealthDisplay (Figure 3). The name, time stamp and state are required fields. Not all cloud components have a corresponding ping URI. For instance, the state of some components can be obtained by pinging another component, and it will show as an upstream service (please note the JSON array named upstreamservice as the last line in the JSON example of Figure 2). Therefore, the ping URI field may be empty for a given component. Figure 3. HealthSystemDisplay data model. As a sample of the database, its header is given in Figure 4. For confidentiality purposes, the service names and ping URIs currently stored in the database are not displayed in Figure 4. Figure 4. Database view of SystemHealthDisplay.

6 Controller The controller is composed of three parts: 1) the client code, which sends the ping URL request (i.e., a Hypertext Transfer Transport Protocol (HTTP) request) to the Cisco cloud (PingAPIGet); 2) the JSON parser, which parses the content of the JSON response and saves it in the database (PingAPIReader); 3) the SystemHealthDisplay controller, which has actions with corresponding views. For instance, the showhtml action creates sorted lists from the database and passes it to the view. The sorted lists in the SystemHealthDisplay controller classify and group the cloud components in three categories: Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and Authorization Services. In addition, our Grails application has a HealthDisplayJob file that is triggered every five seconds (Figure 5). It includes a plugin called Quartz [3] which allows job scheduling. Figure 5. HealthDisplayJob is a job that is called every five seconds. The trigger affects the execute() method inside this file. Following the triggering function inside the HealthDisplayJob, there is an execute() method. The flowchart shown in Figure 6 provides an overview of the code inside the execute() method. It goes through each SystemHealthDisplay object stored in the database, sends the ping command to the cloud, parses the JSON output, and saves the updated information in the database.

7 Figure 6. Actions that are executed every 5 seconds to collect the alarm data. View The HTML view is created using Grails Groovy Server Pages (GSP) files. Its syntax is similar to HTML, but it links the view with the actions in the controller. In other words, views can be called when a button is pressed, or values from the controller can be passed to the view. Our main view is called showhtml. In one of the first lines, we have a simple expression that allows this web page to be reloaded automatically every 15 seconds: <script> settimeout(function(){location.reload()},15000); </script> This was an important requirement of the project, for the page may be displayed in a TV monitor, without an active user pressing the refresh button of the browser. The display was implemented using HTML5 Canvas [5] to display the cloud components and their state in the browser. More details about HTML5 Canvas are presented in the Retrospective session in this paper, and it was written by one of the students who developed the Grails view. The view receives three lists from the controller (PaaS, IaaS, and Authorization Services list).

8 For each list, the controller has an initial (x, y) coordinate that defines where they should be drawn in the page. The result can be seen in Figure 7. Note in Figure 7 that the view displays the Production version of the cloud. There is another Grails application developed by the students with the same functionality, but a different database name and contents (such as ping URI), for the Integration version of the cloud. Cisco Systems uses a development process called DevOps, in which the cycle from development to operation is very short, maybe hours or days, until the latest cloud system is available to the customers. That is why there are Integration and Production versions; both need to be closely monitored by its engineers. Figure 7. SystemHealthDisplay production view. Project Retrospective Industry The concept, design and development of the SystemHealthDisplay occurred in an unusually haphazard way, and with an aggressive schedule. Usually the flow from concept to deployment follows an internal agile process: The concept is encapsulated in a high- level User Story, Product owner evaluates its priority, Subject matter experts flesh out requirement specifications, creating further refined user stories.

9 Once specifications are clear, low- level design and implementation are turned over to the development teams. The SystemHealthDisplay began as a request from management to create a display on in- house TV monitors to show the current state of the new cloud- based telephony system. Logs from open source tools such as Kibana and Sensu should be used, in order to react to change events in the system components. Results had to be displayed in labeled rectangular tiles on a TV monitor, each rectangle representing one component, colored red or green respectively as the component is failed or operational. With that terse set of instructions, we initially divided the students into groups, one to focus on how to represent results as a web page that could self- refresh, and another to investigate interacting with Sensu/Kibana to receive event notifications. The students even deployed an OpenStack cloud testbed with Sensu in order to experiment with that framework. Within three weeks, however, management changed our direction. Instead of relying on internal event reports, we were to use a set of external Ping APIs being developed by some engineering teams. This meant adjusting the scope of one intern group to developing tools for retrieving information in JSON format that would be returned from the Ping, and developing a backend store to maintain the mapping of components to their particular Ping URIs. Our first obstacle was getting commitment about the format of the returned JSON objects, and their semantic meaning, i.e., the possible states and their meanings. Without written documentation explaining that and the requirements of the Ping functionality itself, we were left to make reasonable guesses. For instance, we expected that each component would implement a Ping API and provide a URI to use; and we supposed that there would be three states: Failed, Online, Not- Implemented. As the telephony cloud application finally neared its first production phase, our work became very important as upper management wanted to see a simple display of system health. Finally, we were provided the answers that we needed in the form of a brief document. We discovered that while some of our guesses were correct, some were not. The document explained the various system states, which corresponded closely to what was implemented, but some new requirements were discovered: each Ping API also tests the health of any upstream components on which it is dependent. our code would need to rely on upstream results of the Ping to determine the state of other components. only a subset of components expose a Ping API. Fortunately the new specifications were fairly easy to incorporate so that integration was not difficult. One new requirement was that, instead of having colored rectangles on the web page, the page should organize components into PaaS (Platform as a Service), IaaS (Infrastructure as a

10 Service), Authorization Service, and provide a key explaining the colors. A border around the IaaS portion was also requested. We could have used some mentoring from in- house experts to suggest possible implementation mechanisms, because we had no expertise in that area. We again split the group to have one investigate Scalable Vector Graphics (SVG) using, for instance, Data- Driven Documents ( and another to look into HTML5 Canvas. The latter was selected, and within a short time, we had the System Health Display on TV monitors throughout the company. It is understandable that product features are of high priority; however, looking forward, it was clear that for DevOps, such a display was needed, and that the Ping APIs, which were immune to internal failures that might be experienced by Kibana or Sensu, would be important. The DevOps part was indeed integral to development, since daily deployment was of high priority. This means that our work should have been afforded the same Agile treatment that feature development was given. Had this been done, our work would have been more efficient and we would have been able to ask probing questions of the Product Owner to suggest alternatives and possibly improve the final product. In fact, without having input from a Product Owner and giving demonstrations to the internal customer resulted in a longer time to complete, as one would expect. Project Retrospective Student 1 During our time developing the monitoring tool for the cloud- based system at Cisco Systems, we gained experience with a wide range of open source tools. With a vision in mind of a monitor on the wall which could be easily seen and indicate the health status of various systems, we began to research efficient ways to dynamically display incoming status updates. The storing, interpretation, and display of the system updates were handled by a series of Controllers and Displays in a Groovy/ Grails Tool Suite Project called SystemHealthDisplay. Our initial attempt to dynamically display the database of statuses was run through a Google Visualization Tool called the Treemap, which was discovered after researching the open source tools at our disposal. What first drew us to the Treemap was its ability to display a given array of numbers in a color- coded fashion. After some time, the project needed to take a step towards becoming a grouped and well- arranged display rather than a prototype,. The Treemap proved to be insufficient. At this time, a variety of tools was recommended by online resources and advisors at Cisco. We were led to the HTML Canvas. This tool appealed to our desire for a display in which systems can be arranged and grouped according to the type of system dynamically. An x- y plane was defined using measurements of a typical internet browser and inside this grid we were easily able to indicate start points of where each system should be displayed in relation to its grouping. The health of a system was related to a color scale in which a healthy online system was displayed as green, a system that was offline as red, an unknown status as grey, and a system at fault as yellow. In this process, many lessons were learned such as how many open source resources are

11 available for use and the manner in which updates can be handled and interpreted using these tools. In doing this, a wealth of knowledge was discovered online with a variety of tools accompanied by instructional videos and tutorials. This made a conversion to this new, html based display tool, HTML5 Canvas, quite smooth. PingAPI was used to obtain a JSON file containing fields, which indicated the health of systems at the time of the update. This experience explored retrieving values from the JSON file and using this data to update a database, which was used for the display. Many aspects of the project went quite well and operated smoothly after developing a solution to the task at hand. The HTML5 Canvas tool immediately integrated into our system and accomplished the requirements of dynamically updating and easily manipulating the position of images in the graphic. Using a grid style approach and an x- y axis, HTML Canvas allowed us to easily position the systems that were to be displayed and grouped into sections which fit the desired look of the graphic. Additionally, the Groovy Grails Tools Suite and a groovy project proved itself to be a very powerful tool in retrieving data from a database and moving the values to the correct pages for reference. Due to the project s ability to quickly update and continue running while making changes, the graphic display was easily edited and tested with the incoming status updates. The aspects of this project that could be improved would be a more professional style display or a faster updating system. The current display is the result of simple fonts and a line drawing system, which creates the sections of the Canvas to represent different systems. Ideally we would develop a sharper looking display that directly correlates entire to a value. In a more advanced system, faster updates would take place to ensure that all systems stay functional in order to keep the infrastructure behaving properly. Project Retrospective Student 2 SystemHealthDisplay is a cloud- monitoring tool used to monitor the state of the virtual machines or services in the cloud. This tool is open source as it is developed using Groovy/Grails framework. Before actually implementing this tool, we did a lot of research on cloud. We created a private cloud in the lab at our university by installing Ubuntu Operating System and DevStack (a development version of OpenStack). We were also able to access the DevStack dashboard from any of the desktop computers in the same local area network (LAN). We were also able to create virtual machines and access Internet from virtual machines. This was our first step in understanding the cloud. After doing lot of research on various existing cloud monitoring tools, we started to implement our version of cloud monitoring tool, using Groovy/Grails which can be directly used in Production environment by Cisco Systems. While developing this tool, we learned how to create the HTTP client in Groovy/Grails to implement the Ping API that can be used to ping the various services which gives JSON file as the result which is stored in the database. We also learned how to create the scheduler in Groovy/Grails so that we can call the Ping API every 5 seconds to get the latest JSON file and update the database if there are any changes. Finally, we learned how to create the graphical user interface (GUI) using HTML5 canvas, which displays

12 the status of various services in the cloud by reading the database. Our version of cloud monitoring tool - SystemHealthDisplay - is simple to use and easily scalable compared to existing open source cloud- monitoring tools. Currently, we have the view that is developed for Cisco Systems. In order to make this tool more generic and to be widely used by all the cloud monitoring companies, we may have to create a generic view, which will be one of the improvements or the future work. Conclusions The HealthSystemDisplay application relied on open source components, and exposed undergraduate engineering students to practical concepts used in industry such as Internet of Things or remote monitoring, cloud systems, DevOps, and web- based applications. Moreover, given a generic problem statement, which did not provide many implementation details (as shown in the second section of this paper), and a dynamic and fast- paced environment in the industry side, the students, faculty, and Cisco engineer worked as a team to develop a tool that helped Cisco s internal operations. The team responded to requirement changes, researched new tools, and explored different possibilities. This was a great learning experience for the students and the faculty member. References 1. Vatjus-Anttila, J., 2015, Internet of Things: a Big Picture, presented at IIT Real-time Communications Conference, Chicago, Illinois, October 2015, 2. Mahjoub, M., Mdhaffar, A., Ben Halima, R., and JMaiel, M., 2011, A comparative study of the current Cloud Computing Technologies and Offers, Proceedings of the First International Symposium on Network Cloud Computing and Applications, Davis, S., 2008, Groovy Recipes: Greasing the wheels of Java, Pragmatic Programmers, Grails, 5. HTML5 Canvas,