4th Bachelor of Science (Information Technology) Android Application for Monitoring Website Status and Statistics. Project Definition Document

Transcription

1 4th Bachelor of Science (Information Technology) Android Application for Monitoring Website Status and Statistics Project Definition Document Studen t Eric Bannon e.bannon2@nuigalway.ie Academic Supervisor Dr. Michael Madden 15/10/2010

2 Table of Contents 1 Introduction Project Brief Overall Objective Setup Monitoring Systems Develop Application Relevant Technologies Monitoring the Web Server Status Ping Smokeping Telnet Comparison Monitoring Web Server Statistics Analog AWStats Webalizer Comparison Development Java Bash Requirements Functional Non-Functional Architecture & Design Statistics-Sharing Web Service Monitoring the Web Server s Status...10

3 6.3 Application Backend Project Management Deliverables and Milestones Work Breakdown Structure Gantt Chart Shareholders Developer Dr. Michael Madden National University of Ireland, Galway Risk Analysis Bibliography...19

4 Introduction 1 Introduction In this document, I will outline my final year project; detailing goals, requirements, technologies and project management required to develop the application in the given timeframe. 2 Project Brief The application is designed to run on an Android smartphone. The term smartphone comes from the fact that such mobile telephones have more powerful CPUs, a significant amount of memory and storage in comparison to older generations. Smartphones offer extra functionality over their predecessors, including multitasking applications, locationbased services and web/ access. The primary goal of this project is to produce a smartphone application which enables a webmaster to know if their site is still online ( up ) or not. It will also provide usage usage statistics for the site, in both graphical and textual form, in as close to real-time as possible so the webmaster can react promptly to restoring service. The application should make use of push notifications for passing monitoring messages to the webmaster. 3 Overall Objective 3.1 Setup Monitoring Systems Pre-existing software will be used to monitoring the web server in order to generate usage statistics for the site running on it. A small program will be developed to periodically check the web server and alert its webmaster(s) if it does not appear to be online. 3.2 Develop Application The data generated above will need to be accessible to the application running on the smartphone. The text and graphics generated on the device should be updated in as close to real-time as possible so that the webmaster(s) can respond to an issue quickly. 1

5 Relevant Technologies 4 Relevant Technologies 4.1 Monitoring the Web Server Status Ping Ping is a useful command-line network utility that comes pre-installed with the most popular operating systems. Assuming the remote machine does not deny pinging, it is the simplest approach to checking if a remote host is online, in addition to providing statistics about round-trip times and packet loss (Figure 1). An Internet Control Message Protocol (ICMP) echo request is sent to the remote host and a reply is requested. The outcome of requesting a reply determines if the remote host can in fact be reached. A network error along the path between the two hosts may cause pinging to fail (e.g. substantial packet loss). In terms of implementation, a script could attempt to ping a web server and examine the program s exit code to determine if the remote host is reachable. Figure 1: Pinging the IT Department's web server with five attempts 2

6 Relevant Technologies Smokeping This web application builds upon the ping utility but also provides graphical trends of uptime, round-trip times and packet loss. Installing it requires Perl and a web server that supports CGI scripts (Figure 2). Figure 2: Smokeping graphing packet loss and round-trip time for a host over a period of fifteen days Telnet Telnet is used to connect to a host and execute commands on it. It is be considered as the insecure predecessor to the Secure Shell (SSH) in that telnet traffic is sent across a network unencrypted, i.e. in the clear for interception (Figure 3). Telnet could be used to monitor the status of a website. A possible solution might be to telnet to the remote web server and then execute a command to retrieve the website s home page (e.g. index.php, Default.aspx) in HTML to the console: GET / The text returned by the command could be check for one or more expected keywords on a particular page s HTML code or, indeed, lack thereof. However, telnet access to web servers tends to be disabled in preference for SSH. Additionally, the above command can present problems when run on a web server that is serving pages using the HTTPS protocol. With these two issues in mind, it would not be entirely advisable to rely on this method for checking the status of a web server. 3

7 Relevant Technologies Figure 3: Making a telnet connection to a router Comparison Can be trusted to provide a reliable answer Has software dependencies to be installed Pre-installed with most operating systems Ping Telnet Generates Graphics Smokeping Ping can be trusted the most to provide a reliable indication of the status of a host. While graphics are a great addition for such a web application, they are of little use in this project as smaller screen resolutions found on modern smartphones prove to be more restrictive than modern monitors. 4

8 Relevant Technologies 4.2 Monitoring Web Server Statistics All the below web statistics packages parse web server log files as opposed to embedding Javascript code into every single web page of the website, as is the case with the popular StatCounter service, for example. This can offer a significant performance increase as the Javascript approach requires extra processing on both the client- and server-side to function. It is becoming more common for users to make use of browser add-ons, such as NoScript, to block such scripts from executing in an attempt to protect their privacy from tracking cookies and to help reduce page loading times Analog Figure 4: Analog statistics for the Analog website This is an open source solution for monitoring website statistics. Only textual statistics are generated by Analog unless extra software is used (Figure 4). It is cross-platform and claims to scale well a highly desirable attribute for popular websites. Unfortunately, this project appears to have been abandoned; with the last release having been in This is discouraging to see for many reasons: unless the open source community steps in, end users will see no new features. More worryingly, any security 5

9 Relevant Technologies vulnerabilities in the software are almost certain to remain unpatched, exposing the end users to unnecessary risk AWStats AWStats is a popular open source solution for generating website and bandwidth statistics. It is used by ISP HEAnet for hosting websites for up to 4,000 primary and secondary schools in the country. Compatible with many log file formats, it offers more statistics over its rivals, including native support for Microsoft s popular IIS web server, the ability to track session duration and to provide WHOIS information on hosts (Figure 5). Figure 5: Website statistics generated by AWStats 6

10 Relevant Technologies Webalizer Similar in functionality to both Analog and AWStats (Figure 6). Figure 6: Webalizer detailing hits, file requests, page loads, visits and traffic for a website 7

11 Relevant Technologies Comparison Open Source Active Development IIS Support Can output to XML for thirdparty use Command line queries supported Analog AWStats Webalizer (needs a patch) With its regular releases, support for Microsoft s IIS web server out-of-the-box and the ability to export previously imported web server log files to XML, I opt to use AWStats for gathering statistics on the web server. All three utilities support the Apache web server as standard. 4.3 Development Java Java is the development language for the Android platform. Is it, more or less, based on J2SE but with some modifications for Android-specific libraries and APIs for user interface elements (such as buttons, text fields, radio buttons) to replace Swing and smartphone features, including GPS and bluetooth functionality. The backend, which will provide real-time website information and statistics to the frontend application, will also be developed using the J2SE APIs. Additionally, the service for pulling statistics from the utility running on the web server for use on the frontend application will be coded in Java Bash The script for periodically pinging the web server will be written in the Bourne Again Shell (Bash) scripting language. This shell is commonly installed on Unix-like systems, including Mac OS and Linux. 8

12 Requirements 5 Requirements 5.1 Functional 1. Setup the Android SDK and emulator to establish a working development environment 2. Select a precise Android target, i.e. API version 3. Identify suitable free, open source software (FOSS) in active development for reuse 4. Identify if a web server is up or down 5. Provide usage statistics in both graphical and textual representations in the application: inbound and outbound traffic to and from web/database server, requests to the web server for particular files (assets) and pages, number of hits for each page 6. Use push notifications to smartphone(s) for altering webmaster(s) of problems and confirmation of the system working as expected (i.e. an all OK message), even if the application is not running on the device 7. Secure access to an Android phone for development and testing 8. Acquire sample data set of web server logs 9. Deploy necessary hardware (servers) either physically or through use of virtualisation 10. Identify or launch a test website 11. Make the entire application as general as possible to ease deployment: cross platform programs and scripts, support for IIS and Apache, an installation script of some description for different platforms 12. Facilitate persistence of a selection of user preferences in the smartphone application, e.g. what data to display, font colour 9

13 Requirements 5.2 Non-Functional 1. Investigate appropriate designs for the application s user interface to facilitate communicating the data clearly 2. Aim to minimise potential false positives, e.g. the application stating a web server is inaccessible when, in fact, it is 6 Architecture & Design 6.1 Statistics-Sharing Web Service The web statistics monitor will run on the local web server. It does not appear to be possible for such utilities to easily log to a remote host, such as the primary monitoring server. For this reason, it will be necessary to write a service on the primary monitoring server to poll the web server for its usage statics so they can be relayed to the Android application. Having one (or more) instances of the smartphone application directly polling the web server could unnecessarily add to its load so it is my aim to reduce this by having the polling service selectively pull data from the web server and then cache it on the primary monitoring server (Figure 7). However, even were it possible to have the web server s logs interpreted on a remote host by AWStats, this would mean using extra bandwidth for replicating data elsewhere as opposed to having the service supplying only the desired data from the logs. This sort of mass replication of data would also introduce a time penalty to some degree, detracting from the aim of a real-time smartphone application. This service could make use of Java s extensive API support for sockets and networking in general. Ideally, this service should require even a basic level of authentication before sharing data to deter casual snoopers and crackers. 6.2 Monitoring the Web Server s Status It will be necessary to monitor the server hosting the webmaster s site. Depending on the popularity of the site, shared hosting may be adequate or the website may even require several servers and load balancing to remain online in dealing with high demand. In either case, the hardware required for hosting the site is a given. 10

14 Architecture & Design Another independent server most likely running in a virtual machine for development will poll the web/database server to see whether it responds or not. The pinging script previously outlined could flag the server as being down on the smartphone application after 5-10 failed ping attempts. It is highly undesirable to have false positives here so repeating the attempt to ping the server may be advisable after an initial failure for reassurance. 6.3 Application Backend The backend for the application will also run on the primary monitoring server as pinging one or more devices requires only minimal system resources. Website statistics will be gathered on the local web server, using AWStats, and a service will be written for sharing this data with the application s backend on the primary monitoring server. It is also desirable to have a tertiary server (or, again, virtual machine) for resilience to ensure the machine that polls the web server is also up. Without this, the webmaster would fail to be notified should the web server actually go down, rendering the entire application and service pointless (Figure 7). 11

15 Architecture & Design Figure 7: Application infrastructure 12

16 Project Management 7 Project Management 7.1 Deliverables and Milestones Project Definition Document 15/10/2010 Website 22/10/2010 Backend Completed December 2010 Frontend Completed February 2011 Testing Completed February 2011 Final Report 25/3/2011 Project Demonstration 28/3/2011 Viva Voce April 2011 project outline a summary of the project and blog detailing development pinging script, statistics generator and service written application (GUI and logic) completed and interfacing with the backend for its data testing of full application, backend and frontend, completed a detailed analysis of the project and evaluation of results a bench demonstration of and verbal report on the project interview/oral examination of the project 13

17 Project Management 7.2 Work Breakdown Structure 1 Initial Research 1.1 status monitoring software 1.2 website statistics generators 1.3 Android platform for development 2 Project Definition Document 2.1 draft 2.2 finalise and submit 3 Website 3.1 secure webspace 3.2 select CMS 3.3 design 3.4 test 3.5 add and maintain content 3.6 add link to IT Department intranet 4 Application Development 4.1 setup development environment install Eclipse install and configure Android SDK set up necessary servers/virtual machines 4.2 monitoring systems download read documentation configure collect sample data 14

18 Project Management test 4.3 develop backend implement AWStats sharing service plan and design develop test develop backend service plan and design develop test 5 Application Frontend 5.1 further Android API research 5.2 devise UI 5.3 interface with backend 5.4 testing 6 Final Report 6.1 plan 6.2 draft 6.3 finalise and submit 7 Bench Demonstration 7.1 preparation 8 Viva Voce 8.1 preparation 15

19 Project Management 7.3 Gantt Chart 16

20 Project Management 7.4 Shareholders Developer It is my duty to ensure the end product satisfies the project brief in meeting its requirements and delivering the project on time Dr. Michael Madden As my academic supervisor, Dr. Madden will want the end product to satisfy the brief and deadlines to be met National University of Ireland, Galway The University is providing me with the necessary resources to complete the project. 17

21 Project Management 7.5 Risk Analysis Risk Explanation Impact Probability Inability to meet the requirements Overly-steep learning curve Final Year Workload and Personal Commitments Scope Creep Data Loss, Corruption or Theft Illness Underperforming at the Bench Demonstration Underperforming at the Viva Voce Failing to manage the time available wisely could results in not all requirements being met. The requirements outlined in this document have been prioritised accordingly Struggling to get acquainted with new technologies in the project s timeframe Other project work, assignments and extra-curricular activities could impact the time dedicated to the project. However, I will have fewer hours of lectures in the second semester so the overall workload should be reduced considerably Failure to interpret and fulfil the requirements obediently could easily lead to the addition of superfluous functionality Failure to have a recent backup of all project documents, code, configure files and virtual machines would be catastrophic. A suitable backup procedure will be established and tested Seasonal illness may result in some project tasks falling behind by a few days to a week A failed demonstration or experiencing technical difficulties would be less than ideal. I plan to rehearse the demonstration beforehand and anticipate potential questions I feel I have properly researched the project s technologies and will be well versed to defend technical decisions made High 10% Medium 5% High 45% Medium 10% Critical 5% Critical 15% Low 5% Medium 5% 18

22 Bibliography 8 Bibliography 1. ping(8) Linux man page. Available at: (Accessed: 14 October 2010) 2. Ping (2010) Available at: (Accessed: 14 October 2010) 3. Oetiker, T. (2008) The Detail Graph [Online image]. Available at: c/reading.en.html (Accessed: 14 October 2010) 4. FAQ (2010) Available at: (Accessed: 14 October 2010) 5. How does it work? (2010) Available at: ks.html (Accessed: 14 October 2010) 6. Maone, G. (2010) NoScript JavaScript/Java/Flash blocker for a safer Firefox experience! what is it? InformAction. Available at: (Accessed: 14 October 2010) 7. Turner, T. (2004) Analog: Introduction. Available at: (Accessed: 14 October 2010) 8. What is AWStats (2010) Available at ATURES (Accessed: 14 October 2010) 9. Statistics for: destailleur.fr (2010) Available at wstats.pl?config=destailleur.fr (Accessed: 14 October 2010) 10. Barrett, B. (2008) FAQ. Available at (Accessed: 14 October 2010) 11. Log analyzers Comparisons (2010) Available at awstats_compare.html (Accessed: 14 October 2010) 12. Hashimi, S. & Komatineni et al. (2010) Introducing the Android Computing Platform, in Anglin, S. (ed.) Pro Android 2. USA: Apress, pp Bash Reference Manual (2009) Available at manual/bashref.html#what-is- Bash_003f (Accessed: 14 October 2010) 19