Software Maintenance - Whitepaper 1 Introduction

M&M Software Maintenance Author: Thomas Weller Created: 2011-08-02 Version: 1.2 Software Maintenance - Whitepaper 1 Introduction This white paper describes software maintenance from customer s perspective. It depicts what we do, but not exactly how we do it. The focus is on maintenance for external products; that is software which was not developed by M&M. The scenario is told by an example customer like a testimonial but in long form. M&M Software GmbH +49(0)7724/9415-0 Industriestr. 5 Fax +49(0)7724/9415-23 78112 St. Georgen Email: info@mm-software.com GERMANY Internet: www.mm-software.com 1 / 9 Registered Office: St.Georgen, Germany Registry Court: Freiburg HRB 602021 Directors: Erwin Mueller, Klaus Huebschle, Andreas Boerngen

2 Scenario 2.1 Initial situation I m Cono Scenza 1 from Senstrumenta 2 in Italy. Senstrumenta is a company in automation industry which produces mainly sensors, actuators and valves. Our products use Profibus, CANopen and DeviceNet as protocols. Our 750 employees generate 80 million turnover per year and about 8 million profit. I am a software product manager located in the headquarter of Senstrumenta in Torino 3. The reason I came to M&M is our product e- catalog and collaboration platform community. Our customers can select and customize their products online, place orders and even upload construction sheets for individual fabrication of pieces according to customer s needs. This e-catalog was originally developed by the German company CADenas, a specialist for product catalogs. It is based on JSP and our servers are running Linux, Apache, the Tomcat connector and a PostGre SQL database. However, the catalog was customized by Senstrumenta to integrate collaboration features and the integration with Quicksizer, a legacy application for pneumatic calculations which integrates seamlessly with the e-catalog. Quicksizer is developed in Visual Basic. Figu re 1: Senstru me nta e - c atalog Meanwhile, Senstrumenta decided to implement a new version of the e-catalog which has Web 2.0 features, integrated 3D preview of customized devices and many more features which can be seen in competitor s webshops. The new catalog is based on ASP.NET and Silverlight and will run on IIS server. The total development time is expected to be three years and started 8 months ago. Unfortunately, a critical security issue made an update of the existing system necessary. CADenas provided an update of their base components which fixes the bug. However, with this bug fix, the API changed a lot and the integration with Quicksizer was broken. Since then, Senstrumenta has spent all effort on bringing the e-catalog back online again. During the stress time of the bug fix, the developer of Quicksizer was so frustrated that he left the company. The development of the new system is delayed by 6 months now. At this time, I was contacted by an M&M sales engineer who was on a trip through Italy. He was Figu re 2: Quicksizer 1 Italian: conoscenza = poetic form of customer 2 Italian: sensore = sensor; strumento = device 3 Italian: Torino = Turin 2 / 9

talking about software lifecycle and offered Software Maintenance. Talking about community, M&M immediately recognized my need for help in this area. My hope was to free developers for the implementation of the new system as well as IT administrators so that they can concentrate on new stuff like IIS and Microsoft SQL server. We agreed to have a more detailed meeting on the tasks and software itself two weeks later directly here in Torino where our systems and developers are available. Two days after the sales meeting I received a summary description of the problem statement and a confirmation for the date of the next meeting 2.2 Maintenance analysis meeting Another three days later, M&M sent a questionnaire to ask for our expectations together with a list of technology and IT related questions for us to prepare answers. One week before the meeting, M&M sent out the agenda for the meeting. On the agreed date, M&M sent two technical experts to Senstrumenta in Torino. The meeting was divided into four parts, a 1½ hours management meeting, a 3 hours development meeting, a 2 hours IT meeting and another 1 hours management meeting in the end. 2.2.1 Management meeting First they described the possible tasks for software maintenance offered by M&M and I was surprised which areas could be covered. We agreed to focus on all development related tasks, hosting of the legacy IT systems and 2 nd level support. We can imagine 1 st level support as well, but we first wanted to see how maintenance works in reality. 1 st level support for the old system could start one year later. Once we had chosen the services to be sourced to M&M we talked about the characteristics of these services. As Senstrumenta we wanted to ensure that M&M supports the same service level we offer to the end users. Depending on the problem class we agreed on different reaction times which range from 2 hours for show stoppers to 5 days for minor problems. A developer will work dedicatedly on show stoppers within 8 hours and within 2 weeks for normal problems. As an ISO 9001 certified company, Senstrumenta needs to follow various processes and M&M s services must fit in there. In a mind map overview we documented various boundary conditions like preliminary lead times, time for status reports and the need for non-disclosure agreements on personal data stored in the database of the web server. Finally we agreed on a three years maintenance phase which is expected to end when the new system is available. 2.2.2 Development meeting In the development meeting, M&M gave a summary on the management meeting so that the developers were on the same level of knowledge. While I just leaned back to follow the discussion, Mr. Silvio Patore 4 took over and explained technical parts. M&M wanted to know much about our systems, e.g. our bug tracking system Mantis, the source control system CVS as well as development environment Eclipse and Visual Studio 4 for the development of our 16 bit application Quicksizer. Regarding our software, M&M asked for product characteristics such as different versions of the software on the market or special OEM versions. None of these applied to us. The web portal is 4 Italian: sviluppatore = developer 3 / 9

installed only once and the Quicksizer is a small download so customers are expected to update often. Regarding the system documentation, we provided M&M the requirements documents originally used for the development by CADenas (PDF) as well as additional requirements documents we have written for changes internally (Word). For Quicksizer, no such documentation exists. The architecture and design of the e-catalog is documented in UML diagrams made with Borland Together. For Quicksizer, only a very old architecture description was found for which it s assumed to be totally outdated. As the developer of Quicksizer left the company, there s also nobody to ask for details. Next, M&M wanted to know how the system is tested. Our e-catalog is mainly tested with Selenium and some additional manual tests which are described in proprietary Excel sheets. For the test hardware, Senstrumenta has a 1:1 copy test system as virtual machines bas ed on VMWare ESXi. M&M also wanted to know the number of bug entries, number of users reporting issues and the average effort per year. We simply looked the values up in Mantis: there are about 10 critical issues per year, 25 major issues, 80 normal issues and 250 various other requests. Senstrumenta spends about 1200 hours per year for maintenance of the products, IT department not included because their times are not recorded. For continuous integration we apologized. This topic was rather new for us and we haven t set it up until now. However, this was a good idea for the new system. Finally, Mr. Patore explained the software development processes at Senstrumenta and that the e-catalog is built using Ant scripts and Hudson as continuous integration server. For the source code analysis, we agreed to give M&M a copy for later evaluation. Regarding the systems, Mantis shall remain at Senstrumenta in order to keep existing entries, users and ensure that 1 st level support can be handled by Senstrumenta. This may change when 1 st level support goes to M&M. For the new system, TFS is used anyway. At the end of the development meeting, Mr. Silvio Patore seemed to be very happy in the expectation of getting rid of all the legacy stuff. M&M gave a summary and an overview of the next steps. So after the evaluation of the source code, there would be a project plan setup and service level agreement defined. We decided not to need a quality level agreement as the software is phasing out anyway. 2.2.3 IT meeting In the IT meeting, Mr. Reti Colato 5 took over. We explained that our server infrastructure has 8 Linux Apache webservers, one PostGre SQL database server and a backup database server and is handled by a network load balancer. During maintenance on the system, we typically update one server after another as long as the database structure remains the same. Like this we 5 Italian: reticolato = network Figu re 3: Network arc hitect ure 4 / 9

can ensure that 7 webservers are always up and running. M&M documented the network architecture in Visio and asked whether the load balancer could redirect single users to a new server and such, the migration concept was born: M&M could set up one server after another and the load balancer would redirect requests to M&M in the same way as M&M is able to set it up. Even testing seemed not a problem: the plan was to set up a subdomain which redirects to M&M in any case. The physical servers would not be handed over to M&M as the IT department would like to reuse them for the new system. However, the test environment (our ESXi server) would be handed o ver to M&M at maintenance start, because IT department has already set up a Hyper-V test environment for the ongoing development. The only issue remaining open was the access to the database servers during the migration phase. M&M wanted to work on a concept after the meeting. Mr. Colato mentioned that it should not be such a big risk, because database activity is typically low. The 1000 Mbit link was mainly designed for the database replication in case of problems, not for daily use. 2.2.4 Management meeting In the second management meeting, M&M summarized the meetings before and presented the idea on how to do maintenance for the system. As no major topics were open, Senstrumenta and M&M signed an outline agreement for the maintenance of the system, so that M&M could work on the next steps. For Senstrumenta, M&M wanted to provide the solution concept for the database server issue, the Service Level Agreement and the project plan. Meanwhile, Senstrumenta should prepare the source code and any system documentation, set up the subdomain for testin g and prepare the load balancer for the switch to M&M. 2.3 The time between Some days after the management meeting, we got emails from M&M including user names and passwords for access to a system called Extranet, where we could review and download documents from M&M or upload documents to M&M. Initially we found all the presentations by M&M, any notes from the meetings as well as the outline agreement. A list of contact persons and their VCF 6 files was available for download as well. In the list of open tasks we could see what has to be done by M&M and Senstrumenta and what the due dates are. About one week after Senstrumenta uploaded the source code, we got feedback on it. M&M seemed to have done a thorough analysis and came up with a maintainability index of our two systems. As expected, the JSP website was not that bad, but the undocumented Quicksizer was. It was also M&M first finding out that Quicksizer did not work on Windows 64 bit at all just one week before the first customer complained about this. That s also why we asked M&M to resolve this issue even before the maintenance contract was signed. 2.3.1 Windows 7 64 bit The Quicksizer application did not work on 64 bit operating systems at all. This was a pitty, because we knew that many customers were likely to upgrade to Windows 7 x64 at the moment. 6 VCF: vcard file; Electronic business card 5 / 9

Figu re 4: Q uicksizer do es not work on 64 bit M&M s suggestion was to convert the 16 bit Visual Basic 4 application to Visual Basic 6, compile it as a 32 bit application which should run on 64 bit operating systems in some kind of compatibility mode. The setup also needed to be adapted, but it was a Visual Basic Setup assistant as well which should undergo the same procedure. It was great how M&M simply accepted a by-effort budget of 40 hours and presented a solution within only one week, even before our purchase department actually ordered the task. With this in mind, we were confident that maintenance for Quicksizer was in good hands. 2.4 Start of maintenance 2.4.1 Contracts Senstrumenta was notified that the Quicksizer intermezzo would delay the documents by one week, but of course this was expected and acceptable my Senstrumenta. Senstrumenta received the suggestion for the service level agreement. Especially the certificate of performance was a good starting point to get a rough overview. The amount of support hours per year was set to 1600, divided into four sections: 200 hours management from Germany, 400 hours high qualified work in China, 800 hours normal work in China and 200 hours IT management and webhosting from Germany. Invoices are sent every month for 130 hours mixed efforts. Status reports are sent on every 10 th of a month so that Senstrumenta has them for internal meetings on every 15 th. The maintenance start was defined almost immediately and should run for at least two years. M&M already considered that the new website should be up in three years, so the contract does not automatically renew into the fourth year. Instead we would define this at that point of time in the future. As the end of lifetime is foreseen, M&M didn t want to buy physical servers at their site but rather do hosting with an external provider at 50 per server which have the same performance as the ones at Senstrumenta. This approach was also expected to resolve the database server issue during the migration phase. The database server could be accessed at full 100 Mbit speed with some port forwarding of the load balancer. This topic has been discussed with the IT department at Senstrumenta already and was agreed in advance. The contract was ok so far. We only included the long term goal to migrate Quicksizer to.net so that it can be handed over to Senstrumenta again when the new website is up. This was a new requirement and not known to M&M before, but it was accepted without further modification of anything else in the contract. With this migration M&M brought up the idea of registering the applic ation at Windows Error Reporting to get more feedback from customers, which we thought is a really good idea. So we included it in the contract. The second change was to not setup a virtual development environment for the website as we couldn t clearly see the benefits for that due to the limited lifetime. 6 / 9

So the maintenance points by M&M were for the JSP website: Hosting of the servers and database servers Corrective maintenance (freely) Adaptive maintenance (freely) Perfective maintenance (upon request) Source code version control at M&M 2 nd level support Release management And for Quicksizer Hosting (is actually part of the JSP website) Corrective maintenance (freely) Adaptive maintenance (freely) Perfective maintenance (upon request) Migration to.net and registration at WER Source code version control at M&M Setup up continuous integration 2 nd level support Release management In most points, Senstrumenta wanted to trust M&M, so there was almost no control on which bugs are fixed, when tests are performed and when new updates of the website are installed. This should help up free as many developers as possible. 2.4.2 Project plan The project plan started with the setup of systems at M&M. The existing collaboration platform (Extranet) should be extended by access information to source code, additional contact persons when assigned, a section for status report etc. Source code version control was available one week after project start. The test system was set up the week after and the first server was expected two weeks later, others following every third day. Figu re 5: Project pl an for mai ntenance start Next, a minimum of continuous integration for the JSP website was set up and bug fixing for Quicksizer started. With all major bugs fixed, migration to.net should start and continuous integration for the.net version should be implemented. 7 / 9

2.5 First incidents Actually, the first incident has already been handled by M&M before the maintenance contracts were signed (see 2.3.1). 2.5.1 Website error Short after M&M started doing Software Maintenance, a customer reported a problem. The website showed an error message and asked the user to inform the administrator. We reported the issue by phone after our 1 st level support had already entered it into Mantis. Having hung up the phone, we could almost feel how M&M became acti ve. An email was sent which announced a new major item in the Extranet having the according due dates in matters of bug classifications agreed before. We were called back within 1 hour having someone on the line who already understood the problem and it was clear that he has had a look into Mantis, seen the screen shot already and was ready to ask the right questions. The problem occurred first time when all servers were running at M&M only, so the initial idea was that it had to do with the server setup at M&M. At Senstrumenta, we still had some servers on standby, so we simply switched on one server and directed the customer via load balancer to Italy. However, the problem still occurred, so it was just a normal problem. As M&M analyzed the issue, it turned out that the customer wanted to access a page via a hyperlink saved as favorite. The session had timed out and the given navigation path could not be retrieved. This was identified with help of Senstrumenta and the end user within 5 hours and we could come up with a workaround. Again, the Extranet calendar sent out a status change that the major issue has been resolved into a normal issue which needs implementation actions. We were asked to confirm this via an acceptance protocol online so that M&M can Figu re 6: Scre enshot of err or as rep orted by c ustome r approve it and ensure that critical issues have been handled with the according priority. This, however, is only done for class 1 issues. For the solution of this, M&M decided to detect outdated session information, inform the user about the issue and redirect him to the login page if not logged in yet or instruct him to navigate to the place manually. For the development of the new system, this topic was considered as a requirement as well and we decided to look at all major issues in the future. Some days later, the issue was resolved and we could easily track the status changes in the Extranet task list. 2.5.2 Security update Six weeks after maintenance was taken over by M&M, Quicksizer was already ported to.net. This was great, because we could start integrating it into the new website already. 8 / 9

Until now, M&M performed tests with the.net version on Windows 7, and it worked well on XP at Senstrumenta as well. Suddenly, from one day to another, it stopped working on one machine and more machines on consecutive days. We were glad that Windows Error Reporting was already active, because we were demonstrated how powerful this actually is. With the second crash, Microsoft had collected enough information for analyzing the issue. M&M downloaded the crash report and found that it was caused by a new security update for the.net framework on Windows XP. 2.6 Current status Today, we are on schedule with the development of the new website. M&M successfully hosts the old version and has delivered a working.net version of Quicksizer which is already fully integrated into the new demo website. This worked so well in the past that we imagine continuing the maintenance for Quicksizer even when the new website is launched. There are no complaints known. Any major support requests were handled in time; all status reports are on time as well and available for download. Source code backups of the JSP website were provided every two months via USB stick and Senstrumenta meanwhile has access to the source code of Quicksizer.NET via TFS. Continuous integration is set up and M&M from time to time answers questions regarding continuous integration of our new system on an hourly basis. We have just been invited to the annual review meeting and I think we will discuss on a quality level agreement for Quicksizer.NET. Another topic suggested by M&M is user tracking on the old website via Google Analytics to get additional information for improving the new site. They also want to have a consultant join the meeting for a project review meetin g on our side in order to help us improve the development. In general, I like the way how M&M is coming up with new ideas it s not a simple customer relationship, it s more like a partnership. 9 / 9