Comparing Microsoft.NET and IBM WebSphere/J2EE

Transcription

1 RESEARCH REPORT Comparing Microsoft.NET and IBM WebSphere/J2EE A Productivity, Performance, Reliability and Manageability Analysis David Herst with William Edwards and Steve Wilkes September 2004 research@middleware-company.com

2 1 Disclosures 1.1 Research Code of Conduct The Middleware Company offers the world s leading knowledge network for middleware professionals. The Middleware Company operates communities, sells consulting and conducts research. As a research organization, The Middleware Company is dedicated to producing independent intelligence about techniques, technologies, products and practices in the middleware industry. Our goal is to provide practical information to aid technical decision making. Our research is credible. We publish only what we believe and can stand behind. Our research is honest. To the greatest extent allowable by law we publish the parameters, methodology and artifacts of a research endeavor. Where the research adheres to a specification, we publish that specification. Where the research produces source code, we publish the code for inspection. Where it produces quantitative results, we fully explain how they were produced and calculated. Our research is community-based. Where possible, we engage the community and relevant experts for participation, feedback, and validation. If the research is sponsored, we give the sponsor the opportunity to prevent publication if they deem that publishing the results would harm them. This policy allows us to preserve our research integrity, and simultaneously creates incentives for organizations to sponsor creative experiments as opposed to scenarios they can win. This Code of Conduct applies to all research conducted and authored by The Middleware Company, and is reproduced in all our research reports. It does not apply to research products conducted by other organizations that we may publish or mention because we consider them of interest to the community. 1.2 Disclosure This study was commissioned by Microsoft. The Middleware Company has in the past done other business with both Microsoft and IBM. Moreover, The Middleware Company is an independently operating but wholly owned subsidiary of VERITAS Software ( NASDAQ:VRTS). VERITAS and IBM have a number of business relationships in certain technology areas, and compete directly against each other in other technology areas. Microsoft commissioned The Middleware Company to perform this study on the expectation that we would remain vendor-neutral and therefore unbiased in the outcome. The Middleware Company stands behind the results of this study and pledges its impartiality in conducting this study. 1.3 Why are we doing this study? What is our agenda? We are compelled to answer questions such as this one, due to controversy that sponsored studies occasionally create. Page 2 of 109

3 First, what our agenda is not: It is not to demonstrate that a particular company, product, technology, or approach is better than others. Simple words such as better or faster are gross and ultimately useless generalizations. Life, especially when it involves critical enterprise applications, is more complicated. We do our best to openly discuss the meaning (or lack of meaning) of our results and go to great lengths to point out the several cases in which the result cannot and should not be generalized. Our agenda is to provide useful, reliable, and profitable research and consulting services to our clients and to the community at large. To help our clients in the future, we believe we need to be experienced in and be proficient in a number of platforms, tools, and technologies. We conduct serious experiments such as this one because they are great learning experiences, and because we feel that every technology consulting firm should conduct some learning experiments to provide their clients with the best value. If we go one step further and ask technology vendors to sponsor the studies (with both expertise and expenses), if we involve the community and known experts, and if we document and disclose what we re doing, then we can: Lower our cost of doing these studies Do bigger studies Do more studies Make sure we don t do anything silly in these studies and reach the wrong conclusions Make the studies learning experiences for the entire community (not just us) 1.4 Does a sponsored study always produce results favorable to the sponsor? No. Our arrangement with sponsors is that we will write only what we believe, and only what we can stand behind, but we allow them the option to prevent us from publishing the study if they feel it would be harmful publicity. We refuse to be influenced by the sponsor in the writing of this report. Sponsorship fees are not contingent upon the results. We make these constraints clear to sponsors up front and urge them to consider the constraints carefully before they commission us to perform a study. Page 3 of 109

4 2 TABLE OF CONTENTS 1 DISCLOSURES Research Code of Conduct Disclosure Why are we doing this study? What is our agenda? Does a sponsored study always produce results favorable to the sponsor? TABLE OF CONTENTS EXECUTIVE SUMMARY The Teams The System The Implementations Developer Productivity Results Configuration and Tuning Results Performance Results Reliability and Manageability Results INTRODUCTION How this Report is Organized Goals of the Study The Approach The ITS System Development Environments Tested Application Platform Technologies Tested Application Code Availability THE EVALUATION METHODOLOGY The Teams The IBM WebSphere Team The Microsoft.Net Team Controlling the Laboratory and Conducting the Analysis The Project Timeline Target Schedule Division of Lab Time Between the Teams Detailed Schedule Laboratory Rules and Conditions Overall Rules Development Phase Deployment and Tuning Phase Testing Phase The Evaluation Tests THE ITS PHYSICAL ARCHITECTURE Details of the WebSphere Architecture IBM WebSphere IBM HTTP Server (Apache) IBM Edge Server IBM WebSphere MQ Details of the.net Architecture Microsoft Internet Information Services (IIS) Microsoft Network Load Balancing (NLB) Page 4 of 109

5 6.2.3 Microsoft Message Queue (MSMQ) TOOLS CHOSEN Tools Used by the J2EE Team Development Tools Rational Rapid Developer Implementation WebSphere Studio Application Developer Implementation Analysis, Profiling and Tuning Tools Tools Used by the.net Team Development Tools Analysis, Profiling and Tuning Tools DEVELOPER PRODUCTIVITY RESULTS Quantitative Results The Basic Data NET vs. RRD NET vs. WSAD RRD Development Process Architecture Summary RRD Applications Database Access Overall Shape of the Code Distributed Transactions What Went Well Web Interfaces Web Service Integration Significant Technical Roadblocks Holding Data in Sessions Web Service Integration Configuring and Using WebSphere MQ Handling Null Strings in Oracle Building the Handheld Module Miscellaneous RRD Headaches WSAD Development Process Architecture Summary Overall Shape of the Code Distributed Transactions Organization of Applications in WSAD What Went Well Navigating the IDE Building for Deployment Testing in WebSphere Common Logic in JSPs Signficant Technical Roadblocks XA Recovery Errors from Server Miscellaneous WSAD Headaches Microsoft.NET Development Process NET Architecture Summary Organization of.net Applications Database Access Distributed Transactions ASP.NET Session State What Went Well Significant Technical Roadblocks Transactional MSMQ Remote Read Miscellaneous.NET Headaches Page 5 of 109

6 DataGrid Paging Web Services Returning DataSets The Mobile Application Model Object Class Creation CONFIGURATION AND TUNING RESULTS WEBSPHERE CONFIGURATION AND TUNING PROCESS SUMMARY RRD Round: Installing Software Starting Point Installing WebSphere Network Deployment Installing IBM HTTP Server Installing IBM Edge Server RRD Round: Configuring the System Configuring JNDI Configuring the WebSphere Web Server Plugin RRD Round: Resolving Code Bottlenecks Rogue Threads Optimizing Database Calls Optimizing the Web Service Paging Query Results Caching JNDI Objects Using DTOs for Work Tickets Handling Queues in Customer Service Application RRD Round: Tuning the System for Performance Tuning Strategy Performance Indicators Tuning the JVM Garbage Collection Heap Size Vertical Scaling Database Tuning Tuning JDBC Settings Web Container Tuning Web Thread Pool Maximum HTTP Sessions Web Server Tuning Session Persistence WSAD Round: Issues Use of External Libraries and Classloading in WebSphere Pooling Objects Streamlining the Web Service I/O Optimizing Queries Significant Technical Roadblocks Switching JVMs with WebSphere Configuring Linux for Edge Server, Act Configuring Linux for Edge Server, Act Configuring Linux for Edge Server, Act Configuring JNDI for WebSphere ND Edge Server s Erratic Behavior Session Persistence Persisting to a Database In-Memory Replication Tuning Session Persistence Hot Deploying Changes to an Application Configuring for Graceful Failover Page 6 of 109

7 Failover Requirements Standard Topology Non-Standard Topology Modified Standard Topology Deploying the WSAD Web Service The Sudden, Bizarre Failure of the Work Order Application Using Mercury LoadRunner NET CONFIGURATION AND TUNING PROCESS SUMMARY Installing and Configuring Software Network Load Balancing (NLB) ASP.NET Session State Server Resolving Code Bottlenecks Base Tuning Process Tuning the Database Tuning the Web Applications Tuning the Servers Tuning the Session State Server Code Modifications Tuning Data Access Logic Tuning Message Processing Other Changes Changes to Machine.config Changes Not Pursued Significant Technical Roadblocks Performance Dips in Web Service Lost Session Server Connections PERFORMANCE TESTING Performance Testing Overview Performance Test Results ITS Customer Service Application ITS Work Order Web Application Integrated Scenario Message Processing Conclusions from Performance Tests MANAGEABILITY TESTING Manageability Testing Overview Manageability Test Results Change Request 1: Changing a Database Query Change Request 2: Adding a Web Page Change Request 3: Binding a Web Page Field to a Database Conclusions from Manageability Tests RELIABILITY TESTING Reliability Testing Overview Reliability Test Results Controlled Shutdown Test Catastrophic Hardware Failure Test Loosely Coupled Test Long Duration Test Page 7 of 109

8 14.3 Conclusions from Reliability Tests OVERALL CONCLUSIONS APPENDIX: RELATED DOCUMENTS APPENDIX: SOURCES USED Sources Used by the IBM WebSphere Team Sources Used by the Microsoft.NET Team APPENDIX: SOFTWARE PRICING DATA IBM Software Microsoft Software Page 8 of 109

9 3 EXECUTIVE SUMMARY This study compares the productivity, performance, manageability and reliability of an IBM WebSphere/J2EE system running on Linux to that of a Microsoft.NET system running on Windows Server The Teams To conduct the study, The Middleware Company assembled two independent teams, one for J2EE using IBM WebSphere, the other for Microsoft.NET. Each team consisted of senior developers similarly skilled on their respective platforms in terms of development, deployment, configuration, and performance tuning experience. 3.2 The System Each team received the same specification for a loosely-coupled system to be developed, deployed, tuned and tested in a controlled laboratory setting. The system consisted of two Web application subsystems and a handheld device user interface, all integrated via messaging and Web services. 3.3 The Implementations The WebSphere team developed two different implementations of the specification, one using IBM s model-driven tool Rational Rapid Developer (RRD), the other with IBM s code-centric tool WebSphere Studio Application Developer (WSAD). The.NET team developed its single implementation using Visual Studio.NET as the primary development tool. 3.4 Developer Productivity Results In the development phase of the study, the time it took each team to complete the initial implementation (including installing all necessary development and runtime software) was carefully measured to determine overall developer productivity. The.NET implementation was completed significantly faster than the RRD implementation, and also faster than the WSAD implementation. Page 9 of 109

10 .NET vs. RRD.NET vs. WSAD RRD vs. WSAD Significantly better. Greatest difference in product installs; less to install/configure on Windows Server side. Also differences in developer productivity for all subsystems..net team had longer history w/ VS than J2EE team w/ RRD. Development Productivity Better; uncertain how much.* Worse, uncertain how much. * * The team using WSAD had already built the same application in RRD, and hence realized productivity advantages not realized for RRD or.net, since they were familiar with the specification. 3.5 Configuration and Tuning Results After developing their system, each team was measured in how long they took to configure and tune it in preparation for a series of performance, manageability, and reliability tests. The.NET team completed this stage in significantly less time than the WebSphere team for the RRD implementation. The.NET team took 16 man days for configuration and tuning, while the WebSphere team took 71 man days (much of it spent addressing software installation issues and patching the operating system, however). Later, when they deployed the WSAD implementation to the existing WebSphere infrastructure, the WebSphere team spent an additional 24 man days tuning and configuring..net vs. RRD.NET vs. WSAD RRD vs. WSAD Tuning Productivity Significantly better. Huge part of RRD time taken patching Linux, dealing with Edge Server issues. Significant time also spent re-working RRDgenerated code to get better performance. Better; uncertain how much. Since complete runtime platform had already been installed/configured and tuned for the RRD implementation, this stage was completed much more quickly for WSAD. Uncertain. 3.6 Performance Results In a battery of performance tests, the.net implementation running on Windows Server 2003 significantly outperformed the RRD implementation running on Linux in four tests. Compared to the WSAD implementation, the.net version performed about equally well overall, doing better on some tests and not as well on others. Page 10 of 109

11 .NET vs. RRD.NET vs. WSAD RRD vs. WSAD Performance Significantly better on 3 of 4 tests..net achieved user throughput % higher than RRD in 3 tests. In 4 th test,.net achieved 40% higher message processing thruput. About equal..net achieved higher user throughput in 1 test, slightly higher in 1, worse in 1. In the 4 th test, WSAD achieved nearly 3 times the message processing thruput. Significantly worse on all 4 tests. 3.7 Reliability and Manageability Results Manageability and reliability tests revealed better results for.net on Windows Server 2003; it significantly surpassed the two J2EE implementations on Linux in terms of deploying changes under load, gracefully downing servers, and handling catastrophic failover. In terms of sustained, long-term operation under normal load, all three implementations performed equally well..net vs. RRD.NET vs. WSAD RRD vs. WSAD Manageability Significantly better..net had many fewer errors during deployment..net slightly faster to deploy..net preserved sessions more reliably. Significantly better..net had many fewer errors during deployment..net slightly faster to deploy..net preserved sessions more reliably. Better. RRD had many fewer errors during deployment. RRD preserved sessions more reliably. Time to deploy about the same. Significantly better. RRD implementation: Could not add server to cluster after graceful shutdown. RRD implementation could not handle catastrophic failover. Reliability: Handling Failover Significantly better. WSAD implementation could not handle catastrophic failover. Worse. RRD implementation: Could not add server to cluster after graceful shutdown. Reliability: Sustained Operation Over 12 Hour Period Under Moderate Load Equal. Equal. Equal. Page 11 of 109

12 4 INTRODUCTION Previous studies by The Middleware Company have compared tools or platforms on the basis of one criterion or another, such as developer productivity, ease of maintenance or application performance. This study compares two enterprise application platforms, Microsoft.NET and IBM WebSphere/ J2EE, across a full range of technical criteria: developer productivity, application performance, application reliability, and application manageability. Although sponsored by Microsoft, the study was conducted independently, in a strictly controlled laboratory environment, with no direct vendor involvement by either Microsoft or IBM. The Middleware Company cannot emphasize enough that Microsoft had no control over the development, testing, and results of the study, and we firmly stand by those results as accurate and unbiased. Towards that end, TMC has published the methodology used and the source code for both the.net and J2EE application implementations for public download and scrutiny. Customers can review and comment on the methodology, examine the code, and even repeat the tests in their own testing environment. 4.1 How this Report is Organized This report covers every aspect of the Microsoft.NET IBM WebSphere/J2EE Comparison Study: its purpose and methodology, participants, rules and procedures, schedule and working conditions; not to mention the results, both qualitative and quantitative. Section 1 discloses the conditions under which The Middleware Company conducted this study, including our research code of conduct and our policy regarding sponsored studies such as this one. Section 3 gives a brief, high-level summary of the study and its results. Section 4 (this section) introduces the study. It describes: The goals we tried to achieve with the study The unique overall approach that the study takes The software system that the two teams developed and tuned The development environments that were tested The technologies of the.net and WebSphere platforms that were tested What study artifacts are available and how to obtain them Section 5 covers the study s methodology in detail: The composition of the two teams The independent auditor who controlled the study conditions and conducted the analysis The project schedule The rules and conditions that governed the two teams in the laboratory A summary of the tests conducted Section 6 details the physical architecture of the system: The hardware infrastructure used by both teams The software infrastructure that each team installed Page 12 of 109

13 Section 7 describes the tools that each team used during the different phases of the study. In particular, since the J2EE team built two implementations of the system using two different development tools, this section compares the two IDEs. Section 8 presents the developer productivity results: The quantitative results broken out by core development tasks The qualitative experiences of the two teams developing each of the three implementations, including important technical roadblocks Sections 9-11 present the deployment, configuration and tuning results: Section 9 lays out the quantitative results Section 10 describes the WebSphere team s experience, including significant technical roadblocks Section 11 describes the.net team s experience, including significant technical roadblocks Section 12 presents the results of the performance tests Section 13 presents the results of the manageability tests Section 14 presents the results of the reliability tests Section 15 presents our conclusions The final sections, from 16 on, contain various appendices: Where to find documents related to this report Important sources used by both teams Pricing data on the software used in this study 4.2 Goals of the Study Commentary abounds about the technical merits of both J2EE and Microsoft.NET for enterprise application development. The Middleware Company in particular has conducted various studies in the past to compare these two enterprise platforms. Some of these studies, such as Model Driven Development with IBM Rational Rapid Developer, address developer productivity. Others, such as J2EE and.net Application Server and Web Services Performance Comparison, focus on performance. None, however, has spanned a wider set of technical criteria that includes not only productivity and performance, but application platform manageability and reliability as well. This study is the first of its kind to measure all of these criteria, using a novel evaluation approach. While we expect the study to spark controversy, we also hope it will fulfill two important goals: Provide valuable insight into the Microsoft.NET and IBM WebSphere/J2EE development platforms. Suggest a controlled, hands-on evaluation approach that organizations can use to structure their own comparisons and technical evaluations of competing vendor offerings. Page 13 of 109

14 4.3 The Approach This study took the approach of simulating a corporate evaluation scenario. In it, a development team is tasked with building, deploying and testing a pilot B2B integrated application in a fixed amount of time, after which we evaluate the results the team was able to achieve in this time period. In the study we executed the scenario three times, once using Microsoft.NET 1.1 running on the Windows 2003 platform, and twice using IBM WebSphere 5.1 running on the Red Hat Enterprise Linux AS 2.1 platform. (The latter two cases differed in the development tool used; more on this in Section 4.5.) We assembled two different teams, one for each platform, each similarly skilled on their respective platforms. Each team consisted of senior developers experienced in enterprise application architecture, application development, and/or performance tuning. The rules limited each team to no more than two members in the lab at any time, but did not require the same two members for all phases of the exercise. The IBM WebSphere/J2EE team consisted of three senior developers from The Middleware Company with 16 years combined experience in J2EE. The same two of these developers built both J2EE implementations, and all three participated at different times in the deployment, tuning and testing phases. For the installation, deployment and initial tuning of the WebSphere platform, the J2EE team also used two independent, WebSphere-certified consultants having a total of 7 years experience with the WebSphere platform. The Microsoft.NET team consisted of three senior developers from Vertigo Software, a California-based Microsoft Solution Provider, with a combined 10 years experience building software on Microsoft.NET. The Middleware Company took pains to keep the study free of vendor influence: We subcontracted CN2 Technology, a third-party testing company, to prepare the application specification, set up the testing lab, audit the development process for each team, and independently perform the actual application testing for each platform. The teams did not communicate with each other during the study. Neither team had knowledge of the other team s results until after the study was completed. Neither Microsoft nor IBM had any influence over the development teams during the study. It is important to note that this study represents what the development teams could achieve using only publicly available technical materials and vendor support channels for their platform. It does not represent what the vendors themselves might have achieved, nor what each team might have achieved if given a longer development and tuning schedule or allowed direct interaction with vendor consultants. Therefore, the resulting applications developed by the two teams may not fully represent vendor best practices or vendor-approved architectures. Rather, they reflect what customers themselves might achieve if tasked with independently building their own custom application using publicly available development patterns, technical guidance and vendor support channels. 4.4 The ITS System The comparison at the heart of the study centers around the development and testing of a loosely coupled system known as ITS. ITS is a facilities management system created for the fictitious ITS Facilities Management Company (ITS-FMC). The system represents a B2B integration scenario, allowing corporate customers of ITS-FMC to use a Web-based hosted Page 14 of 109

15 application to create and track work order requests for facilities management on their corporate premises. The ITS system comprises three core subsystems that operate together in in both a loosely coupled fashion (vi a messaging) and a tightly coupled fashion (via synchronous Web Service requests): The ITS Customer Service Application. ITS-FMC s corporate clients use this Web-based application to create and track work order requests for facilities management at their premises. The application automatically dispatches work order requests via messaging to the central ITS system, which operates across the Internet on a separate ITS-FMC internal network. The ITS Customer Service Application also allows customers to track the status of their work orders via Web service calls to the ITS central system, as well as view/modify customer and user information. The ITS Central Work Order Processing Application. This application is operated by ITS-FMC itself on a separate corporate network. The application receives incoming work order requests (as messages) from the ITS Customer Service Application. It places the requests into a database for further business processing, including assignment to a specific on-site technician. The application hosts the Web service that returns work order status and historical information to the ITS Customer Service Application. Additionally, this application has a Web user interface that ITS-FMC s central dispatching clerks can use to search, track and update work order requests, as well as query customer information and query/modify technician data. The Technician Work Order Mobile Device Application. This application operates on a handheld device, allowing technicians to retrieve their newly assigned work items and update work order status as they complete their work orders at the customer premises. Technicians use this application for dispatching purposes, and to log the time spent working on an issue so that customer billing can occur. Page 15 of 109

16 The following diagram illustrates these three subsystems and their interactions: Technician Mobile Device Application ITS Corporate Network ITS Customer Service Application B2B Internet Connectivity ITS Work Order Message Queue Server ITS Work Order Processing Application ITS Customer Service Database ITS Durable Message Queue ITS Work Order Processing Database Figure 1. ITS Connected System Diagram 4.5 Development Environments Tested In a study that focuses on both developer productivity and application performance, the development environment can influence the study s outcome as much as the deployment platform. The choice of development environment affects how quickly and easily the developer can build the application initially alter the code to eliminate performance bottlenecks While there are.net development tools from third party vendors such as Borland, the vast majority of.net development is done using Visual Studio.NET from Microsoft. This is the development environment that the.net team used to produce its implementation. The J2EE world, on the other hand, offers many competing development tools with different approaches and advantages. Even within IBM s domain, choices exist. To reflect this range of offerings and enhance the study s usefulness, we had the J2EE team develop two different implementations of the ITS system using two different IBM tools: Rational Rapid Developer (RRD) and WebSphere Studio Application Developer (WSAD). Since both IDEs belong to IBM and are designed to work well with WebSphere, they are both consistent with the study s focus on the IBM WebSphere platform. But the two IDEs have important differences that ultimately led to different results. Page 16 of 109

17 Details on these tools, how they compare, how they were used, and other development software used with them can be found in Section Application Platform Technologies Tested The ITS system tests the following functionality of the two application platforms: Web application development Web application configuration/tuning Web application manageability, reliability and performance Message-based application development Message queue reliability and performance Mobile device application development 4.7 Application Code Availability The application code for both the.net and J2EE implementations can be downloaded from Customers can download the applications and install them in their own environments for further testing and confirmation of the results. The discussion forum for the study is located at Finally, customers and vendors can The Middleware Company to discuss the report and propose further testing or offer comments by ing to: Page 17 of 109

18 5 THE EVALUATION METHODOLOGY This study was designed to simulate two enterprise development teams given a fixed amount of time to build and tune a working pilot application according to a set of business and technical requirements. One team developed the application using IBM WebSphere running on Linux, while the other team developed the application using Microsoft.NET running on Windows Development took place in a controlled laboratory environment where the time taken to complete the system was carefully measured. The two teams worked from a common application specification derived from a set of business and technical requirements. Neither team had access to the specification until development started in the controlled lab setting. After developing an implementation, the team then tuned and configured it as part of a measured deployment phase. Each implementation was then put through a set of basic performance, manageability and reliability tests while running under load on the production equipment. Hence this study not only compares the relative productivity achieved by each development team, but also captures the base performance, manageability and reliability of each application in a deployed production environment. It is extremely important to note that the study allocated a fixed amount of time to each phase of the project, and hence objectively documents what each team was able to achieve in this fixed amount of time. 1 The study objectively documents exactly what each team was able to achieve, inclusive of detailed notes documenting technical roadblocks encountered by each team, and how these were resolved. As such, the study tells an interesting story that will undoubtedly spark much debate, but also shed valuable light on each platform based on actual hands-on development and testing of a pilot business application. 5.1 The Teams Each team fielded two developers skilled in their respective development platform, and each team was selected such that their product experience levels and skill sets matched as closely as possible. As noted in Section 4.3, each team could have only two members in the lab at one time, but did not have to use the same two throughout the exercise. Neither team included any representative from either IBM or Microsoft, and neither team was allowed any direct interaction with vendor technicians from IBM or Microsoft other than the standard online customer support channels available to any customer. In cases where a team used a vendor support channel, support technicians were not told they were assisting a research project conducted by The Middleware Company; so the team received only the standard treatment afforded any developer on these channels. To mirror the development process of a typical corporate development team, we allowed the teams to consult with other members of their organizations outside the lab, to answer technical questions and provide guidance as required. Such access to external resources was monitored and logged, and we extended the rule prohibiting direct vendor interactions (other than with standard customer support channels) to all resources contacted during the development and testing phases of the project. Here are details on the makeup and experience of the two teams. 1 Note that under certain circumstances we allowed a team to go beyond that fixed time period. See section for details. Page 18 of 109

19 5.1.1 The IBM WebSphere Team The WebSphere team consisted of three developers from The Middleware Company, described in the following table. Members A and B developed both the RRD and WSAD implementations, while all three members participated at different times in the tuning and testing phases. J2EE Team Members from The Middleware Company Team Member Development Experience (years) Java Experience (years) J2EE Experience (years) Other Relevant Experience A Broad experience with development tools and platforms. Particular strength in design. B * Experienced in RRD, modeling and design. C * Extensive experience in tuning enterprise applications for performance. * Includes experience with Java servlet API predating the introduction of J2EE in Additionally, the J2EE team used two independent, IBM-certified WebSphere consultants at different times during the deployment and tuning phase. One had three years experience as a WebSphere administrator on various Unix platforms, including Linux. The other had over four years experience installing, configuring and supporting IBM WebSphere on multiple platforms, including Linux The Microsoft.Net Team The.NET team consisted of three senior developers from Vertigo Software, a California-based Microsoft Solution Provider, with the following credentials:.net Team Members from Vertigo Software Team Member Development Experience (years) Microsoft Platform Experience (years).net Experience (years) Other Relevant Experience A Experienced in Web application development and design B Experienced in design in the presentation, business, and database tiers C Experienced in development and performance tuning Page 19 of 109

20 5.2 Controlling the Laboratory and Conducting the Analysis The Middleware Company subcontracted a third-party testing organization, CN2 Technology, to write a specification for the ITS system, set up the lab environment, design the tests, monitor and control the testing environment, and conduct the actual tests of the J2EE and.net implementations. CN2 strictly monitored the time spent by each development team on the various phases of the project, and controlled the lab environment. CN2 also strictly monitored Internet access and access, including logging all such access from within the lab, to ensure that neither team violated the rules of the lab. For details on those rules, see Section The Project Timeline Target Schedule This study was designed with the objective that each team would complete its work in 25 workdays (five calendar weeks), distributed as follows: Phase Description Days Phase 1 Development 10 Phase 2 Deployment and tuning 10 Phase 3 Formal evaluation testing up to 5 (as needed) While we felt confident that the teams could complete Phases 1 and 3 in the allotted time, we were less certain about Phase 2. If, after ten days of deployment and tuning, the implementation did not perform up to even minimal standards, the results of formal testing in Phase 3 would have little meaning. So we added a requirement that each team continue their configuration and performance tuning until satisfied that their implementation would perform well enough to actually undergo the tests in the final week. This meant that each team was allowed to go beyond their allotted ten days if they desired, with the understanding that all time spent would be monitored and reported. Page 20 of 109

21 5.3.2 Division of Lab Time Between the Teams To keep the two teams from communicating with each other, while at the same time preserving the continuity of their work, we interleaved their time in the lab in the following sequence:.net Team J2EE Team Implementation Phase Implementation Phase.NET 1: Development RRD 1: Development.NET.NET 2: Deployment / tuning 3: Evaluation testing RRD RRD WSAD * WSAD 2: Deployment / tuning 3: Evaluation testing 1: Development 2: Deployment / tuning WSAD 3: Evaluation testing *Note that the J2EE team developed the WSAD implementation offsite, not in the controlled lab environment. This implementation was not in the initial scope of the project, but was added to ensure that the performance, reliability and manageability tests painted a more complete picture of J2EE/WebSphere for the community Detailed Schedule The following table documents the desired project schedule including the schedule goals established for the development, tuning/configuration and testing of each implementation. As explained in Section 5.3.2, the two teams occupied the lab at different times, so this schedule was repeated for each implementation. Desired Development and Testing Schedule (established prior to start of exercise) Schedule Timeline Task/Event Description Phase 1: Development Day 1 Day 1 (1 hour) Day 1 (2 hours) Development team arrives in lab. Overview of lab rules and hardware environment. Development team given application specification for first time. Two hour application specification overview with Q&A. Team was introduced to the lab environment for the first time, lab rules were explained, and a walkthrough of the hardware was conducted. CN2 Technology provided a detailed walkthrough of the application specification and answered initial questions about the specification. Page 21 of 109

22 Desired Development and Testing Schedule (established prior to start of exercise) Schedule Timeline Task/Event Description Day 1 Application specification review, development tool and application server setup. Team reviewed the application specification in detail, and created a strategy for dividing the work and beginning development. Days 1-10 Application development. Team developed the application according to the provided specification. All development time in the lab was carefully tracked for each component of the system. CN2 deemed development completed when the implementation passed a series of functional tests. Phase 2: Deployment and Tuning Day 11 Days Review of base performance, manageability and reliability tests and requirements, including review of Mercury Load Runner test scripts and test tool. Application performance and configuration tuning. CN2 reviewed with team the tests to be performed and technical requirements/goals for these tests. CN2 provided a walkthrough of the Mercury LoadRunner testing environment and base test scripts so the team could begin configuring and tuning. Ten 8-hour days were initially allotted for tuning in preparation for evaluation tests. However, the team was allowed more time if required to ensure they felt ready to conduct the actual tests. Phase 3: Evaluation Testing Days Performance, manageability and reliability tests conducted. Performance, manageability and reliability tests were conducted in the lab and results logged. 5.4 Laboratory Rules and Conditions This section describes the conditions each team faced as they started each phase of the study and the various rules governing their behavior inside and outside the laboratory environment Overall Rules Several rules applied to the entire exercise: Team members could only use the provided machines for development work and Internet access. Personal laptops were barred from the lab. Each day was limited to 8 hours working time in the lab, with an additional hour for lunch. The team could seek technical support and guidance from other members of their organization outside the lab as required. They could communicate via telephone or . Page 22 of 109

23 Neither team members nor their offsite colleagues could have any interaction with vendor technicians from IBM or Microsoft, other than through standard online customer support channels. If they did use vendor support channels, team members could not reveal that they were participating in a study involving IBM and Microsoft software; they received only the standard treatment afforded any developer on these channels. Note, however, that the WSAD implementation was developed after the RRD implementation, and was developed offsite, not in the controlled lab environment Development Phase When a team entered the lab for the first time, they were given the following initial environment: A development machine for each developer, pre-configured with Windows XP and Internet access. Two machines with the two ITS databases pre-installed and pre-populated with data. The database server was Microsoft SQL Server for the.net team, Oracle for the WebSphere team. Four application server machines pre-configured with the base OS installation only (Windows Server 2003 for the.net team, Red Hat Enterprise Linux 2.1 for the WebSphere team). As for augmenting or modifying this initial environment, both teams were under the same restrictions: They had to install/configure their development environment (tools, source control, etc) as part of the measured time to complete the application development phase. They had to install the application server software separately on each server as part of the measured development time. They could not make changes to the database schemas, other than adding functions, stored procedures, or indexes. This rule applied specifically to coding of the RRD and.net implementations: Team members were not allowed to work on code outside the lab. This meant they could not remove code from or bring code into the lab. For all implementations (RRD, WSAD and.net) this rule applied: We allowed use of publicly available sample code and publicly available pre-packaged libraries, since a typical corporate development team would also have access to such code Deployment and Tuning Phase When Phase 2 began, the CN2 auditor introduced the test environment that the team would be using. This environment consisted of: Mercury LoadRunner to simulate load A dedicated machine for the LoadRunner controller Some 40 additional machines to generate client load Base tests scripts created by CN2 so that the development team did not have to spend time doing so Page 23 of 109

24 5.4.4 Testing Phase The rules for Phase 3 were the most restrictive, since this phase consisted of the formal evaluation tests conducted by the CN2 auditor: Team members could not modify application code or system configurations except as needed during a test. After a load test was launched, the team would have to leave the lab until the test reached completion (typically 1-4 hours later). 5.5 The Evaluation Tests During Phase 3 the CN2 auditor conducted a variety of tests to measure manageability, reliability and performance of the implementations. Some of these tests required the active participation of the teams; others did not. Most of the tests were performed under load. As mentioned above, in this study Mercury LoadRunner running on 40 client machines was used to simulate load. CN2 provided the teams with a set of LoadRunner scripts for each implementation. The three sets of scripts were carefully constructed to perform the same set of actions, ensuring that they tested the exact same functionality for each implementation in a consistent manner. 2 Here is a summary of the tests performed; for more details and for test results see Sections 12 to 14. Performance capacity (stress test). How many users can the system handle before response times become unacceptable or errors occur at a significant rate? Performance reliability. Given a reasonable load (based on the results of the stress test), how reliably does the system perform over a sustained period (say, 12 hours)? Efficiency of message processing. How quickly can the Work Order module process a backlog of messages in the queue? Ease of implementing change requests. How quickly and easily can a developer implement a requested change to the specification? Ease and reliability of planned maintenance. How easily and seamlessly can system updates be deployed to the system while under load? Graceful failover. How well does the clustered Customer Service module respond when an instance goes down. Session sharing under load. If one of the clustered Customer Service instances fails under load, are the sessions that were handled by the failed instance seamlessly resumed by the other Customer Service instance? 2 CN2 could not provide a single set of scripts for all three implementations because the three differed in certain low-level details, such as the URL of a given page, the names of fields in that page and whether that page was to be invoked with GET or POST. Page 24 of 109

25 6 THE ITS PHYSICAL ARCHITECTURE This section describes the hardware and software infrastructure each team used to run its implementation of the ITS system. The specification required that the teams deploy to identical hardware; in fact, they used the same hardware. On the machines hosting the applications and the message server, each team had its own removable hard drive that was swapped in. On the machines hosting databases, the two teams DBMSs shared the same drive, but were never run simultaneously. In this way all three implementations used the very same processors, memory and network hardware. On the software side, the teams started with the operating systems and database engines already installed. They were responsible for installing the application server, message server, load balancing software and handheld device software. Page 25 of 109

26 This table lists the hardware and software used each by each team: ITS Subsystem Servers Hardware.NET Software J2EE Software Customer Service application Work Order Processing application 2 (identical, loadbalanced) 1 Hewlett Packard DL580 with GHz processors, 2GB of RAM and Gigabit networking Windows Server 2003.NET 1.1 development framework and runtime (part of Windows Server 2003) Red Hat Enterprise Linux AS 2.1 IBM WebSphere Network Deployment 5.1 Dedicated durable message queue server 1 Windows Server 2003 Microsoft MSMQ (part of Windows Server 2003) Red Hat Enterprise Linux AS 2.1 IBM MQSeries IBM WebSphere Deployment Manager IBM Edge Server Customer Service database Work Order database 1 1 Hewlett Packard DL760 with MHz processors and 4 GB RAM attached to a SANs network storage array with 500 GB of storage in a RAID 10 configuration Windows Server 2003 SQL Server 2000 Enterprise Edition Windows Server 2003 Oracle 9i Enterprise Edition Technician Mobile Device application n/a Hewlett Packard ipaq 5500 PocketPC.NET Compact Framework Insignia Jeode JVM Mobile Information Device Profile (MIDP) 2.0 Page 26 of 109

27 The following diagram shows the physical deployment of the ITS system to the network, including all the machines listed above. It also shows the machine hosting the Mercury LoadRunner controller and the 40 machines providing client load. Figure 2. ITS Connected System Physical Deployment Diagram 6.1 Details of the WebSphere Architecture The basic WebSphere infrastructure described below was used with both J2EE implementations. The J2EE team installed it during the RRD round and did not substantially change it during the WSAD round IBM WebSphere The team used WebSphere Network Deployment (ND) Edition version 5.1. This version of WebSphere has the same core functionality as basic WebSphere, but allows for central administration of multiple WebSphere instances across a network. It also allows instances to be clustered for the purpose of application deployment, so that, for example, one can deploy the Customer Service application to two WebSphere instances at once. Page 27 of 109