.NET 2.0 vs. IBM WebSphere 6.0 Data-Driven Web Application Server Performance Comparison

Transcription

1 .NET 2.0 vs. IBM WebSphere 6.0 Data-Driven Web Application Server Performance Comparison November 2005

2 The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property Microsoft Corporation. All rights reserved. Microsoft, the.net logo, Visual Studio, Win32, Windows, and Windows Server 2003 are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. Microsoft Corporation One Microsoft Way Redmond, WA USA

3 Executive Summary... 2 Full Disclosure Notice... 2 Implementations Tested... 2 Summary of Results... 3 The Benchmark... 4 Optimizations Made to the PlantsByWebSphere Application... 4 The.NET Benchmark Implementation: DotNetGardens... 7 Diagrams of the Implementations Tested... 8 PlantsByWebSphere EJB Implementation... 8 PlantsByWebSphere JDBC Implementation... 8 DotNetGardens.NET Implementation... 9 The Benchmark Hardware The Application Server The Database Server The Test Scripts Benchmark Results Peak TPS Price and Price/Performance Comparisons Tabular Peak TPS Results Tuning the IBM WebSphere Heap Size Conclusion Appendix 1: Pricing Pricing Notes Appendix 2: Tuning WebSphere 6 Tuning NET 2.0 Tuning Mercury Settings Appendix 3: New Microsoft SQL Server JDBC Driver Page 1

4 Executive Summary The goal of this paper is to present a comparison of Web application servers: the Microsoft.NET Framework 2.0 running on Windows Server 2003, and the latest release of IBM WebSphere Application Server, version According to a recent IDC study from October of 2005, roughly 78% of mission critical application projects under development worldwide use an application server. In this study, IDC also found that the combination of.net and Windows Server 2003 is now the most commonly used application server for mission critical applications, exceeding IBM WebSphere usage with 37% usage for.net/windows Server vs. 12% usage for IBM WebSphere 1. This paper presents the results of a new benchmark that compares the performance of those two application servers. The benchmark is based on the PlantsByWebSphere sample application, the primary sample application included with WebSphere 6. For this benchmark, we implemented a functionally equivalent.net application (DotNetGardens), and also made several performance optimizations to the IBM application code base to ensure a fair comparison of performance for data-driven Web applications. Full Disclosure Notice The complete source code, all test scripts and all testing methodology for this benchmark are available online. Any reader may download and view the actual code for all implementations tested, and may further perform the benchmark for themselves to verify the results. The benchmark kit can be downloaded from Implementations Tested The following shows the complete matrix of the implementations tested (all are tested on the exact same hardware). In the future, we also plan to publish a WebSphere EJB implementation that works with SQL Server 2005 using the new, soon-to-be released Microsoft SQL Server JDBC driver. Implementation Name Windows Server 2003 as App Server OS Backend Backend Oracle10G SQL Server DB 2005 DB PlantsByWebSphereEJB X X Red Hat Linux as App Server OS Backend Backend Oracle10G SQL DB Server 2005 DB PlantsByWebSphereJDBC X X X X DotNetGardens X X 1 See IDC Research 2005 Mission Critical Survey (Oct 2005) at Page 2

5 Summary of Results The results show that the.net 2.0/Windows Server 2003 implementation of the benchmark application running against SQL Server 2005 outperforms the Java EJB-based WebSphere /RedHat Linux implementation by up to 183%. The results also show that.net achieves significantly better price/performance ratios than WebSphere 6 on all implementations tested. This paper will also show that Java applications can get good performance when connecting to SQL Server 2005: the performance of the PlantsByWebSphere JDBC-based application that uses the soon-to-be-released Microsoft SQL Server 2005 JDBC driver is approximately equivalent to the performance of the equivalent application that connects to Oracle 10G. Plants Application Benchmark.NET 2.0 and WebSphere Peak TPS bit.NET 2.0/Oracle10G 32-bit WebSphere 6 EJB/Oracle10G Window s 32-bit WebSphere 6 JDBC/Oracle10G Window s 32-bit WebSphere 6 EJB/Oracle 10G Linux 32-bit WebSphere 32-bit.NET 2.0/SQL 6 JDBC/Oracle10G Server 2005 Linux 32-bit WebSphere 6 JDBC/SQL Server 2005 Window s 32-bit WebSphere 6 JDBC/SQL Server 2005 Linux Page 3

6 Total Middle Tier Application Server License Costs To Run Plants Applications As Tested $14, $12, $12, $10, $6, $8, $6, $4, $2, $- IBM WebSphere 6/RedHat Linux.NET 2.0/Windows Server 2003 See the Appendix for a detailed breakdown The Benchmark In order to conduct the benchmark in a fair and complete manner using an application that conforms to IBM s best coding practices, we chose the PlantsByWebSphere sample application as the basis for comparison, since IBM developed the Java application. The PlantsByWebSphere sample application is a J2EE-based application created by IBM according to their EJB-based development guidelines, and ships with every copy of WebSphere 6.0 as their primary J2EE sample application. The application uses a very typical J2EE design: Enterprise Java Beans including a stateless session bean façade to wrap entity beans that use container-managed persistence to access the database. The user interface is Web-based, relying on Java Server Pages and Servlets as the core architectural elements of the UI layer. The IBM application uses model objects to represent database information, and these model objects are passed between the EJBs and the UI layer to maintain a clean separation of UI, business and data layers of the application. Optimizations Made to the PlantsByWebSphere Application In order to provide a comprehensive assessment of the scalability and performance of the two platforms for this benchmark, several optimizations and additions were made to the PlantsByWebSphere sample application as provided by IBM with WebSphere. 1. Converted it to use an enterprise-class database. The original sample application is meant to work with a simple IBM CloudScape starter database, Page 4

7 which is not a suitable database to handle large concurrent user loads. While this benchmark is not focused on measuring database throughput, having a development-class database would present a bottleneck during performance tests. Hence, the PlantsByWebSphere EJB application was ported to work with an Oracle 10G Enterprise Database running on a dedicated server, via the Oracle 10G (Release 2) thin JDBC driver. The DotNetGardens implementation works with this same database, and also is designed to work with SQL Server Optimized image serving. The IBM implementation stores the product images in the database and uses an image servlet to retrieve them on each request and then render them in the browser. To improve performance of the application, instead of storing images as blobs in the database, we changed the application to store just a URL in the database, with the actual product images being served directly by the Web server (IBM HTTP Server for WebSphere and IIS 6.0 for.net). This has a dramatic impact on performance for all implementations tested, for further details on the performance of the image servlet (and a related asp.net image serving page), see the related benchmark paper titled: CachePerf: Examining the Impact of 64-bit Extended Memory Addressability on.net and WebSphere Middle Tier Applications at 3. Refactored several servlets into one. To enable users to easily use servlet caching to cache product data (search results, category browses, product details), several application functions were consolidated into a Category servlet so that if servlet caching is enabled for this servlet, these parts of the application can take advantage of it. Note that for the benchmark, however, no caching was used in any implementation (.NET or WebSphere) because we wanted to specifically test the performance of the data access portions of the application which is the bulk of the application logic. 4. Modified the EJBs to use local interfaces. Since the performance test is conducted on a single application server on which the EJBs and Web application are co-located, the EJB-based implementation was changed to use all local interfaces instead of remote interfaces to ensure the fastest invocation of EJBs from the JSP Web tier. 5. Introduced caching of JNDI lookups and local home interfaces, per IBM s best practice guidance as published in the IBM WebSphere 6.0 Performance and Scalability Redbook. JNDI lookups of cached local home interfaces are performed only in the initial servlet initialization (init) method. 6. Added a search feature to the application in order to be able to test random searches of the product inventory database, a typical user interaction with any ecommerce-style web application. Page 5

8 7. Added paged navigation of search results. The IBM sample application does not support paged navigation of record sets, so this functionality was added. 8. Modified the new search function to use JDBC record sets instead of EJBs. For the EJB implementation, we changed the search results page to use straight JDBC vs. EJBs to improve performance for the new product name search functionality we added. Using EJBs to perform a wildcard search using the EJBSQL Locate command is extremely slow, so by using JDBC instead, we were able to significantly increase the performance of the new search page for the EJBbased PlantsByWebSphere sample application. All of the other EJB logic remains intact in the tested Oracle10G/EJB implementation. 9. We removed the use of frames in the applications to make it easier for customers to benchmark them using a wide variety of load-test tools. We elected to make these changes to better reflect the design a customer would use when actually deploying an application on WebSphere Application Server in a production environment. All of these design changes are supported by the common design guidelines for J2EE applications. Each of these changes independently resulted in better performance of the modified application design, or better, more practical function. The aggregate affect of all of the performance-related changes on the actual performance of the application was significantly positive. Finally, in addition to those changes, we also constructed a new, alternative EJB-free implementation. There is some disagreement in the J2EE community about whether EJBs should be considered harmful, in terms of overall performance and scalability as well as application complexity. While some (including IBM) maintain that EJBs are the best architecture for building a data-driven Web application, to paint a complete picture of application performance, we created a second implementation of the PlantsByWebSphere application. This implementation uses no EJBs at all (either session beans or entity beans), but is precisely functionally equivalent to the original IBM EJB implementation and the DotNetGardens application. This implementation instead uses only JDBC logic to access the database. The JDBC logic is encapsulated in a separate data access layer (much like the ADO.NET implementation), activated via the same servlet architecture employed in the original EJB-based PlantsByWebSphere sample application. This second implementation shares much code and design with the original EJB-based implementation: the same model objects are used to pass data between tiers, the same base servlets and UI layer is used, and the other optimizations described above also apply. While we tested the EJB-based application against only the Oracle 10G backend, we tested this second implementation against Oracle10G as well as SQL Server 2005 using the new and soon-to-be released Microsoft SQL Server JDBC driver. A Beta version of this driver is available on MSDN for free download. The development of a new, optimized (and free) JDBC driver for SQL Server 2000/2005 by Microsoft with full support from Microsoft is an exciting development for J2EE developers that want to use Page 6

9 SQL Server 2005 with their J2EE-based applications. We expect to release this driver commercially in January 2006, and the Beta 2 release is available on MSDN. We are providing on the Web the full source to the optimized EJB implementation as well as the new JDBC/Servlet implementation as tested, in the benchmark kit. The.NET Benchmark Implementation: DotNetGardens We created a functionally equivalent application to PlantsByWebSphere, called DotNetGardens, in order to compare the performance of the two application server platforms. DotNetGardens operates against the same backend database schema as PlantsByWebSphere. The DotNetGardens application matches the functional behavior of the PlantsByWebSphere application, and employs a similar 3-tier logical architecture that maintains a clean separation of UI, business and data-tier operations. This 3-tier architecture is based on Microsoft-recommended best practices for enterprise development. The DotNetGardens application uses.net assemblies to encapsulate all data access in a single layer of data-access classes. These data access classes use ADO.NET 2.0 to provide the data access functionality of the application just as the EJB entity beans provide data access for the WebSphere application. The.NET implementation uses no stored procedures, and instead uses pre-prepared SQL statements to match the WebSphere implementation. 2 The data access classes are front-ended by a distinct business tier. Finally, the UI layer of the application was developed using ASP.NET 2.0 Web Forms. Like the WebSphere implementation, all data records are encapsulated in model classes that are passed between tiers to maintain a clean separation of UI, business and data tier operations. The DotNetGardens application will work against both an Oracle10G backend database using the.net Framework Data Provider for Oracle (which is integrated into the.net framework itself), and with SQL Server 2005/2000. The DotNetGardens application is also available for download as part of the benchmark kit. 2 In an earlier Web application performance comparison based on the Sun Java PetStore application (conducted in October 2002), many pointed to the use of SQL Server stored procedures as the primary reason.net outperformed the Java implementation. While a follow up study conducted by the Middleware Company (June 2003) showed the same performance results for.net with all stored procedures removed, we opted to use no stored procedures in DotNetGardens in order to remove this architectural issue altogether with the Plants benchmark. Page 7

10 Diagrams of the Implementations Tested PlantsByWebSphere EJB Implementation User Interface Catalog Servlet Account Servlet Shopping Servlet Admin Servlet Business Logic Catalog Login Shopping Cart BackOrderSes Suppliers Stateless Stateless Stateful Stateless Stateless Session Bean Session Bean Session Bean Session Bean Session Bean Data Access Inventory Customer Orders OrderItem BackOrder Supplier CMP Entity CMP Entity CMP Entity CMP Entity CMP Entity CMP Entity Bean Bean Bean Bean Bean Bean Ineventory Customers Orders OrderItem BackOrder Supplier PlantsByWebSphere JDBC Implementation User Interface Catalog Servlet Account Servlet Shopping Servlet Admin Servlet Business Logic Catalog Login Shopping Cart BackOrder Supplier Data Access Inventory Data Access Customer Data Access Orders Data Access BackOrder Data Access Suppliers Data Access Inventory Customers Orders OrderItem BackOrder Supplier Page 8

11 DotNetGardens.NET Implementation User Interface Product Detail Web Form Login Web Form Shopping Web Form Shopping Cart Web Form Account Web Form Business Logic Inventory Account Shopping Cart BackOrder Supplier Data Access Inventory Data Access Customer Data Access Orders Data Access BackOrder Data Access Supplier Data Access Inventory Customers Orders OrderItem BackOrder Supplier Page 9

12 The Benchmark Hardware The Application Server All implementations were tested on the following application server hardware: HP DL585 4 x 1.8 GHz AMD Opteron server 4GB RAM Gigabit networking The software installed on this server included the following OS/App Server combinations: Windows Server 2003 Standard Edition x86 (32-bit) 32-bit WebSphere Network Deployment Edition with the IBM HTTP Server (Apache) Oracle 10G Client, Release bit.NET 2.0 retail product for x86 (build is the commercially released build available on MSDN) Red Hat Linux Advanced Server (Release 4) for AMD Opteron 32-bit WebSphere Network Deployment Edition with the IBM HTTP Server (Apache) The Database Server All implementations were tested with the following dedicated database server hardware: HardDrives Northwest dual-core AMD Opteron 2 x 2.4 GHz server (4 processors total) 16GB RAM Gigabit networking Two dedicated fast SCSI controllers Two RAID arrays, 14 drives each configured in RAID 0+1 configuration (One RAID array used for core datafiles, one used for logging) The software installed for the databases included: Windows Server 2003 x86 Enterprise Edition (32-bit) Oracle 10G Enterprise Edition Release 1 for Windows x86 SQL Server 2005 Enterprise Edition commercial release for x86 The database was closely monitored to ensure that in no case it represented a bottleneck on performance (either in terms of CPU utilization, network utilization, or physical disk Page 10

13 utilization). In other words, the benchmark results indicate the top-end performance achieved by the application server for each implementation tested given plenty of database capacity to handle all peak-performance loads. In all benchmark runs, we were able to fully saturate the application server/tested benchmark implementation at above 96% CPU utilization to accurately capture as intended the peak throughput of the application server itself, inclusive of the middle-tier data access technology used by the various implementations. It is important to remember that this is not a database benchmark, since both SQL Server 2005 and Oracle10G could have handled user loads well-beyond those supported by the middle tier hardware/software platforms we tested. Instead the benchmark measures the performance of the middle tier and how fast it is at connecting to and working with various backend databases in a Web-based scenario, using the most commonly employed data access techniques and drivers for each platform. The Test Scripts Mercury LoadRunner 7.51 was used to benchmark the various implementations. The LoadRunner test scripts are published with the source code so that customers can easily test the applications for themselves. 20 physical agents machines were used to generate load, and all agents were set to run with a 1-second think time. Hence, peak TPS rates also roughly equate to the number of concurrent users supported on the tested platform for each implementation tested. The scripts exercised the following functionality in the application: 1. Home Page 2. Category Browse 3. Add to Cart 4. Category Browse 5. Add to Cart 6. Search (25% wildcard/multiple results, 75% exact match) 7. Product detail 8. Search (25% wildcard/multiple results, 75% exact match) 9. Product detail 10. Search (25% wildcard/multiple results, 75% exact match) 11. Product detail 12. Search (25% wildcard/multiple results, 75% exact match) 13. Product detail 14. Search (25% wildcard/multiple results, 75% exact match) 15. Product detail 16. Login 17. Checkout 18. Finalize Checkout 19. Submit Order 20. View/Modify Account Information For the benchmark, the Plants customer database was loaded with 10,000 unique customers, and the inventory database was loaded with 20,000 unique products. Page 11

14 Benchmark Results Peak TPS Plants with SQL Server 2005 Backend Database Peak TPS bit.NET 2.0/SQL Server bit WebSphere 6 JDBC/SQL Server 2005 Window s 32-bit WebSphere 6 JDBC/SQL Server 2005 Linux Plants with Oracle10G Backend Database Peak TPS bit.NET 2.0/Oracle10G 32-bit WebSphere 6 EJB/Oracle10G Window s 32-bit WebSphere 6 JDBC/Oracle10G Window s 32-bit WebSphere 6 EJB/Oracle 10G Linux 32-bit WebSphere 6 JDBC/Oracle10G Linux Page 12

15 Price and Price/Performance Comparisons To aid customers in comparing value, and not just raw performance, we priced the software licenses required to support the application servers used in each of the configurations tested (exclusive of the backend database license costs). To build and run the PlantsByWebSphere application as configured for IBM WebSphere Application Server Express v6.0 3 on Red Hat Linux ES, the total software license cost is $12,054. To build and run the functionally equivalent DotNetGardens application with.net 2.0 on Windows Server 2003, the software license cost is $6, The IBM configuration is nearly twice as expensive in middle-tier software license costs. Total Middle Tier Application Server License Costs To Run Plants Applications As Tested $14, $12, $12, $10, $6, $8, $6, $4, $2, $- IBM WebSphere 6/RedHat Linux.NET 2.0/Windows Server 2003 See the Appendix for detailed breakdown and information sources. The following chart depicts the Price/Performance ratio. The numbers in this chart are computed by dividing the software license cost for each configuration, by the peak TPS achieved by that configuration. This is a measure of the value of the application server software. For example, an application server that performs twice as well but at double the cost would equate to an equivalent price/performance score. Lower price performance numbers represent greater value (lower cost/better performance). No database license cost is included in the calculations below, it is based solely on the 3 We priced the least expensive IBM WebSphere edition. Page 13

16 middle-tier software license cost. See the appendix for the detailed cost breakdowns and their sources. Price Performance Comparison (lower bar is better) $12.00 $11.70 $10.00 $8.00 $5.87 $5.73 $ Cost Per TPS $6.00 $4.00 $2.29 $3.70 $2.00 $- 32-bit.NET 2.0/SQL Server bit.NET 2.0/Oracle10G 32-bit WebSphere 6 JDBC/SQL Server 2005 Linux 32-bit WebSphere 6 JDBC/Oracle10G Linux 32-bit WebSphere 6 EJB/Oracle 10G Linux Tabular Peak TPS Results In most cases a valid benchmark result of peak TPS rates should be reported at the point the CPUs of the system under test are saturated in other words, it should be running at or near 100% CPU utilization without over-saturation. When the CPU capacity is saturated, the machine is achieving peak throughput for the given configuration and application because at 100% CPU utilization, there are no other bottlenecks in the system other than the processing efficiency of the system under test. This table shows the CPU levels and peak transaction rates achieved, for each different application implementation that we tested: App Server CPU Utilization Peak Implementation Peak TPS WAS 6 EJB/Oracle10G/Windows Server % WAS 6 JDBC/Oracle10G/Windows Server % WAS 6 JDBC/SQL Server 2005/Windows Server %.NET 2.0/Oracle10G/Windows Server %.NET 2.0/SQL Server 2005/Windows Server % WAS 6 EJB/Oracle 10G/RH Linux Enterprise Server % WAS 6 JDBC/Oracle10G/RH Linux Enterprise Server % WAS 6 JDBC/SQL Server 2005/RH Linux Enterprise Server % Tuning the IBM WebSphere Heap Size One of the most sensitive tuning areas for many J2EE applications is the java heap size. We tested all IBM implementations with various heap size settings to determine the Page 14

17 optimal heap sizes for both the EJB and JDBC-based implementations. For this application we did not see significant performance gains if the heap size was configured beyond 1000 megabytes of RAM, since the application itself has a small memory footprint. We also found that the EJB-based implementation performed at its best with a larger heap size than the JDBC implementation, possibly because of the memory usage for the EJB cache. We did find that increasing the heap size beyond 1800MB for the JDBC implementation caused a slight decrease in performance, probably because of larger overhead associated with garbage collection in a larger heap. In other applications, this threshold will be different. Note that it is not necessary to optimize the heap size with a web application based on.net running on Windows Server While we did not measure the time and labor consumed by performing this optimization on the WebSphere application, it can often represent a significant amount of skilled labor. In addition, this analysis must be performed again, each time the profile of the application (either the source code, or the machine configuration, or both) changes significantly. The following tables show the data we measured to help us choose the heap size for both the JDBC and EJB implementations. The data presented here is for the RedHat Linux OS. However, we followed the same analysis for WebSphere 6 on the Windows OS. For a complete list of all tuning parameters set for both IBM WebSphere and.net, please refer to the appendix. Heap Tuning JDBC/Linux Peak TPS Heap Size (MB) Page 15

18 Heap Tuning EJB/Linux Peak TPS Heap Size (MB) Conclusion The results of this benchmark show that: The peak performance of.net 2.0 as an application server when running with a Microsoft SQL Server 2005 backend database in this benchmark exceeds that of the IBM WebSphere 6.0 optimized EJB implementation by over 183%. The.NET 2.0 implementation running against an Oracle 10G backend database exceeds the performance of the optimized IBM EJB/J2EE implementation by 75%. The WebSphere JDBC implementation we created provides significantly better performance than the IBM-provided EJB implementation for this application. The PlantsByWebSphere JDBC implementation running against a SQL Server 2005 backend database using the soon-to-be-released Microsoft-released JDBC provider for SQL Server roughly matches the performance of the same application running against an equivalent Oracle 10G database. The middle tier license costs for IBM WebSphere 6.0 (Express) on Red Hat Linux is almost twice that of the middle-tier license costs required to run an equivalent.net application on Windows Server NET has a significantly better price/performance ratio than IBM WebSphere 6 across all implementations of the benchmark tested..net delivers up to a 4x better performance per dollar as compared to WebSphere in this non-clustered benchmark. For tests that include clusters of application servers running IBM Page 16

19 WebSphere Network Deployment Editon vs. the lower end WebSphere Express product, the price/performance advantage of Microsoft Windows and.net can increase significantly. Results in an actual data-driven Web application will vary. Most customer applications are more complex than this benchmark. No benchmark can hope to capture all actual real-world requirements for all customers, and no single benchmark can answer all performance questions. Nevertheless this benchmark is interesting for enterprise developers because the application, while simple, uses and exercises many of the same enterprise architectural building blocks used in complex, real-world applications. We hope this benchmark provides solid evidence that.net 2.0 performance and price/performance for building and running a data-driven Web application can be superior to IBM WebSphere 6. We encourage customers to download the benchmark kit, modify it according to their own environment, and perform their own tests. Page 17

20 Appendix 1: Pricing The following table shows the pricing breakdown for the application server middle tier to run the Plants application on a single 4-cpu middle tier server, as done in this benchmark. The pricing compares.net running on Windows Server 2003 to IBM WebSphere Express running on Red Hat Linux 4. The pricing is inclusive of operating system license cost for a supported version of each OS, the cost of the application server itself, and one developer seat license for the Microsoft and IBM development tool used to build the applications. Note that database pricing is not included in this chart, please refer to Oracle and SQL Server 2005 pricing comparisons at if you want to compare database pricing. Microsoft.NET 2.0 vs. IBM WebSphere 6.0 License Cost Comparison As Configured for the Plants Benchmark Item Microsoft Product Cost IBM Product Cost Base Server OS Windows Server 2003 Standard $ / system $ Red Hat Enterprise Linux 1 year subscription $ Application Server Function (Included in Windows Server) -- WebSphere Application Server Express, $2000/cpu $ Unlimited External Connections External $ / system $ (Included in WAS Express) -- One Developer Visual Studio 2005 $ Rational $ Note that we priced the less expensive IBM WebSphere Express even though we tested with the enterprise-recommended IBM WebSphere Network Deployment Edition because technically on a single 4- processor server a customer could use IBM Websphere Express vs. the more expensive Network Deployment Edition and reduce their WebSphere license costs. However, should a customer want to implement the Plants application in a cluster with load balancing and shared session state, they would typically license IBM WebSphere Network Deployment Edition which would cost an additional $13,000 per CPU vs. Express on each server it is deployed to. The.NET implementation supports network load balancing and shared session state for clusters with no additional license cost beyond purchasing additional copies of the operating system for each server in the cluster (which also applies to WebSphere clusters). Page 18

21 seat Professional with MSDN Premium Subscription, 1 annual charge ** see note Application Developer for WebSphere software, floating user, 1 st year Developmenttime runtime license (Included in MSDN Enterprise subscription) -- WAS Express Developer User $400 Media CD kit for Windows Server 2003 Standard Edition $28.00 (included in IBM.com price) $-- Total $ $ Pricing Notes Costs for IBM were obtained from the IBM.com online store, and the Red Hat online store. Red Hat Enterprise Linux AS is required, as opposed to the less expensive Red Hat Enterprise Linux ES, because this configuration uses 4 cpus, and the maximum number of CPUs supported by the Red Hat Enterprise Linux ES license is 2. Windows Server 2003 Standard Edition supports up to 4 CPUs. A single seat license for the RWD Developer tool is included in WAS Express, but the EJB implementation of the Plants app (as originally developed by IBM) requires support for EJBs, and therefore we separately priced Rational Application Developer for WebSphere 6 to meet this requirement. The price of a floating user license for RAD is $ for the initial year under a current (as of October 2005) promotion; the normal price is $7000/user. License costs for all Microsoft software were obtained from a Microsoft authorized reseller, except for Visual Studio **Note: The pricing for Visual Studio 2005 Professional with MSDN Premium Subscription is estimated retail pricing, not actual reseller pricing. The actual reseller price of this item was not available at the time of printing, but will likely be slightly lower than the estimated price used here. Page 19

22 Appendix 2: Tuning WebSphere 6 Tuning WAS Session State set to 5 minute expiration (in-process session state) Access Log Turned Off Performance Monitor Infrastructure Turned Off Diagnostic Trace Turned Off System Out Off EJB Cache Size = HTTP Channel maximum persistent requests = -1 HTTP Channel readtimeout = 6000 HTTP Channel writetimeout = 6000 HTTP Channel persistenttimeout = 3000 Minimum Web Container threads = 75 Maximum Web Container threads = 75 Minimum ORB threads = 40 Maximum ORB threads = 40 Minimum Default threads = 20 Maximum Default threads = 20 Heap Sizes: MB EJB Implementation on Windows Server 2003 (this is max for 32-bit WebSphere on Windows) MB EJB Implementation on Linux (this is max for 32-bit WebSphere on Linux) MB JDBC Implementation on Windows MB JDBC Implementation on Linux EJB implementation: Access intent policy set to Optimistic Read Isolation Level for read methods of the Inventory and Customer entity beans IBM HTTP Server IBM HTTP Server Windows: Access Log Off Max KeepAlive Requests Max threads 2048 Threads/child IBM HTTP Server Linux: Access Log Off Max KeepAlive Requests 3000 ThreadLimit 50 ServerLimit 64 StartServers 50 MaxClients 3200 MinSpareThreads 100 MaxSpareThreads 100 Threads/Child 50 MaxRequests/Child 0 Linux Tuning net.ipv4.tcp_max_syn_backlog=1024 kernel.msgmni=1024 kernel.sem= Page 20

23 fs.file-max=65535 kernel.shmmax = net.core.netdev_max_backlog = net.core.somaxconn = net.ipv4.tcp_fin_timeout = 30 net.ipv4.tcp_syn_retries = 20 net.ipv4.tcp_synack_retries = 20 net.ipv4.tcp_sack = 0 net.ipv4.tcp_timestamps = 0 net.ipv4.conf.all.arp_ignore = 3 net.ipv4.conf.all.arp_announce = 2 Open File Handle limit (soft) increased to Windows Server 2003 Tuning Added registry parameter settings for tcpip service (CurrentControlSet/Services/tcpip/parameters): TcpTimedWaitDelay set to decimal 30 MaxUserPort set to decimal NET 2.0 Tuning.NET Worker Process Rapid Fail Protection off Pinging off Recycle Worker Process off ASP.NET Session state timeout set to 5 minutes ((in-process session state) Authentication set to None to match anonymous access of IBM WebSphere IIS 6.0 Virtual Directory Authentication Basic Only Access Logging Off Mercury Settings Agents were set to: 1. Simulate a new user on each request (each script iteration opens, then at the end closes its own connection to the app server, although http keepalives are enabled in all cases). 2. Download only HTML resources returned by the application server, and not graphics files or any other non-html resources. Page 21

24 Appendix 3: New Microsoft SQL Server JDBC Driver The PlantsByWebSphere Java application we created for use with an MS SQL Server backend will operate with the Beta2 release of the new Microsoft SQL Server JDBC driver available on MSDN. This driver is based on a completely different codebase from our prior driver for SQL Server 2000 and offers up-to-date features and better performance. However, we tested with a pre-beta2 release of this driver do to a current bug in the Beta 2 release that affects only the new search logic we added to the application (a single query in the app). This bug prevents SQL Server from properly using its query optimizer on LIKE clauses when run via the beta JDBC driver. We have since addressed this bug for the upcoming commercial release. However, to replicate our results or perform further benchmarks on the SQL Server Java application in the benchmark kit, customers can obtain a copy of the pre-beta 2 build we used by ing benchnet@microsoft.com. We are also working on a PlantsByWebSphere EJB implementation for SQL Server 2005 that will demonstrate the use of this new driver with Container Managed Persistence/EJBs. Page 22