White Paper Proving Scalability: A Critical Element of System Evaluation Jointly Presented by NextGen Healthcare & HP NextGen Healthcare Information Systems, Inc. www.nextgen.com
A Successful Load Test Can Prove Scalability When investing millions of dollars in an electronic medical records (EMR) implementation, scalability cannot be an after-thought especially for large healthcare companies with thousands of users depending on access to millions of patient records. With this in mind, NextGen Healthcare partnered with Hewlett-Packard Company (HP) to demonstrate the scalability of its Windows-based EMR application under the pressure of thousands of users during a load test simulation conducted for a large prospective client. The results of the load test simulation provided the prospective client, which employs about 2,100 end-users, with concrete evidence that NextGen EMR software, paired with the HP high-performance database servers, could more than handle what the firm described as a typical day in their busy, 350- location practice. This document describes the testing environment, database staging, load test scripts, and proven results. Testing Environment NextGen Healthcare conducted the testing at HP s Partner Technical Access Center (PTAC-MA/TX) in Marlborough, MA. HP offers usage of the lab to its third-party software partners to provide an environment to test or demonstrate the partners applications using state-of-the-art HP equipment. The lab provided a controlled environment to test the stability of NextGen s EMR application when scaled up to 3,000 users simultaneously accessing patient data on an SQL Database Server. NextGen Healthcare staged 15 million patient records in an SQL Server 2005 (64-bit) database on an HP Integrity rx8620 server, featuring industry-standard Intel Itanium 2 processors. The HP Integrity server was running the Microsoft Windows Server 2003 operating system, and an HP StorageWorks 5000 Enterprise Virtual Array (EVA5000) provided additional storage space. Two HP ProLiant DL580 servers running load generator software and one HP ProLiant DL140 server running load controller software were used to emulate the concurrent users. Finally, three workstations were connected to the staged system so live users could perform functions in the EMR and monitor the application s performance from a true end-user perspective. -2-
Testing Environment Network Diagram (See Appendix B for a complete list of hardware specifications.) -3-
Database Staging HP Integrity rx8620 Database Server Staging for the Load Test To replicate the production environment of a large healthcare organization s patient database, a representative female patient record and a male patient record were created. Each was replicated 7.5 million times while randomizing the first and last names, addresses, social security numbers and medical record numbers. Each patient had four visits, which were multiplied by 15 million to create a total of 60 million visits. Each initial visit was unique and modeled after a typical patient visit in an occupational health practice. The data from these visits was contained in over 180 tables in the database. The number of records in these tables mostly varied from 15 million to 60 million records, but the largest table contained 850 million records. The total database size was approximately 2.32 TB. Load Test Scripts With direction from the prospective client, NextGen Healthcare wrote seven scripts depicting common scenarios that are performed in the client s legacy EMR during any given day. The pool of 3,000 virtual users running through each script was broken down into the following percentages, based on the large healthcare firm s real world experience. Of the 3,000 virtual users, the test assumed 950 were full-time providers seeing 32 patients per day, creating a total of 30,400 encounters. Summary of Scripted Actions Script Name % of Users Running Script (Based on 3,000 users) No. of Steps in Script Nurse Intake 25% 18 Physician Exam 20% 134 Abbreviated Nurse Intake 10% 13 Medication Change 5% 28 Physical Therapy Evaluation & Treatment Pre-Employment Physical Nursing Intake 15% 46 15% 22 Provider Pre-Employment Exam 10% 16-4-
Description of Scripts Nurse Intake: Includes documentation of visit type, past medical history review, vital signs, and indication that drug screen has been performed. Physician Exam: Includes documentation of patient complaints, exam findings, and procedure (laceration repair). Abbreviated Nurse Intake: Includes visit type selection, review of open orders, and lab result data entry. Medication Change: Active medication list for patient accessed and meds updated. Physical Therapy Evaluation & Treatment: Includes documentation of visit type, chief complaint, physical therapy information and treatment codes. Pre-Employment Physical Nursing Intake: Includes documentation of visit type, vital signs, audiometry and vision screening, lab orders, and chest x-ray order. Provider Pre-Employment Exam: Includes documentation of encounter, digital image added to patient s record, and physician sign-off. Examples of EMR Screens Processed During Scripted Simulations During the Physician Exam script, virtual physicians accessed the Master Orthopedic screen and entered the reason for the encounter. -5-
During the Physical Therapy Evaluation & Treatment script, virtual users simulated accessing the physical therapy home page, and entering the visit type and chief complaint. During the Pre-Employment Physical Nursing Intake script, labs were ordered after vital signs and audiometry and vision screening were entered. -6-
Results The Microsoft SQL Database Server performance was measured using the Microsoft performance monitor application that ships with the server s operating system. It was configured to sample key performance counters for the operating system and the SQL Server process every 10 seconds. The performance monitor application recorded the information in the native binary format. Below are the results of three major performance indicators that demonstrate the stress on the database server as the user load ramped up to 3,000 concurrent users. SQL Transactions Per Second: Average number of transactions per second was approximately 600, with spikes approaching 1,400. While the simulation was running, the three live end-users working in the EMR application experienced an average of 1-2 second screen refreshes. (See Appendix A, graphs 1 & 2.) CPU Utilization: Average CPU was approximately 46%, with some processors running under 25% and a few experiencing short duration spikes of 90%. (See Appendix A, graph 3.) Disk I/O: The average bytes read was about 4,000,000 per second, and the average bytes written was about 850,000 bytes per second, with short spikes reaching 22,000,000. As the duration of the test increased, there was a drop in the number of bytes read from the disk, indicating that the database was starting to replace actual disk reads with buffered data. (See Appendix A, graph 4.) The data generated by these three major performance indicators during the load test show that when scaled up to 3,000 concurrent users, NextGen EMR performed with: Speed: End-users experienced quick screen loads even with hundreds of transactions processing per second. Reliability: On average, more than half the CPU remained available, indicating that the database was far from being taxed and even had additional processing power for further scalability. Efficiency: The drop in disk bytes read during the test shows that the database was starting to improve sequential disk I/O operations by allocating memory for reading in larger blocks of data, which will help improve average response time even further. To find out more about NextGen Healthcare, contact a representative at (215) 657-7010 or visit www.nextgen.com. -7-
Appendix A: Results Graphs 1,600.00 1,400.00 1,200.00 1,000.00 800.00 600.00 400.00 200.00 0.00 9:08:47 AM 9:12:12 AM 9:15:37 AM 9:19:02 AM 9:22:27 AM 9:25:52 AM 9:29:17 AM 9:32:42 AM 9:36:07 AM 9:39:32 AM 9:42:57 AM 9:46:22 AM 9:49:47 AM 9:53:12 AM 9:56:37 AM 10:00:02 AM 10:03:27 AM 10:06:52 AM 10:10:17 AM 10:13:42 AM 10:17:07 AM 10:20:32 AM 10:23:57 AM 10:27:22 AM 10:30:47 AM 10:34:12 AM 10:37:37 AM 10:41:02 AM 10:44:27 AM 10:47:52 AM 10:51:17 AM 10:54:42 AM 10:58:07 AM 11:01:32 AM 11:04:57 AM 11:08:22 AM 11:11:47 AM Graph 1. SQL Transactions Per Second Ave. number of transactions per second was approximately 600, with spikes approaching 1,400 transactions. Graph 2. Average Response Time Using the Physical Exam s HPI Back template response time as a representative sample, this graph shows that it took an average of 1.6 seconds for templates to display when accessed by users. -8-
100 90 80 70 60 50 40 30 20 10 0 9:08:47 AM 9:11:57 AM 9:15:07 AM 9:18:17 AM 9:21:27 AM 9:24:37 AM 9:27:47 AM 9:30:57 AM 9:34:07 AM 9:37:17 AM 9:40:27 AM 9:43:37 AM 9:46:47 AM 9:49:57 AM 9:53:07 AM 9:56:17 AM 9:59:27 AM 10:02:37 AM 10:05:47 AM 10:08:57 AM 10:12:07 AM 10:15:17 AM 10:18:27 AM 10:21:37 AM 10:24:47 AM 10:27:57 AM 10:31:07 AM 10:34:17 AM 10:37:27 AM 10:40:37 AM 10:43:47 AM 10:46:57 AM 10:50:07 AM 10:53:17 AM 10:56:27 AM 10:59:37 AM 11:02:47 AM 11:05:57 AM 11:09:07 AM Graph 3. CPU Utilization The orange line shows total average CPU utilization, which was approximately 46%. The green and blue lines represent short-term spikes of the individual processors to provide a reference for the maximum/minimum range of data. 30,000,000 27,000,000 24,000,000 21,000,000 18,000,000 15,000,000 12,000,000 9,000,000 6,000,000 3,000,000 0 Time 9:11:57 AM 9:15:17 AM 9:18:37 AM 9:21:57 AM 9:25:17 AM 9:28:37 AM 9:31:57 AM 9:35:17 AM 9:38:37 AM 9:41:57 AM 9:45:17 AM 9:48:37 AM 9:51:57 AM 9:55:17 AM 9:58:37 AM 10:01:57 AM 10:05:17 AM 10:08:37 AM 10:11:57 AM 10:15:17 AM 10:18:37 AM 10:21:57 AM 10:25:17 AM 10:28:37 AM 10:31:57 AM 10:35:17 AM 10:38:37 AM 10:41:57 AM 10:45:17 AM 10:48:37 AM 10:51:57 AM 10:55:17 AM 10:58:37 AM 11:01:57 AM 11:05:17 AM 11:08:37 AM Graph 4. Disk I/O The dotted green represents measured values for actual bytes read per second, with the solid green line showing a running 10-point average for the bytes read. The dotted blue represents measured values for actual bytes written per second, with the solid blue line showing a running 10-point average for the bytes written. -9-
Appendix B: Hardware Specifications HP Integrity rx8620 Database Server Product Specifications System Description Computer Name Server Hardware Server Operating System Additional Software Processors Memory Server Drives (All 10K RPM Drives) Additional Disk Space SQL Database Server spa231 HP Integrity rx8620 Windows Server 2003 Datacenter Edition 64-bit SQL Server 2005 64-bit 16 x 1.5 GHz (Intel Itanium 2 processors) 64GB 2x 146GB, Raid 1, OS Mirror 2x 146GB, Raid 1, SQL Log 4x 146GB, Raid 5, NextGen Files / Document Images 1x 6TB HP StorageWorks 5000 Enterprise Virtual Array 1.8TB, Raid 10, SQL Database 800GB, Raid 10, SQL Indexes 400GB, Raid 10, SQL Temp DB Specifications for Load Generators and Controller Load Test System Specs for LoadRunner Load Generator 1 System Description LoadRunner Load Generator 1 Computer Name spa18 Server Hardware HP ProLiant DL580 G3 Server Operating System Windows Server 2003 Enterprise Edition Additional Software LoadRunner Load Generator Processors 4 x 3.6 GHz Intel Xeon Memory 16GB Server Drives 4 x 146GB Load Test System Specs for LoadRunner Load Generator 2 System Description LoadRunner Load Generator 2 Computer Name spb50 Server Hardware HP ProLiant DL580 G3 Server Operating System Windows Server 2003 Enterprise Edition Additional Software LoadRunner Load Generator Processors 4 x 3.6 GHz Intel Xeon Memory 16GB Server Drives 4 x 72GB Load Test System Specs for LoadRunner Controller System Description LoadRunner Controller Computer Name spb66 Server Hardware HP ProLiant DL140 G1 Server Operating System Windows XP SP2 Additional Software LoadRunner Controller 8.1.3 Processors 2 x 3.2 GHz Xeon Memory 2GB Server Drives 226 GB -10-
About NextGen Healthcare NextGen Healthcare Information Systems, Inc. a wholly owned subsidiary of Quality Systems, Inc. (NASDAQ: QSII), develops and markets computer-based practice management and electronic medical records systems for medical group practices. For more information about NextGen, please visit www.nextgen.com and www.qsii.com. NextGen EMR is an innovative electronic medical records system designed to help large, growing practices continue to improve quality of care, reduce risk, cut costs, and increase revenues. This feature-rich system allows healthcare organizations to create and manage medical records faster and more efficiently than ever before. About HP HP is a technology solutions provider to consumers, businesses and institutions globally. The company s offerings span IT infrastructure, global services, business and home computing, and imaging and printing. For the four fiscal quarters ended October 31, 2006, HP revenue totaled $91.7 billion. More information about HP (NYSE, Nasdaq: HPQ) is available at www.hp.com. -11-