Benchmark Testing Results: Telvent ArcFM Solution Running on Microsoft SQL Server Benchmark testing confirms the superior performance and scalability of Telvent ArcFM Solution 10.0.1 SP1 running on SQL Server 2008 R2 Enterprise and Windows Server 2008 R2 Enterprise Technical White Paper Published: October 2011 Applies to: Microsoft SQL Server 2008 R2 Enterprise Abstract In May and June 2011, Telvent and Microsoft worked together to conduct a series of performance and scalability tests for Telvent ArcFM Solution 10.0.1 service pack 1 (SP1) running on Microsoft SQL Server 2008 R2 Enterprise and Windows Server 2008 R2 Enterprise. The benchmark results validate that ArcFM Solution is a highly scalable system on SQL Server under realworld conditions simulating a large utility with 10 million customers. The ArcFM geographic information systems (GIS) functionality of ArcFM Solution running on SQL Server demonstrated excellent performance and scalability under a heavy load. The Responder functionality also performed very well under a heavy call load, handling 70,000 incoming calls per hour for 15 hours. The benchmark results confirm that ArcFM Solution on SQL Server can scale and perform to meet the needs of the world s largest utilities. Utilities can be confident that ArcFM Solution 10.0.1 SP1, deployed on SQL Server 2008 R2 Enterprise and Windows Server 2008 R2 Enterprise, provides a reliable foundation for a smart grid solution with a leading price-to-performance ratio.
2011 Microsoft Corporation. All rights reserved. This document is provided as-is. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes. 2
Table of Contents Introduction... 4 Brief Overview of Telvent ArcFM Solution... 5 ArcFM... 6 Responder... 6 Benefits of SQL Server for ArcFM Solution... 7 Benchmark Testing Confirms Performance and Scalability of ArcFM Solution on SQL Server... 9 Benchmark Testing Goals... 9 Benchmarking Test Scenarios... 10 Test Database... 10 ArcFM Testing Process... 11 Responder Testing Process... 12 Benchmark Testing Setup... 13 Database and Application Servers... 13 Initial Setup and Tuning... 14 Benchmark Testing Results... 14 Summary of Results for ArcFM... 15 Summary of Results for Responder... 16 Conclusion... 16 Additional Information... 17 About Telvent... 17 About the Microsoft Worldwide Utilities Group... 17 3
Introduction Today s utilities face the challenges of tighter IT budgets, reduced in-house development staff, and the need to quickly deliver business benefits through applications. Utilities also need to adapt to the rapid changes in smart grid technology, the electricity distribution system of the future. Telvent and Microsoft work together to provide integrated solutions that can help utilities meet these challenges and better use their distribution assets for maintenance and growth. By working together to integrate key aspects of their solutions including Telvent ArcFM Solution, Microsoft SQL Server, and Windows Server Telvent and Microsoft address the most pressing needs of the smart grid age, including performance metrics, response times, and security. ArcFM Solution running on SQL Server and Windows Server provides a complete, out-ofthe-box solution for facilities management designed to meet the needs of utility businesses. The solution offers a map-centric, intuitive way to model, design, maintain, and manage facility and land based information. In May and June 2011, Telvent, a Microsoft Gold Certified Partner, joined with Microsoft to conduct a series of benchmarking tests to verify the performance and scalability of ArcFM Solution 10.0.1 SP1 running on SQL Server 2008 R2 Enterprise and Windows Server 2008 R2 Enterprise. The results of the benchmark testing were impressive. Highlights of the results include: ArcFM Solution on SQL Server was able to support the requirements of a typical 10 million customer utility for all processes tested. The Responder functionality successfully processed 70,000 calls per hour for 15 consecutive hours (with no user load simulated during the testing). The ArcFM Solution user experience in the benchmark test was comparable to the user experience in tests run locally on significantly smaller datasets and with a smaller load. ArcFM Solution running on SQL Server performed equally well on a large system (32 cores, 512 GB RAM) and a smaller system (8 cores, 24 GB RAM). This white paper presents an overview of ArcFM Solution and of SQL Server, and details the results of the benchmark testing. It also includes links for further information. 4
Brief Overview of Telvent ArcFM Solution Telvent ArcFM Solution is an extension of the ESRI ArcGIS platform, a market-leading geographic information systems (GIS) software platform. ArcFM Solution consists of a family of models and a set of sophisticated tools that provide a comprehensive application for electric, gas, water, and wastewater utilities, and for telecommunication providers. ArcFM Solution enables maintenance and data viewing using companyspecific business rules, and includes applications to configure the data models and tools to work together. Telvent applications gain leverage for enterprise GIS through the use of open technologies. Key components of this strategy include an industry-standard, opendevelopment environment, open databases, and Component Object Model (COM) architecture. All Telvent application software takes advantage of this object-oriented architecture and provides for fully networked solutions in a client-server environment. The ArcFM Solution suite was developed using Visual Basic 6 (VB6), C++, and C#; these languages can also be used for customization and extension of the applications. Figure 1 shows the ArcFM Solution architecture for utilities. Figure 1. ArcFM Solution architecture for utilities For more information, see ArcFM Solution Overview. 5
Microsoft and Telvent conducted benchmark testing on two core elements of the ArcFM Solution ArcFM and Responder which together provide a solid foundation for the smart grid. ArcFM ArcFM is a complete GIS utility solution for modeling, editing, maintaining, and managing facility asset data both in the field and on the desktop. ArcFM includes tracing tasks to automate utility operations, and an extensive set of editing tools such as automated snapping and attribute inheritance. ArcFM also includes tools to quickly define and recall map sheet collections for map production. Users can create and edit a session in the field, then send it back to the enterprise to be used for editing the database, or geodatabase, using the Session Manager (see Figure 2). Figure 2. ArcFM Session Manager For more information about ArcFM, see Telvent Smart Grid Solutions Suite: ArcFM Enterprise GIS. Responder Responder is a GIS-hosted outage management system (OMS) that takes advantage of Microsoft.NET and ASP technologies to enable trouble call and outage incident management in a web-based, scalable, and configurable desktop environment. The Responder Trouble Call Analysis (TCA) Engine uses a sophisticated and iterative prediction algorithm to determine which interruptible network device caused an outage. High-performance GIS display capabilities provide utility personnel with a spatial view of trouble call locations, enabling faster analysis of outages and immediate 6
dispatch of crews (see Figure 3). The application also supports historical archiving and performance indices reporting. Figure 3. Responder For more information, see Smart Grid Solution Suite: Responder. Benefits of SQL Server for ArcFM Solution SQL Server 2008 R2 Enterprise provides a reliable, cost-effective, low-maintenance database framework for ArcFM Solution. SQL Server is a comprehensive, integrated, and enterprise-ready data management software that can support the largest and most process-intensive deployments. Using SQL Server, Telvent customers can take advantage of reduced licensing, hardware, administration, and support fees, in addition to enterprise-class performance and scalability. Hundreds of organizations are currently running 10 terabyte (TB) and larger transactional databases on SQL Server. 1 1 http://www.microsoft.com/presspass/itanalyst/docs/06-30-09enterprisedatabasemanagementsystems.pdf 7
Running ArcFM Solution on SQL Server provides customers with many benefits. These benefits include: Scalability and performance. SQL Server includes many new features that help Telvent customers scale up and scale out as their businesses grow. 2 Lower hardware costs. SQL Server runs on standard commodity server hardware, providing dramatically lower total cost of ownership (TCO) for utilities compared to mainframe computers. Lower software costs. With SQL Server, customers can typically enjoy a 3:1 reduction over the largest competitor in licensing costs. 3 Simpler systems management and lower staffing costs. 4 SQL Server database administrators (DBAs) can typically manage three times as many physical databases as a competitor s DBAs. 5 Lower maintenance costs. 6 SQL Server first-year maintenance costs are up to 77 percent less than those of the largest competitor solutions. Fewer security vulnerabilities. SQL Server has consistently had fewer security vulnerabilities than the largest competitor solutions. 7 With SQL Server, Telvent customers can save with reduced licensing, hardware, administration, and support fees, which translate into substantially lower costs over the life of the system. For more information, see the SQL Server home page. 2 http://www.microsoft.com/sqlserver/2008/en/us/performance-scale.aspx 3 http://www.microsoft.com/sqlserver/2008/en/us/value-calc.aspx 4 http://www.microsoft.com/sqlserver/2008/en/us/compare-oracle.aspx 5 http://www.alinean.com/pdfs/microsoft_sql_server_and_oracle-alinean_tca_study_2010.pdf 6 http://www.microsoft.com/sqlserver/2008/en/us/compare-oracle-calc.aspx 7 NIST National Vulnerability Database, http://nvd.nist.gov/ 8
Benchmark Testing Confirms Performance and Scalability of ArcFM Solution on SQL Server As part of their ongoing commitment to provide utilities with truly scalable GIS systems, Telvent and Microsoft conducted scalability and performance testing of the ArcFM and Responder functionalities of ArcFM Solution running on SQL Server 2008 R2 Enterprise and Windows Sever 2008 R2 Enterprise. The benchmark testing confirmed that Telvent customers can choose SQL Server as a database platform for ArcFM Solution with confidence for several reasons: The lower cost of using SQL Server does not translate to a compromise in performance. Some of the largest and smallest utilities in the world use ArcFM Solution. The benchmark testing results demonstrate that SQL Server can scale easily across the spectrum of usage patterns without any significant performance degradation. Geometric network operations are not affected by system load or server capacity. In the benchmark testing, a geodatabase powered by ArcFM Solution and managed by SQL Server showed no significant changes in performance as loads were increased from very small to very large. Performance remained consistent from a single user to 400 concurrent users. ArcFM functionality scales well as system usage increases over time. The benchmark testing looked at the effects of a rapidly growing dataset when multiple users added features to the geometric model over time. It confirmed that with SQL Server as a database, users will not experience a significant decrease in performance in creating, editing, and saving ArcFM sessions as their dataset grows. Benchmark Testing Goals The goal of the benchmark testing was to confirm that ArcFM and Responder running on the Microsoft platform can meet the requirements of the largest Telvent utility customers. To test the ArcFM functionality, Microsoft and Telvent simulated the most common GIS activities typically conducted at Telvent customer sites. Table 1 lists the goals for ArcFM. 9
Table 1. Goals for ArcFM benchmark testing Panning and zooming within a map Tracing the geometric network Verify that performance for these processes does not degrade as the database size and load increase Creating and completing edit sessions To test Responder, Microsoft and Telvent used the custom load simulation tool TroubleMaker to simulate trouble or outage calls. Table 2 lists the goals for the Responder benchmark testing. Table 2 Goals for Responder benchmark testing Prediction performance testing (call processing) User experience testing Process at least 50,000 trouble calls/trouble events per hour Verify that as the database size and load increase, performance does not degrade and remains comparable to that of SQL Server competitors Note that Telvent and Microsoft also set the goal of identifying and taking advantage of any SQL Server 2008 R2 Enterprise performance improvement opportunities. Benchmarking Test Scenarios To ensure that the testing was as realistic as possible, Telvent and Microsoft used test scenarios that simulated a very large utility with approximately 10 million customers. Test Database One of the goals of the benchmark testing was to verify that performance remained constant as the database size and load increased. Two full database configurations were therefore used in the testing: 32 cores and 512 GB of RAM 8 cores and 24 GB of RAM 10
The geodatabase was designed to emulate the characteristics of a large city: 10 million customers 3.0 million service points 0.9 million transformers 1.9 million primary conductors, overhead (OH) 1.3 million primary conductors, underground (UG) 2.7 million secondary conductors, OH 2.6 million secondary conductors, UG ArcFM Testing Process To test ArcFM, Telvent used Citrix EdgeSight to simulate common client activities running concurrently for 30 minutes: Panning and zooming within the map (400 simultaneous users for 30 minutes) Tracing (400 simultaneous users for 30 minutes) Editing (200 simultaneous users for 30 minutes) Note: The number of simultaneous users used in the benchmark testing is twice the known number of concurrent users at any Telvent customer site. Panning and Zooming The Pan and Zoom test simulated 400 simultaneous users panning and zooming within a map and using the following procedure: 1. Set minimum and maximum XY coordinates to establish a working area in the map. 2. Select a random XY coordinate within the designated area and pan to it. 3. Select a scale between 1:1,000 and 1:100,000. Tracing The Trace test simulated 400 simultaneous users performing several tracing operations on Dynamic Protective Device (DPD) features using the following procedure: 1. Open the Locator tool and perform an attribute search for all DPD features. 2. Enter a number in the ObjectID field to locate a random DPD feature. 3. Zoom to the located DPD feature and center it in the map. 4. Select the Downstream Trace tool and click on the DPD feature to initiate a trace. 11
5. Clear the results and perform a Downstream Protective Device trace on the same feature. 6. Clear the results and return to the Locator tool to search for another random DPD feature. Editing The Edit test simulated 200 simultaneous users completing an editing session using the following procedure: 1. Open Process Framework and create a new session. Name it based on the current time. 2. Start the edit session in ArcMap, and zoom to a specific location on the map. 3. Sketch a number of poles (approximately 50). 4. Sketch a primary overhead conductor along the placed poles. 5. Save and close the session. Responder Testing Process To test Responder, Microsoft and Telvent used Citrix EdgeSight and TroubleMaker to simulate 70,000 trouble or outage calls per hour. These calls were submitted and processed through the Responder Prediction Engine for 15 consecutive hours. This load is more than double the load Telvent has observed at their largest existing customer sites. Two types of Responder tests were conducted: Prediction performance User experience Prediction Performance Testing The benchmark test used the TroubleMaker event simulation agent to submit calls at various rates (between 25,000 calls/hour and 100,000 calls/hour). Telvent reviewed the accuracy of the outage-prediction metrics (based on calls). Telvent also confirmed that the number of outage groups created from the predictions (referred to as incidents in Responder) was the expected number given the prediction thresholds for the areas. User Experience Testing The benchmark test calculated the time to perform various client activities such as cancel, create, confirm, and restore an incident/outage. Activities were performed with various system loads, and calls were submitted concurrently. 12
Benchmark Testing Setup The benchmarking test was conducted in the Microsoft Enterprise Engineering Center (EEC) labs in Redmond, Washington, using standard, commodity servers and storage to confirm a realistic cost of ownership for the solution. Database and Application Servers For the ArcFM testing, the environment consisted of one database server and 12 application servers in a Citrix farm (using Citrix XenApp 5.0). For the Responder testing, the environment consisted of three business logic servers and one Responder message-router server running the Responder business logic. Testing also included a Responder web server and the Citrix EdgeSight server farm to provide the simulated user load. Figure 4 shows the hardware configuration used. Figure 4. Benchmark testing environment Table 3 shows details of the hardware used in the benchmark testing. 13
Table 3. Details of hardware used Backend tier Middle tier Frontend tier Database server Responder business logic servers (3) Responder web servers (1) Responder message router server (1) License server (1) Domain controller (1) Citrix farm servers (12) Citrix EdgeSight simulation (5) 32 cores, 512 GB RAM; also reduced to 8 cores, 24 GB RAM 8 cores, 64 GB RAM 8 cores, 64 GB RAM 8 cores, 64 GB RAM 2 cores, 4 GB RAM 2 cores, 4 GB RAM 8 cores, 48 GB RAM 8 cores, 32 GB RAM The database server was connected to a XioTech Magnitude 3D 3000 storage area network (SAN) with one hundred and ten 146 GB-15K-FC drives. Initial Setup and Tuning The databases and application servers were set up according to the ArcFM and Responder configuration guides. Telvent installed Citrix XenApp 5.0 (an application management system that virtualizes, centralizes, and manages applications) along with Citrix EdgeSight, and configured both to support up to 500 concurrent users. Note that the testing included a large number of domain users to ensure real-world scenarios. Each user profile was set to roaming to ensure the environments were consistent across the Citrix server farm. Benchmark Testing Results The benchmark testing results confirm that Telvent ArcFM and Responder running on SQL Server and Windows Server are highly scalable systems that provide the performance required by large Telvent customers. 14
Summary of Results for ArcFM The benchmark testing showed that ArcFM is capable of supporting more than 400 users simultaneously panning different portions of the map without degradation of the redraw performance. Table 4 summarizes the results for the various ArcFM processes tested: Table 4. Summary of results for ArcFM Pan and zoom scaled successfully up to 400 simultaneous users (there was no degradation in system performance between a single user and 400 users). Pan and zoom In addition, even with 400 users, there was no performance degradation between the large, scaled-up database configuration (32 cores, 512 GB RAM) and the scaled-down configuration (8 cores, 24 GB RAM). This confirms that smaller hardware configurations can provide high performance and scalability. The time of operation for pan and zoom in all scenarios remained at 0.5 seconds. Tracing scaled successfully up to 400 simultaneous users (there was minimal degradation in system performance between a single user and 400 users). Tracing In addition, even with 400 users, there was no performance degradation between the large, scaled-up database configuration (32 cores, 512 GB RAM) and the scaled-down configuration (8 cores, 24 GB RAM). This confirms that smaller hardware configurations can provide high performance and scalability. Time of operation for tracing functions varied by only milliseconds. Edit session The time of operation for opening and closing edit sessions increased by less than 5 seconds when scaling from one user to 200 concurrent users. The time of operation for saving and closing a session grew by less than 5 seconds when scaling from one user to 200 concurrent users. 15
Summary of Results for Responder The benchmark testing showed that the ArcFM Solution Responder functionality is able to support high call volumes, with a maximum call processing of 70,000 calls per hour over the course of 15 hours. Table 5 summarizes the results for the Responder processes tested: Table 5. Summary of results for Responder Prediction performance testing (call processing) Responder processed 70,000 calls per hour for an extended period (approximately 15 hours, no user load). User experience metrics were comparable with tests run locally on significantly smaller datasets with less of a load and a smaller geodatabase size. User experience testing User experience time measures for create, confirm, restore, complete, edit, and close an incident/outage tasks were comparable with tests run locally on significantly smaller datasets with less of a load and a smaller geodatabase size. Note that the user experience measures for cancel, consolidate, and move upstream an incident/outage tasks were slow and/or timed out. The performance issues for these particular operations under significant load and scale were identified as issues with the Responder solution code, and will be addressed in future versions. Conclusion The benchmark testing results confirm that ArcFM and Responder are highly scalable systems capable of supporting a geodatabase containing 10 million customers on SQL Server 2008 R2 Enterprise and Windows Server 2008 R2 Enterprise. The benchmark testing showed that the cost effectiveness of the Microsoft platform does not mean a compromise in performance. ArcFM Solution running on SQL Server showed no significant performance degradation as loads were increased from a single user to 400 concurrent users. For ArcFM functionality, the testing showed that performance remained consistent between a larger hardware configuration and a 16
smaller, less powerful system. For Responder, the user experience for most tasks was comparable to when run locally on significantly smaller datasets with a smaller load. Highlights of the benchmark testing include the following results: Both ArcFM and Responder were able to support the real-world processes for a typical 10 million customer utility within target time windows for all processes tested. Responder successfully processed 70,000 calls per hour through the Responder Prediction Engine for 15 consecutive hours. This demonstrates that Responder can successfully handle an unusually high call volume for an extended period of time. ArcFM functionality showed zero degradation when scaling to a significantly larger database with more load and on a less powerful server. This proves the ability of ArcFM running on SQL Server to scale while maintaining performance. Telvent customers can be confident that ArcFM Solution running on the Microsoft platform can meet their performance and scalability requirements. Additional Information The following references provide more information about Telvent and Microsoft. About Telvent Telvent (NASDAQ: TLVT) is a global IT solutions and business information services provider dedicated to helping improve efficiency, safety, and security for the world s leading companies. Telvent serves markets that are critical to the sustainability of the planet, including the energy, transportation, agricultural, and environmental sectors. For more information, visit www.telvent.com. About the Microsoft Worldwide Utilities Group The Microsoft Worldwide Utilities Group offers platform and partner solutions that empower utilities to thrive in today s market-driven environment through optimized business operations in customer care, generation, tracing and risk management, transmission and distribution, regulatory compliance, and enterprise services. For more information, contact your local Microsoft sales representative, or visit www.microsoft.com/utilities. 17