Benchmark Testing Results: Unparalleled Scalability of Itron Enterprise Edition on SQL Server

Benchmark Testing Results: Unparalleled Scalability of Itron Enterprise Edition on SQL Server Benchmark Testing Confirms That Itron Enterprise Edition TM Meter Data Management Can Support 10 Million Meters on a SQL Server 2008 R2 Enterprise Edition Technical White Paper Published: May 2011 Applies to: Microsoft SQL Server 2008 R2 Enterprise Edition Abstract In January 2011, Itron worked with Microsoft to conduct a series of high-volume data tests on Itron Enterprise Edition (IEE) Meter Data Management (MDM) version 7.0 (v7.0) service pack 3 (SP3) running on Microsoft SQL Server 2008 R2 Enterprise Edition, with the goal of replicating the genuine demands of a 10 million meter deployment on commodity server hardware. The test results overwhelmingly confirm that IEE MDM v7.0 SP3 is a highly scalable system, capable of supporting 10 million meters on SQL Server 2008 R2 Enterprise Edition under real-world conditions. IEE MDM has powerful processing capabilities and the ability to scale on SQL Server along with reduced licensing, hardware, administration, and support fees that not only meet the current needs of large utility companies, but can also scale as utilities deploy increasingly sophisticated large-scale Advanced Metering Infrastructure (AMI) solutions. Utilities can be confident that IEE MDM v7.0 SP3, deployed on SQL Server 2008 R2, provides a reliable, costeffective, low-maintenance meter data management solution that can support even their largest and most process-intensive deployments. i

2011 Microsoft Corporation. All rights reserved. This document is provided as-is. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes. i

Table of Contents Introduction... 1 Overview of IEE MDM... 2 The Benefits of SQL Server 2008 R2 Enterprise Edition... 3 Benchmark Testing Overview... 4 Testing Goals... 5 The Benchmarking Test... 5 Testing Setup... 7 Database and Application Servers... 8 Database Server Details... 8 Application Server Details... 9 Disk Configuration... 9 Testing Data... 9 Initial Setup and Tuning... 10 Target Times... 11 The 10 Million Meter Benchmark Testing Results... 12 Summary... 13 Additional Information... 15 About Itron... 15 About the Microsoft Worldwide Utilities Group... 15 About ComTrade... 15 ii

Introduction Energy data drives the business of utilities, and with the emergence of smart metering and the smart grid, deploying a robust meter data management solution has become essential to a utility s future success. The new smart systems gather tremendous amounts of data as millions of meters report unique information in hourly and sub-hourly time intervals. The need to categorize automated meter data by various functions, such as billing and distribution system management, further adds to the challenge of dealing with so much data. Clearly, utilities need MDM systems that can meet these challenges head on. Itron Enterprise Edition MDM, the industry s leading multi-vendor platform solution, is built to maximize the value of utility metering through centralized collection, processing, storage, and complex analysis of meter and other utility-related data. Itron has 55 IEE MDM customers, with 40 million meters worldwide. More than 30 IEE MDM customers are currently in production. Itron has collaborated with Microsoft for more than 13 years to deliver AMI solutions that are easy to access, deploy, and use. As a premier Global Independent Software Vendor for utilities and a Microsoft Gold Certified Partner, Itron works with Microsoft on product roadmap alignment and joint marketing, sales, and product development. Working with Microsoft provides Itron with the resources, experience, and knowledge of a software industry leader to ensure that its solutions deliver business results. The Microsoft and Itron partnership provides customers with the only enterprise-class MDM system that supports SQL Server, a scalable and cost-effective combination that is optimized to address the increasing requirements of tomorrow s smart grid. As part of their partnership, Itron and Microsoft worked together in January 2011 to execute a high-volume benchmarking study in the Microsoft Enterprise Engineering Center (EEC) labs in Redmond, Washington. The goal of the testing was to demonstrate the scalability, performance, and cost efficiency of using IEE MDM v7.0 SP3 with SQL Server 2008 R2 Enterprise Edition on the Windows Server 2008 R2 Enterprise operating system by replicating the demands of a 10 million meter deployment. A day-in-the-life scenario was designed to emulate daily data processing in a typical large utility. To support time-based pricing, the scenario involved processing a mix of hourly, 15-minute interval, and register data from data import, validation, and estimation, to billing determinant calculation and export. The benchmarking tests used standard commodity hardware (servers and storage) to confirm a realistic cost of ownership. It was also assumed that 1% of incoming data would be missing, as is typical in large-scale production. The results of the benchmarking tests were impressive, clearly confirming that IEE MDM is capable of supporting 10 million meters on a SQL Server database. The tests validated the capacity of IEE MDM running on SQL Server to process up to 500 million meter readings in less than three hours, and confirmed that the solution can deliver time-of-use (TOU) billing determinants for a 10 million meter utility. 1

Following are some highlights of the results: The system successfully supported the day-in-the-life scenario for a 10 million meter utility within the target time windows for all of the processes that were tested. The test measured an average import speed of 37,500 meter reads/second, with peak speeds of up to 47,500 reads/second after database tuning. This means that the system validated and imported data for a 10 million meter utility in under four hours, and was able to process 500 million meter readings in less than three hours. The billing determinant calculations averaged 70 bills/second. This means that the process was completed in less than two hours for a 600,000 customer monthly billing cycle, exceeding the daily requirements of a 10 million meter utility. The combined critical extract, transform, and load (ETL) processes that summarize interval and register data into daily usage values for reporting and analysis (such as demand response and presentment) were completed in less than three hours. These results clearly show that IEE MDM v7.0 SP3 is a highly scalable system, capable of supporting 10 million meters on SQL Server 2008 R2 Enterprise Edition. Overview of IEE MDM As a proven out-of-the-box solution, IEE MDM is a missioncritical application that delivers high-volume meter data to enterprise applications. IEE MDM is a scalable, enterprise-wide, open-architecture data-management solution that manages commercial and industrial (C&I) and residential customers interval, register, and event meter data. The architecture of IEE MDM is flexible and extensible, with a central repository that makes it possible for utilities to interface large amounts of data with multiple systems. With a solution on SQL Server, our customers will save on reduced licensing, hardware, administration, and support fees, which translate into substantially lower costs over the life of the system. Julie Hance Vice President of Software Solutions Itron, North America 2

Itron offers commercial, off-the-shelf MDM, in addition to complementary applications to support demand response management, multi-channel customer care, and revenue protection. Figure 1 shows IEE MDM running on a SQL Server database. Note that the IEE MDM architecture can be easily adapted to serve different smart grid architectures. Figure 1 IEE MDM with SQL Server The Benefits of SQL Server 2008 R2 Enterprise Edition SQL Server 2008 R2 Enterprise Edition is comprehensive, integrated, and enterprise-ready data management software for data management and analysis. It provides a reliable, cost-effective, lowmaintenance database framework for IEE MDM that can support the largest and most process-intensive deployments. Hundreds of enterprises are currently running 10 terabyte (TB) and larger transactional databases on SQL Server. 1 Running IEE MDM on SQL Server provides customers with many benefits. These include: Scalability and performance benefits. SQL Server includes many new features that help IEE MDM customers scale up and scale out as their businesses grow. 2 Together, Microsoft and Itron have focused on providing solutions that help utilities execute on their vision of the smart grid and IEE MDM based on SQL Server is a great example of a solution that provides the performance, administration, integration, security and availability that utilities need. Jon C. Arnold Managing Director Power & Utilities, Microsoft 1 http://www.microsoft.com/presspass/itanalyst/docs/06-30-09enterprisedatabasemanagementsystems.pdf 2 http://www.microsoft.com/sqlserver/2008/en/us/performance-scale.aspx 3

Lower hardware costs. SQL Server runs on standard commodity server hardware, dramatically lowering the total cost of ownership (TCO) for utilities. Lower software costs. With SQL Server, customers can typically enjoy a 3:1 reduction over the largest competitor in licensing costs. 3 Simpler systems management and lower staffing costs. 4 SQL Server database administrators (DBAs) can typically manage three times as many physical databases as a competitor s DBAs. 5 Lower maintenance costs. 6 SQL Server first-year maintenance costs are up to 77% less than those of the largest competitor. Fewer security vulnerabilities. SQL Server has consistently had fewer security vulnerabilities than the largest competitor. 7 With SQL Server, IEE MDM customers can save with reduced licensing, hardware, administration, and support fees, which translate into substantially lower costs over the life of the system. Benchmark Testing Overview As part of its ongoing commitment to provide customers with a truly scalable MDM system, Itron conducts regular scale and performance testing of IEE MDM functionality. The in-house Itron performance and scalability testing uses two 5 million meter systems running on Oracle 10g Release 2: One system used for testing individual components. A second long-running system that runs a day-in-the-life scenario that is designed to emulate the business-process environment in a typical large utility. The scenario consists of more than 20 processes that are completed every day for a 5 million meter AMI deployment. It is also assumed that 1% of incoming data is missing; this puts additional strain on the system, but is typical for normal utility operation. Itron has observed that testing with partners like Microsoft brings significant benefits, including independent verification of test results, access to expertise, and access to hardware. Therefore, in January of 2011, Itron worked with Microsoft at their Enterprise Engineering Center lab in 3 http://www.microsoft.com/sqlserver/2008/en/us/value-calc.aspx 4 http://www.microsoft.com/sqlserver/2008/en/us/compare-oracle.aspx 5 http://www.alinean.com/pdfs/microsoft_sql_server_and_oracle-alinean_tca_study_2010.pdf 6 http://www.microsoft.com/sqlserver/2008/en/us/compare-oracle-calc.aspx 7 NIST National Vulnerability Database, http://nvd.nist.gov/ 4

Redmond, Washington, to test IEE MDM at 10 million meters on SQL Server 2008 R2 Enterprise Edition and Windows Server 2008 R2 Enterprise. Itron was assisted by ComTrade, a testing partner with extensive experience with IEE MDM. Testing Goals Itron and Microsoft wanted to confirm that the IEE MDM functionality could achieve acceptable performance at the 10 million meter scale on SQL Server 2008 R2. The goal of the benchmark testing was therefore to validate and import data for a typical 10 million meter utility within a fourhour time window, which represents the amount of time customers typically require to meet their strict daily billing process timelines. Acceptable performance was defined as: Reading import rates greater than 28,000 reads/second. A billing determinant calculation rate greater than 45 bills/second. Itron and Microsoft also set the goal of identifying and taking advantage of any SQL Server 2008 R2 Enterprise Edition performance improvement opportunities. The Benchmarking Test To ensure that the testing was as realistic as possible, Itron and Microsoft used a day-in-the-life scenario. This scenario included all critical automated processes that are performed by IEE MDM on a regular basis in an AMI system: Data validation, estimation, and import, with 1% of incoming data missing Calculation of TOU billing determinants from interval data These processes form the core of meter-to-cash operation in any AMI system. In addition, Itron included two ETL processes in the benchmark testing: One ETL process which occurred in the time between the data import and billing calculations summarized interval and register data into daily usage values. This process produces results needed by the validation, estimation, and rate calculations. A second ETL process which occurred after the billing calculations were completed calculated the metadata that was required for the next day s data estimations, for billing calculations, and for demand-response programs and customer presentment. 5

Table 1 summarizes the processes used in the benchmarking test. Table 1 Testing processes AMI Reading Import (ARI) AMI Billing Export (ABE) Daily Reading Summary (DRS) Like Day Algorithm (LDA) Import, validate, and load 24 hours of data from a full population of AMI meters into the database. Estimate and load data for meters, with 1% of population missing. Calculate and export TOU billing determinants for one full bill cycle for 6% of meter population (slightly more than the 5% that is typically used by utilities to rotate through the entire population in 20 working days). Calculate daily totals, peaks, and reference-period usage for all channels (interval and register), and store the data in warehouse tables for analytic and web use. Pre-calculate like days for demand-response events, estimations, and analytic uses. To validate data, the tests used the validation rules shown in Table 2. Table 2 Validation rules used to validate data Gap Check (GapCheck) Usage High Limit (UsageHighLimit) Usage Low Limit (UsageLowLimit) Usage Tolerance (UsageTolerancePercentDifference) Checks for missing intervals Checks if interval values exceed a high limit threshold Checks if interval values exceed a low limit threshold Checks if the sum of intervals between consecutive register readings did not match the difference between the register readings, within a small percentage difference 6

Table 3 shows which validation rules were applied to the different parts of the meter population. Table 3 Validation rules that were applied Residential meters Delivered channel of C&I meters Received channel of C&I meters GapCheck, UsageHighLimit, and UsageLowLimit GapCheck and UsageTolerancePercentDifference (with the percent difference set to 1%) GapCheck Interval data failing the validation checks shown in Table 2 were considered invalid and were estimated instead using the California historic rules. Using these rules, a missing interval is estimated by averaging intervals from three like days from the most recent three months of data, with no invalid and no estimated values allowed. Like days for this type of estimation are defined as: Weekdays, if the day to be estimated is a weekday. Saturdays or Sundays, if the day to be estimated is a Saturday or a Sunday. Holidays, Saturdays, or Sundays, if the day to be estimated is a holiday. Note that IEE MDM supports a large catalog of validation rules and estimation rules. The California historic rules were used in the benchmark testing because they are relatively popular, in addition to being among the most computationally demanding rules. Testing of these processes was first conducted on a 5 million meter environment (the standard Itron in-house benchmarking test) using five application servers. The test was then run for the target 10 million meters environment using nine application servers. The benchmarking tests measured throughput in terms of meter readings per second (reads/sec) for the import process, and service-point bills per second (bills/sec) for the export process. During the benchmark testing, Itron worked closely with the Microsoft SQL Server Customer Advisory Team (SQLCAT) to identify tuning opportunities and database code changes, all of which have been released in subsequent hotfixes for IEE MDM v7.0 SP3 or are being integrated into existing code for future release. Testing Setup The benchmarking test was conducted in the EEC labs in Redmond, Washington, on standard, commodity servers and storage to confirm a realistic cost of ownership for the solution. 7

Database and Application Servers The test environment consisted of one database server and nine application servers (note that only five application servers were used for preliminary testing at 5 million meters; otherwise, the hardware configuration was the same for both benchmark tests). Figure 2 shows the benchmark testing environment. Figure 2 Benchmark testing environment Database Server Details The database server was connected to an EMC storage area network (SAN) that contains both traditional spinning disks and solid-state drive (SSD) disks to take advantage of the advanced storage features of both Fully Automated Storage Tiering (FAST) and FAST Cache. The following are specifications of the database server: HP ProLiant BL680c G5 Windows Server 2008 R2 Enterprise with SP1 SQL Server 2008 R2 Enterprise Edition Intel Xeon E7450 2.4 GHz (24 cores) 128 GB RAM EMC CLARiiON CX4-960 SAN o RAID 10 pool of 9.6 TB o RAID 5 pool of 32.2 TB o 165 physical disks (150 for use, 10 for FAST cache, 5 spares) o 1 TB FAST cache (10-200 GB flash drives) 8

Application Server Details The nine application servers were HP BladeSystems. Load balancing was applied across the application servers by assigning a portion of the overall meter population to each server, and each application server had a gigabit circuit to the database server. The following are specifications of the nine application servers: IEE MDM v7.0 SP3 (Note: To run the software on the available 64-bit application servers, a custom build of IEE MDM version 7.0 SP3 was used; this version enabled IEE MDM to run as a 32-bit application on a 64-bit operating system. This will be publicly available in an upcoming release.) Windows Server 2008 R2 Enterprise with SP1 Intel Xeon 5150 2.66 GHz (four cores, two physical processors) 16 GB RAM Disk Configuration Disk configuration is a critical success factor in the performance and scalability of any database, particularly large databases with high transaction rates like IEE MDM. In the benchmark testing, the 36 TB of free space on the EMC SAN was partitioned into three logical unit numbers (LUNs): TempDB Reading data Master configuration data and metadata The EMC Flash Cache was used to provide greater input/output (I/O) for hotspots. In other words, no specific tables were assigned to SSDs. Instead, control of the data cached on the SSDs was left entirely to the EMC Flash Cache software, which allocated space according to most frequently used sectors. Testing Data The system was configured to resemble the meter population of a large utility, with 85% residential meters and 15% commercial meters. Residential, or mass-market, meters were configured with one register and one 60 minute interval channel. Commercial, or C&I, meters were configured with two registers and two 15 minute interval channels. 9

To allow for meaningful validation, estimation, and billing determinant calculations, data was generated using a library of 100 real customer usage profiles. Each meter was assigned a reference profile and a globally unique scaling factor, ensuring that every meter had a unique profile. Test databases were populated with three months of historic data prior to testing. This approach was taken to address concerns that performance of the system might degrade over time as more historical data is stored in the database. Table 4 provides the details of the testing data used. Table 4 Details of testing data 5 million meters 10 million meters Total channels 11,500,000 23,000,000 Total readings per day 251,750,000 503,500,000 Total historic readings 22,657,500,000 15,105,000,000 Initial Setup and Tuning The databases and application servers were set up in accordance with the IEE MDM configuration guide. Initial setup and tuning was performed at 5 million meters with five application servers. Meter configuration data was loaded using the IEE MDM program-based configuration interface. Smoke tests of all four key processes (ARI, ABE, DRS ETLs, and LDA ETLs: see Table 3) were conducted, during which Itron and Microsoft personnel monitored and tuned performance. Once smoke testing was completed, a full day-in-the-life series was performed, in which ARI, ABE, DRS ETL, and LDA ETL processes were run in series. After several successful runs at 5 million meters, the results shown in Table 5 were recorded for an average of five runs, with 1% of incoming data missing. 10

Table 5 Results for 5 million meters Process Total elapsed time Throughput rate Importing, validating, and estimating missing or invalid data (ARI) 1 hour, 58 minutes 35,490 reads/second Billing, for 300,000 meters (ABE) 1 hour, 36 minutes 52 bills/second Summarizing interval and register data into daily usage values (DRS ETL) 2 hours, 26 minutes 1,313 channels/second Pre-calculating like days (LDA ETL) 42 minutes 2,249 channels/second Following the initial tests at 5 million meters, several stored procedures within the IEE MDM product were tuned to improve performance. Target Times Target times were selected based on customer's requirements to validate, estimate, and import 24 hours of data, in addition to completing the DRS ETL by 08:00. Billing one-twentieth of customers was to be accomplished in three hours, as per typical utility specifications. When applied to a 10 million meter system, these requirements translate to the throughput rates shown in Table 6. Table 6 Target times and required throughput to achieve them for 10 million meters Process Target time Target throughput rate Import, validation, and estimation of missing or invalid data 5 hours 28,000 readings/second Billing, for 600,000 meters 3 hours 55 bills/second By subtracting the five hour target for import, validation, and estimation from the eight hour total time between midnight and 08:00, testers deduced a target of three hours for the subsequent DRS ETL that also needs to complete. The LDA ETL process is not required prior to billing. 11

The 10 Million Meter Benchmark Testing Results To perform 10 million meter tests, an additional 5 million meters were added to the initial database set up for the 5 million meter run, along with additional historic data (see Table 2). The benchmarking test was then run on the nine application servers. The results were impressive: the goals for the benchmark testing were easily exceeded, confirming that IEE MDM is a highly scalable system capable of supporting 10 million meters on a SQL Server 2008 R2 Enterprise Edition database. The results shown in Table 7 represent the average of five tests that were performed for the complete day-in-the-life scenario, with 1% of data missing. Table 7 Average results for 10 million meters Process Total elapsed time Throughput rate ARI ABE DRS ETL LDA ETL 3 hours, 43 minutes (target: 5 hours) 1 hour, 29 minutes (target: 3 hours) 1 hour, 58 minutes (target: 3 hours) 60 minutes (target: 2 hours) 37,500 reads/second (target: 28,000 reads/second) 70 bills/second (target: 55 bills/second) 3,248 channels/second 3,194 channels/second Table 8 summarizes the peak export and import results. Table 8 Peak results summary for 10 million meters Process Goal Peak results Percent to goal reached Import process 28,000 reads/second 47,500 reads/second 170% Export process 55 bills/second 112 bills/second 204% These results validate the capacity of IEE MDM v7.0 SP3 running on SQL Server 2008 R2 Enterprise Edition to import, validate, and estimate more than 500 million meter readings in well less than six hours; and 500 million reads/day also equates to a 20 million meter utility with all-hourly reads. In some cases, the system took less time to process more data in the exact same processes on identical hardware. Some of this difference can be explained by greater efficiency in larger batches, while other improvements can be explained by the tuning that was made possible by the assistance 12

of the Microsoft SQL Server team. The white paper that describes best practices for running IEE MDM on SQL Server 2008 R2 Enterprise Edition (to be published June 2011) discusses this tuning. The benchmark testing results also confirm the horizontal scalability of IEE MDM v7.0 SP3 on SQL Server 2008 R2. The results in the 10 million meter tests were, in all cases, better than the results for the 5 million meter tests, probably because of the addition of the four application servers. Because of this horizontal scalability at the application layer, and the vertical scalability at the database layer, Itron is confident that IEE MDM is capable of supporting even more than 10 million meters on a SQL Server 2008 R2 Enterprise Edition database. Summary The benchmark testing results overwhelmingly confirm that IEE MDM is a highly scalable system capable of supporting more than 10 million meters on a SQL Server 2008 R2 Enterprise Edition database on commodity hardware. Highlights of the benchmark testing include the following results: The system was able to support the day-in-the-life processes for a typical 10 million meter utility within target time windows for all processes tested. The test measured an average import speed of 37,500 meter reads/second, with peak speeds of up to 47,500 reads/second after database tuning. The system validated and imported data for a 10 million meter utility in less than four hours. At peak speed, the system was able to process 500 million meter readings in less than three hours. The billing determinant calculations averaged 70 bills/second, resulting in the completion of the process in less than two hours for a 600,000 customer monthly billing cycle. This exceeds the requirements of a 10 million meter utility. Critical ETL processes that summarize interval and register data into daily usage values for reporting and analysis were completed in less than three hours. The data processing rates achieved by the system 500 million reads/day equate to 20 million meter utility with all-hourly reads. 13

The results show that IEE MDM, running on SQL Server 2008 R2, is capable of processing the daily load of a typical 10 million meter utility more than 500 million meter readings per day in less than three hours (after optimization), and is able to deliver TOU billing determinants in two hours for a typical daily billing run. The results also confirm the horizontal scalability of the solution. When four application servers were added for the 10 million meter tests, the results were better than those for the initial 5 million meter tests. With this horizontal scalability of the application layer and the vertical scalability of the database layer in addition to the impressive benchmark testing results Itron is confident that the solution can scale even beyond 10 million meters. Utilities can be confident that IEE MDM, deployed on SQL Server 2008 R2, provides a scalable, costeffective MDM solution that can meet their current requirements while making it possible for them to address the increasing requirements of tomorrow s smart grid. 14

Additional Information The following references provide more information about Itron, Microsoft, and ComTrade. About Itron Itron Inc. is a Microsoft certified partner and a leading technology provider to the global energy and water industries. It is the world s leading provider of intelligent metering, data collection, and utility software solutions, with nearly 8,000 utilities worldwide relying on Itron technology to optimize the delivery and use of energy and water. Itron products include electricity, gas, water, and heat meters; data collection and communication systems, including automated meter reading (AMR) and AMI; MDM and related software applications; and project management, installation, and consulting services. For more information, visit www.itron.com. About the Microsoft Worldwide Utilities Group The Microsoft Worldwide Utilities Group offers platform and partner solutions that empower utilities to thrive in today s market-driven environment through optimized business operations in customer care, generation, trading and risk management, transmission and distribution, regulatory compliance, and enterprise services. For more information, contact your local Microsoft Sales Representative, or visit www.microsoft.com/utilities. About ComTrade ComTrade is an international provider of IT solutions and software engineering services whose clients include high-tech vendors, telecommunication companies, financial institutions, and the public sector. With more than 1,000 professionals, ComTrade offers industry-leading expertise in smart meter management, data storage management, embedded systems, network systems management, gaming, telecommunications, e-solutions, and customized application development. For more information, visit www.comtrade.com. 15