SPC BENCHMARK 1 FULL DISCLOSURE REPORT IBM CORPORATION IBM SYSTEM STORAGE DS8700 RELEASE 5.1 (EASY TIER AND SSDS) SPC-1 V1.12

Transcription

1 SPC BENCHMARK 1 FULL DISCLOSURE REPORT IBM CORPORATION IBM SYSTEM STORAGE DS8700 RELEASE 5.1 (EASY TIER AND SSDS) SPC-1 V1.12 Submitted for Review: April 13, 2010 Submission Identifier: A00092

2 ii First Edition April 2010 THE INFORMATION CONTAINED IN THIS DOCUMENT IS DISTRIBUTED ON AN AS IS BASIS WITHOUT ANY WARRANTY EITHER EXPRESS OR IMPLIED. The use of this information or the implementation of any of these techniques is the customer s responsibility and depends on the customer s ability to evaluate and integrate them into the customer s operational environment. While each item has been reviewed by IBM Corporation for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environment do so at their own risk. This publication was produced in the United States. IBM Corporation may not offer the products, services, or features discussed in this document in other countries, and the information is subject to change with notice. Consult your local IBM Corporation representative for information on products and services available in your area. Copyright IBM Corporation All rights reserved. Permission is hereby granted to reproduce this document in whole or in part, provided the copyright notice as printed above is set forth in full text on the title page of each item reproduced. Trademarks SPC Benchmark-1, SPC-1, SPC-1 IOPS, and SPC-1 LRT are trademarks of the Storage Performance Council. IBM, the IBM logo, Easy Tier, and System Storage are trademarks of IBM Corporation in the United States and other countries. All other brands, trademarks, and product names are the property of their respective owners.

3 iii Table of Contents Audit Certification... vii Audit Certification (cont.)...viii Letter of Good Faith...ix Executive Summary Test Sponsor and Contact Information...10 Revision Information and Key Dates...10 Tested Storage Product (TSP) Description...10 Summary of Results...11 Storage Capacities, Relationships, and Utilization...11 Response Time Throughput Curve...13 Response Time Throughput Data...13 Priced Storage Configuration Pricing...14 Differences between the Tested Storage Configuration (TSC) and Priced Storage Configuration...15 Benchmark Configuration (BC)/Tested Storage Configuration (TSC)/ Priced Storage Configuration Diagram...15 Benchmark Configuration (BC)/Tested Storage Configuration (TSC)/ Priced Storage Configuration Components...16 Configuration Information Benchmark Configuration (BC)/Tested Storage Configuration (TSC) Diagram.17 Storage Network Configuration...17 Host System and Tested Storage Configuration (TSC) Table of Components...17 Customer Tunable Parameters and Options...18 Tested Storage Configuration (TSC) Description...18 SPC-1 Workload Generator Storage Configuration...18 SPC-1 Data Repository Storage Capacities and Relationships...19 SPC-1 Storage Capacities...19 SPC-1 Storage Hierarchy Ratios...19 SPC-1 Storage Capacities and Relationships Illustration...20 Logical Volume Capacity and ASU Mapping...20 Storage Capacity Utilization...21 SPC-1 Benchmark Execution Results SPC-1 Tests, Test Phases, and Test Runs...22 Primary Metrics Test Sustainability Test Phase...23 SPC-1 Workload Generator Input Parameters...23

4 iv Sustainability Test Results File...23 Sustainability Data Rate Distribution Data (MB/second)...24 Sustainability Data Rate Distribution Graph...24 Sustainability I/O Request Throughput Distribution Data...25 Sustainability I/O Request Throughput Distribution Graph...25 Sustainability Average Response Time (ms) Distribution Data...26 Sustainability Average Response Time (ms) Distribution Graph...26 Sustainability Response Time Frequency Distribution Data...27 Sustainability Response Time Frequency Distribution Graph...27 Sustainability Measured Intensity Multiplier and Coefficient of Variation...28 Primary Metrics Test IOPS Test Phase...29 SPC-1 Workload Generator Input Parameters...29 IOPS Test Results File...29 IOPS Test Run I/O Request Throughput Distribution Data...30 IOPS Test Run I/O Request Throughput Distribution Graph...30 IOPS Test Run Average Response Time (ms) Distribution Data...31 IOPS Test Run Average Response Time (ms) Distribution Graph...31 IOPS Test Run Response Time Frequency Distribution Data...32 IOPS Test Run Response Time Frequency Distribution Graph...32 IOPS Test Run I/O Request Information...33 IOPS Test Run Measured Intensity Multiplier and Coefficient of Variation...33 Primary Metrics Test Response Time Ramp Test Phase...34 SPC-1 Workload Generator Input Parameters...34 Response Time Ramp Test Results File...34 Response Time Ramp Distribution (IOPS) Data...35 Response Time Ramp Distribution (IOPS) Graph...36 SPC-1 LRT Average Response Time (ms) Distribution Data...37 SPC-1 LRT Average Response Time (ms) Distribution Graph...37 SPC-1 LRT (10%) Measured Intensity Multiplier and Coefficient of Variation...38 Repeatability Test...39 SPC-1 Workload Generator Input Parameters...39 Repeatability Test Results File...40 Repeatability 1 LRT I/O Request Throughput Distribution Data...41 Repeatability 1 LRT I/O Request Throughput Distribution Graph...41 Repeatability 1 LRT Average Response Time (ms) Distribution Data...42 Repeatability 1 LRT Average Response Time (ms) Distribution Graph...42 Repeatability 1 IOPS I/O Request Throughput Distribution Data...43 Repeatability 1 IOPS I/O Request Throughput Distribution Graph...43 Repeatability 1 IOPS Average Response Time (ms) Distribution Data...44

5 v Repeatability 1 IOPS Average Response Time (ms) Distribution Graph...44 Repeatability 2 LRT I/O Request Throughput Distribution Data...45 Repeatability 2 LRT I/O Request Throughput Distribution Graph...45 Repeatability 2 LRT Average Response Time (ms) Distribution Data...46 Repeatability 2 LRT Average Response Time (ms) Distribution Graph...46 Repeatability 2 IOPS I/O Request Throughput Distribution Data...47 Repeatability 2 IOPS I/O Request Throughput Distribution Graph...47 Repeatability 2 IOPS Average Response Time (ms) Distribution Data...48 Repeatability 2 IOPS Average Response Time (ms) Distribution Graph...48 Repeatability 1 (LRT) Measured Intensity Multiplier and Coefficient of Variation...49 Repeatability 1 (IOPS) Measured Intensity Multiplier and Coefficient of Variation...49 Repeatability 2 (LRT) Measured Intensity Multiplier and Coefficient of Variation...49 Repeatability 2 (IOPS) Measured Intensity Multiplier and Coefficient of Variation...50 Data Persistence Test...51 SPC-1 Workload Generator Input Parameters...51 Data Persistence Test Results File...51 Data Persistence Test Results...52 Priced Storage Configuration Availability Date Pricing Information Tested Storage Configuration (TSC) and Priced Storage Configuration Differences Anomalies or Irregularities Appendix A: SPC-1 Glossary Decimal (powers of ten) Measurement Units...54 Binary (powers of two) Measurement Units...54 SPC-1 Data Repository Definitions...54 SPC-1 Data Protection Levels...55 SPC-1 Test Execution Definitions...55 I/O Completion Types...57 SPC-1 Test Run Components...57 Appendix B: Customer Tunable Parameters and Options AIX Parameters...58 Easy Tier Options...58 Appendix C: Tested Storage Configuration (TSC) Creation Easy Tier Configuration...59 DS8700 Configuration Steps...59 Create extent pools...59

6 vi Create the RAID-10 and RAID-5 ranks...59 Create the LUNs...60 Define The LUN access path...60 Deploy SSDs...61 AIX Configuration Steps...62 Appendix D: SPC-1 Workload Generator Storage Commands and Parameters Primary Metrics Test...63 Sustainability...63 Ramp 100 Test Run...64 Ramp 95 Test Run...66 Ramp 90 Test Run...67 Ramp 80 Test Run...68 Ramp 50 Test Run...70 Ramp 10 Test Run...71 Repeatability Test...73 Repeat1 LRT Test Run Repeat1 IOPS Test Run...73 Repeat2 LRT Test Run Repeat2 IOPS Test Run...74 Persistence Test...76 Persistence Test Run Persistence Test Run Appendix E: SPC-1 Workload Generator Input Parameters Slave JVM Invocation...80 runslaves.sh...80 slave1.txt...80 Primary Metrics Test, Repeatability Test, and Persistence Test Run runthem.sh...82 Persistence Test Run runit.sh...82 runpersist1.sh...82 Persistence Test Run runpersist2.sh...83

7 vii AUDIT CERTIFICATION

8 viii AUDIT CERTIFICATION (CONT.)

9 ix LETTER OF GOOD FAITH

10 EXECUTIVE SUMMARY Page 10 of 83 EXECUTIVE SUMMARY Test Sponsor and Contact Information Test Sponsor Primary Contact Test Sponsor Alternate Contact Auditor Test Sponsor and Contact Information IBM Corporation Bruce McNutt 650 Harry Road C2 500 San Jose, CA Phone: (408) FAX: IBM Corporation Joe Hyde 9000 S. Rita Road Tucson, AZ Phone: (520) FAX: (520) Storage Performance Council Walter E. Baker 643 Bair Island Road, Suite 103 Redwood City, CA Phone: (650) FAX: (650) Revision Information and Key Dates Revision Information and Key Dates SPC-1 Specification revision number V1.12 SPC-1 Workload Generator revision number V2.1.0 Date Results were first used publicly April 13, 2010 Date the FDR was submitted to the SPC April 13, 2010 Date the priced storage configuration is available for shipment to customers May 21, 2010 Date the TSC completed audit certification April 12, 2010 Tested Storage Product (TSP) Description The IBM System Storage DS8000 series encompasses the flagship disk enterprise storage products in the IBM System Storage portfolio. The DS8700 represents the latest in this series of enterprise disk storage systems designed for high-performance, high-capacity and resiliency. Major new capabilities include IBM POWER6 Processing technology and PCI-e I/O enclosures. IBM System Storage Easy Tier feature of the DS8700 enables more effective storage consolidation by taking the guesswork out of deploying solid-state drives by automatically and dynamically moving the appropriate data to the appropriate drive tier in the system, based on ongoing performance monitoring. Such effective storage tiering will help ensure systems are optimized for solid-state technology from both a performance and cost perspective.

11 EXECUTIVE SUMMARY Page 11 of 83 Summary of Results SPC-1 Results Tested Storage Configuration (TSC) Name: IBM System Storage DS8700 Release 5.1 (Easy Tier and SSDs) Metric SPC-1 IOPS 32, Reported Result SPC-1 Price-Performance Total ASU Capacity Data Protection Level $47.92/SPC-1 IOPS 34, GB Protected (Mirroring, RAID-5) Total TSC Price (including three-year maintenance) $1,581, SPC-1 IOPS represents the maximum I/O Request Throughput at the 100% load point. Total ASU (Application Storage Unit) Capacity represents the total storage capacity read and written in the course of executing the SPC-1 benchmark. A Data Protection Level of Protected using Mirroring configures two or more identical copies of user data. The SSD portion of the configuration utilized a Data Protection Level of RAID-5 which provides data protection by distributing check data corresponding to user data across multiple disks in the form of bit-by-bit parity. Storage Capacities, Relationships, and Utilization The following diagram and table document the various storage capacities, used in this benchmark, and their relationships, as well as the storage utilization values required to be reported. Application Storage Unit (ASU) Capacity 34, GB ASU 1 15, GB ASU 2 15, GB 36 Logical Volumes 36 Logical Volumes 8 Logical Volumes ( GB per Logical Volume) Addressable Storage Capacity 38, GB ASU 3 3, GB Overhead/Metadata GB Unused 4m284,856 GB Configured Storage Capacity 87, GB Data Protection (Mirroring) GB (includes GB of LVM Overhead and 4, GB of unused capacity) Required SSD Capacity 2, GB Spares 7, GB Unused 1, GB Global Storage Overhead 11, GB Physical Storage Capacity 100, GB

12 EXECUTIVE SUMMARY Page 12 of 83 SPC-1 Storage Capacity Utilization Application Utilization 34.11% Protected Application Utilization 68.24% Unused Storage Ratio 9.78% Application Utilization: Total ASU Capacity (34, GB) divided by Physical Storage Capacity (100, GB) Protected Application Utilization: (Total ASU Capacity (34, GB) plus total Data Protection Capacity (38, GB) minus unused Data Protection Capacity (4, GB) divided by Physical Storage Capacity (100, GB) Unused Storage Ratio: Total Unused Capacity (9, GB) divided by Physical Storage Capacity (100, GB) and may not exceed 45%. Detailed information for the various storage capacities and utilizations is available on pages in the Full Disclosure Report.

13 EXECUTIVE SUMMARY Page 13 of 83 Response Time Throughput Curve The Response Time-Throughput Curve illustrates the Average Response Time (milliseconds) and I/O Request Throughput at 100%, 95%, 90%, 80%, 50%, and 10% of the workload level used to generate the SPC-1 IOPS metric. The Average Response Time measured at any of the above load points cannot exceed 30 milliseconds or the benchmark measurement is invalid. Ramp Phase Response Time / Throughput Curve Response Time (ms) % 95% 100% 80% 50% 10% IO Requests per Second Response Time Throughput Data 10% Load 50% Load 80% Load 90% Load 95% Load 100% Load I/O Request Throughput 3, , , , , , Average Response Time (ms): All ASUs ASU ASU ASU Reads Writes

14 EXECUTIVE SUMMARY Page 14 of 83 Priced Storage Configuration Pricing Product Description Qty Price Extended Price Discount (%) Disc. Price System Storage DS $ 72, $ 72, % $ 36, Eligible for EU Shipment 1 N/C 700 OEL Indicator 1 N/C to TB capacity 1 N/C 900 Non-Standby CoD 1 N/C 932 IBM System p Indicator 1 N/C 1050 Battery Assembly 3 1, $ 5, % $ 2, Line Cord (US/LA/AP/Canada) 1 1, $ 1, % $ Management Console - English Laptop Internal 1 9, $ 9, % $ 4, Disk Enclosure Pair 4 10, $ 40, % $ 20, Disk Drive Cable Group 1 1 1, $ 1, % $ I/O Enclosure Pair PCIE 2 11, $ 23, % $ 11, PCI-E Cable Group 2 1 4, $ 4, % $ 2, um Fibre Cable (LC) $ 1, % $ Release 5 Bundle Family 1 40, $ 40, % $ 20, TB 7.2K SATA Drive Set (16 drives per set) 6 89, $ 538, % $ 269, Disk Enclosure Filler $ % $ Device Adapter Pair III 2 10, $ 20, % $ 10, Gb SW FCP/FICON Adapter PCIE 6 33, $ 203, % $ 101, GB Processor Memory (4-Way) 1 784, $ 784, % $ 392, Way Processor Card Pair 1 80, $ 80, % $ 40, GB SSD Drive Set (16 SSDs per set) 1 923, $ 923, % $ 461, OEL TB 1 N/C 7051 OEL - 1 Value Unit 1 N/C 7054 OEL - 25 Value Unit 1 N/C 7055 OEL - 50 Value Unit 1 N/C 8S1027 SSD System Order 1 N/C 2398-LFA DS8000 Function Authorization 1 N/C 932 IBM System p Indicator 1 N/C 7040 OEL TB 1 N/C 7053 OEL - 10 Value Unit 1 $ 53, $ 53, % $ 32, OEL - 25 Value Unit 1 $ 85, $ 85, % $ 51, OEL - 50 Value Unit 1 $ 140, $ 140, % $ 84, MC4 System Storage Productivity Center 1 $ 4, $ 4, % $ 2, IBM Tivoli Storage Productivity Center via 1 N/C AAS/eConfig 1170 Power Cord, Standard Rack 2 N/C 1810 Dual Power Supply Option 1 $ $ % $ Console Keyboard/Display/Drawer 1 $ 1, $ 1, % $ MMB Gb PCI-e Dual Port FC HBA 11 $ 4, $ 50, % $ 35, Total discounted price $ 1,581, The above pricing includes hardware maintenance and software support for three years, 7 days per week, 24 hours per day. The hardware maintenance and software support provides the following: Acknowledgement of new and existing problems with four (4) hours. Onsite present of a qualified maintenance engineer or provision of a customer replaceable part within four (4) hours of the above acknowledgement for any hardware failure that results in an inoperative Price Storage Configuration that can be remedied by the repair or replacement of a Priced Storage Configuration component.

15 EXECUTIVE SUMMARY Page 15 of 83 Differences between the Tested Storage Configuration (TSC) and Priced Storage Configuration There were no differences between the TSC and Priced Storage Configuration. Benchmark Configuration (BC)/Tested Storage Configuration (TSC)/ Priced Storage Configuration Diagram IBM Power 770 Server 12 FC links on 11 dual-port HBAs 12 FC links 12 FC links on 6 Host Adapters 4-way SMP cluster GHz / 128 GB cache 4-way SMP cluster GHz / 128 GB cache Device Adapter Device Adapter Device Adapter Device Adapter (FC/AL) Rank 0 Rank 1 Rank 2 Rank 3 Rank 4 Rank 5 Rank 6 (FC/AL) (FC/AL) Rank 7 Rank 0 Rank 1 Rank 2 12 Rank FC links 3 Rank 4 Rank 5 Rank 6 (FC/AL) Rank 7 Rank Rank 8 8 disks disks 0 0 Rank Rank 8 8 disks disks 1 1 Rank Rank 8 8 disks disks 2 2 Rank Rank 8 8 disks disks 3 3 Rank Rank 8 8 disks disks 4 4 Rank Rank 8 8 disks disks 5 5 Rank Rank 8 8 disks disks 6 6 Rank Rank 8 8 disks disks 7 7 Rank 28 spare disks 0 Rank 08 spare disks 1 Rank 08 spare disks 2 Rank 08 spare disks 3 Rank 08 spare disks 4 Rank 08 spare disks 5 Rank 08 spare disks 6 Rank 28 spare disks 7 Rank Rank 8 disks 2 spare 8 disks 0 spare 8 disks 0 spare 8 disks 0 spare 8 disks 0 spare 8 disks 0 spare 8 disks 0 spare 8 disks 80 0 disks 2 spare Rank Rank 81 1 disks 0 spare Rank Rank 82 2 disks 0 spare Rank Rank 83 3 disks 0 sparerank Rank 84 4 disks 0 spare Rank Rank 85 5 disks 0 spare Rank Rank 2 spare 86 6 disks 0 spare Rank Rank 87 7 disks 2 spare Rank 8 disks 8 disks 2 spare 0 spare 0 spare 0 spare 0 spare 0 spare 0 spare 2 spare 20 spare Rank 8 disks 8 disks 02 spare 12 8 host disks 8 disks 0 adapter spare Rank 8 disks 8 disks 2 spare 0 spare 0 spare 0 ports spare 06 8 disks spare on 6 Rank 8 8 disks HA disks 8 disks 8 disks 0 spare cards 07 spare 8 disks 0 spare 0 spare Rank 8 disks 0 spare 03 spare Rank 8 disks 2 spare 21 spare 82 SATA spare 80 SATA spare 0 spare 80 SSD spare 80 SSD spare 0 spare 80 SATA spare 82 SATA spare Device Device Adapter Adapter Device Device Adapter Adapter (FC/AL) (FC/AL) Rank Rank 0 0 Rank Rank 1 1 Rank Rank 2 2 Rank 12 Rank 3 FC links 3 Rank Rank 4 4 Rank Rank 5 5 Rank Rank 6 6 (FC/AL) (FC/AL) Rank 7 Rank 8 disks 0 Rank 8 disks 1 Rank 8 disks 2 Rank 8 disks 3 Rank 8 disks 4 Rank 8 disks 5 Rank 8 disks 6 Rank Rank 8 disks 7 7 Rank Rank 28 8 spare disks disks 0 0 Rank Rank 08 8 spare disks disks 1 1 Rank Rank 08 8 spare disks disks 2 2 Rank Rank 08 8 spare disks disks 3 3 Rank Rank 08 8 spare disks disks 4 4 Rank Rank 08 8 spare disks disks 5 5 Rank Rank 08 8 spare disks disks 6 6 Rank Rank 28 8 spare disks disks 7 7 Rank Rank disks 20 disks 2 spare spare Rank Rank 81 8 disks 01 disks 0 spare spare Rank Rank 82 8 disks 02 disks 0 spare spare Rank Rank 83 8 disks 03 disks 0 spare spare Rank Rank 84 8 disks 04 disks 0 spare spare Rank Rank 85 8 disks 05 disks 0 spare spare Rank Rank 86 8 disks 06 disks 0 spare spare Rank Rank 87 8 disks 27 disks 2 spare spare Rank 8 disks 2 spare 8 disks 0 spare 8 disks 0 spare 8 disks 0 spare 8 disks 0 spare 8 disks 0 spare 8 disks 2 spare 0 spare 0 spare 8 disks 8 disks 0 Rank 8 disks 2 2 spare 12 Rank 8 host disks 04 adapter spare Rank 8 disks ports 06 spare on 6 Rank 8 HA disks cards 07 spare Rank 8 disks 05 spare Rank 8 disks 03 spare Rank 8 disks 21 spare 82 SATA spare 2 spare8 0 SATA spare 0 spare8 0 SATA spare 0 spare8 0 SATA spare 0 spare 80 SATA spare 0 spare 8 0 SATA spare 0 spare 8 0 SATA spare 0 spare 8 2 SATA spare 2 spare IBM System Storage DS8700 Version 5.1 (EasyTier and SSDs)

16 EXECUTIVE SUMMARY Page 16 of 83 Benchmark Configuration (BC)/Tested Storage Configuration (TSC)/ Priced Storage Configuration Components Host System: IBM Power 770 (P MMB) 64 cores in: GHz POWER7 processor modules 8 cores per processor module 256 KB L2 cache per core 4 MB L3 cache per core (edram) Tested Storage Configuration (TSC)/ Priced Storage Configuration: 11 8 Gb PCIe dual port FC HBAs IBM System Storage DS GB memory/cache 12 4 Gbps front-end physical connections (2 per FCP/FICON Adapter) 16 2 Gbps backend physical connections (4 per Device Adapter) 256 GB main memory 1 4 Way Processor Card Pair AIX 6.1 TL 01 PCIe 6 4 Gb SW FCP/FICON Adapter PCIe 2 Device Adapter Pair III 1 Management Console 96 1 TB 7.2K RPM SATA disk drive GB SSDs

17 CONFIGURATION INFORMATION Page 17 of 83 In each of the following sections of this document, the appropriate Full Disclosure Report requirement, from the SPC-1 benchmark specification, is stated in italics followed by the information to fulfill the stated requirement. CONFIGURATION INFORMATION Benchmark Configuration (BC)/Tested Storage Configuration (TSC) Diagram Clause A one page Benchmark Configuration (BC)/Tested Storage Configuration (TSC) diagram shall be included in the FDR The Benchmark Configuration (BC)/Tested Storage Configuration (TSC) is illustrated on page 15 (Benchmark Configuration (BC)/Tested Storage Configuration (TSC)/ Priced Storage Configuration Diagram). Storage Network Configuration Clause If the TSC contains network storage, the diagram will include the network configuration. If a single diagram is not sufficient to illustrate both the Benchmark Configuration and network configuration in sufficient detail, the Benchmark Configuration diagram will include a highlevel network illustration as shown in Figure 9-8. In that case, a separate, detailed network configuration diagram will also be included as described in Clause Clause If a storage network was configured as a part of the Tested Storage Configuration and the Benchmark Configuration diagram described in Clause contains a high-level illustration of the network configuration, the Executive Summary will contain a one page topology diagram of the storage network as illustrated in Figure 9-9. The Host System and TSC table of components may be found on page 15 (Benchmark Configuration (BC)/Tested Storage Configuration (TSC)/ Priced Storage Configuration Diagram). Host System and Tested Storage Configuration (TSC) Table of Components Clause The FDR will contain a table that lists the major components of each Host System and the Tested Storage Configuration (TSC). Table 9-10 specifies the content, format, and appearance of the table. The Host System and TSC table of components may be found on page 16 (Benchmark Configuration (BC)/Tested Storage Configuration (TSC)/ Priced Storage Configuration Components).

18 CONFIGURATION INFORMATION Page 18 of 83 Customer Tunable Parameters and Options Clause All Benchmark Configuration (BC) components with customer tunable parameter and options that have been altered from their default values must be listed in the FDR. The FDR entry for each of those components must include both the name of the component and the altered value of the parameter or option. If the parameter name is not self-explanatory to a knowledgeable practitioner, a brief description of the parameter s use must also be included in the FDR entry. Appendix B: Customer Tunable Parameters and Options on page 58 contains the customer tunable parameters and options that have been altered from their default values for this benchmark. Tested Storage Configuration (TSC) Description Clause The FDR must include sufficient information to recreate the logical representation of the TSC. In addition to customer tunable parameters and options (Clause ), that information must include, at a minimum: A diagram and/or description of the following: All physical components that comprise the TSC. Those components are also illustrated in the BC Configuration Diagram in Clause and/or the Storage Network Configuration Diagram in Clause The logical representation of the TSC, configured from the above components that will be presented to the Workload Generator. Listings of scripts used to create the logical representation of the TSC. If scripts were not used, a description of the process used with sufficient detail to recreate the logical representation of the TSC. Appendix C: Tested Storage Configuration (TSC) Creation on page 59 contains the detailed information that describes how to create and configure the logical TSC. SPC-1 Workload Generator Storage Configuration Clause The FDR must include all SPC-1 Workload Generator storage configuration commands and parameters. The SPC-1 Workload Generator storage configuration commands and parameters for this measurement appear in Appendix D: SPC-1 Workload Generator Storage Commands and Parameters on page 63.

19 DATA REPOSITORY Page 19 of 83 SPC-1 DATA REPOSITORY This portion of the Full Disclosure Report presents the detailed information that fully documents the various SPC-1 storage capacities and mappings used in the Tested Storage Configuration. SPC-1 Data Repository Definitions on page 54 contains definitions of terms specific to the SPC-1 Data Repository. Storage Capacities and Relationships Clause Two tables and an illustration documenting the storage capacities and relationships of the SPC-1 Storage Hierarchy (Clause 2.1) shall be included in the FDR. SPC-1 Storage Capacities SPC-1 Storage Capacities Storage Hierarchy Component Units Capacity Total ASU Capacity Gigabytes (GB) 34, Addressable Storage Capacity Gigabytes (GB) 38, Configured Storage Capacity Gigabytes (GB) 87, Physical Storage Capacity Gigabytes (GB) 100, Data Protection (Mirroring) Gigabytes (GB) 38, Required Storage (including spares) Gigabytes (GB) 10, Global Storage Overhead Gigabytes (GB) 11, Total Unused Storage Gigabytes (GB) 9, SPC-1 Storage Hierarchy Ratios Addressable Storage Capacity Configured Storage Capacity Physical Storage Capacity Total ASU Capacity 88.83% 39.04% 34.11% Required for Data Protection (Mirrored) 43.98% 38.43% Addressable Storage Capacity 43.95% 38.40% Required Storage (including spares) 12.11% 10.58% Configured Storage Capacity 87.37% Global Storage Overhead 11.43% Unused Storage: Addressable 11.17% Configured 0.00% Physical 1.20%

20 DATA REPOSITORY Page 20 of 83 The Physical Storage Capacity consisted of 101, GB distributed over 96 disk drives each with a physical capacity of 1, GB and 16 solid state storage devices (SSDs) each with a physical capacity of 256 GB. There was GB (0.00%) of Unused Storage within the Physical Storage Capacity. Global Storage Overhead consisted of 11, GB (11.43%) of Physical Storage Capacity. There was GB (0.00%) of Unused Storage within the Configured Storage Capacity. The Total ASU Capacity utilized 88.83% of the Addressable Storage Capacity resulting in 4, GB (11.17%) of Unused Storage within the Addressable Storage Capacity. The Data Protection (mirroring) capacity was 38, GB of which 34, GB was utilized. The total Unused Storage was 9, GB. SPC-1 Storage Capacities and Relationships Illustration The various storage capacities configured in the benchmark result are illustrated below (not to scale). Application Storage Unit (ASU) Capacity 34, GB ASU 1 15, GB ASU 2 15, GB Addressable Storage Capacity 38, GB ASU 3 3, GB 36 Logical Volumes 36 Logical Volumes 8 Logical Volumes ( GB per Logical Volume) Overhead/Metadata GB Unused 4m284,856 GB Configured Storage Capacity 87, GB Data Protection (Mirroring) GB (includes GB of LVM Overhead and 4, GB of unused capacity) Required SSD Capacity 2, GB Spares 7, GB Unused 1, GB Global Storage Overhead 11, GB Physical Storage Capacity 100, GB Logical Volume Capacity and ASU Mapping Clause A table illustrating the capacity of each ASU and the mapping of Logical Volumes to ASUs shall be provided in the FDR. Logical Volumes shall be sequenced in the table from top to bottom per its position in the contiguous address space of each ASU. The capacity of each Logical Volume shall be stated. In conjunction with this table, the Test Sponsor shall provide a complete description of the type of data protection (see Clause 2.4.5) used on each Logical Volume. Logical Volume Capacity and Mapping ASU-1 (15, GB) ASU-2 (15, GB) ASU-3 (3, GB) 36 Logical Volume GB per Logical Volume ( GB used per Logical Volume) 36 Logical Volume GB per Logical Volume ( GB used per Logical Volume) 8 Logical Volume GB per Logical Volume ( GB used per Logical Volume) The Data Protection Level used for all Logical Volumes was Mirrored as described on page 11. See ASU Configuration in the IOPS Test Results File for more detailed configuration information.

21 DATA REPOSITORY Page 21 of 83 Storage Capacity Utilization Clause The FDR will include a table illustrating the storage capacity utilization values defined for Application Utilization (Clause 2.8.1), Protected Application Utilization (Clause 2.8.2), and Unused Storage Ratio (Clause 2.8.3). Clause 2,8.1 Application Utilization is defined as Total ASU Capacity divided by Physical Storage Capacity. Clause 2,8.2 Protected Application Utilization is defined as (Total ASU Capacity plus total Data Protection Capacity minus unused Data Protection Capacity) divided by Physical Storage Capacity. Clause 2,8.3 Unused Storage Ratio is defined as Total Unused Capacity divided by Physical Storage Capacity and may not exceed 45%. SPC-1 Storage Capacity Utilization Application Utilization 34.11% Protected Application Utilization 68.24% Unused Storage Ratio 9.78%

22 SPC-1 BENCHMARK EXECUTION RESULTS Page 22 of 83 SPC-1 BENCHMARK EXECUTION RESULTS This portion of the Full Disclosure Report documents the results of the various SPC-1 Tests, Test Phases, and Test Runs. SPC-1 Test Execution Definitions on page 55 contains definitions of terms specific to the SPC-1 Tests, Test Phases, and Test Runs. Clause The Tests must be executed in the following sequence: Primary Metrics, Repeatability, and Data Persistence. That required sequence must be uninterrupted from the start of Primary Metrics to the completion of Persistence Test Run 1. Uninterrupted means the Benchmark Configuration shall not be power cycled, restarted, disturbed, altered, or adjusted during the above measurement sequence. If the required sequence is interrupted other than for the Host System/TSC power cycle between the two Persistence Test Runs, the measurement is invalid. SPC-1 Tests, Test Phases, and Test Runs The SPC-1 benchmark consists of the following Tests, Test Phases, and Test Runs: Primary Metrics Test Sustainability Test Phase and Test Run IOPS Test Phase and Test Run Response Time Ramp Test Phase o 95% of IOPS Test Run o 90% of IOPS Test Run o 80% of IOPS Test Run o 50% of IOPS Test Run o 10% of IOPS Test Run (LRT) Repeatability Test Repeatability Test Phase 1 o 10% of IOPS Test Run (LRT) o IOPS Test Run Repeatability Test Phase 2 o 10% of IOPS Test Run (LRT) o IOPS Test Run Data Persistence Test Data Persistence Test Run 1 Data Persistence Test Run 2 Each Test is an atomic unit that must be executed from start to finish before any other Test, Test Phase, or Test Run may be executed. The results from each Test, Test Phase, and Test Run are listed below along with a more detailed explanation of each component.

23 SPC-1 BENCHMARK EXECUTION RESULTS Page 23 of 83 PRIMARY METRICS TEST SUSTAINABILITY TESTPHASE Primary Metrics Test Sustainability Test Phase Clause The Sustainability Test Phase has exactly one Test Run and shall demonstrate the maximum sustainable I/O Request Throughput within at least a continuous three (3) hour Measurement Interval. This Test Phase also serves to insure that the TSC has reached Steady State prior to reporting the final maximum I/O Request Throughput result (SPC-1 IOPS ). Clause The computed I/O Request Throughput of the Sustainability Test must be within 5% of the reported SPC-1 IOPS result. Clause The Average Response Time, as defined in Clause 5.1.1, will be computed and reported for the Sustainability Test Run and cannot exceed 30 milliseconds. If the Average Response time exceeds that 30-milliseconds constraint, the measurement is invalid. Clause For the Sustainability Test Phase the FDR shall contain: 1. A Data Rate Distribution graph and data table. 2. I/O Request Throughput Distribution graph and data table. 3. A Response Time Frequency Distribution graph and table. 4. An Average Response Time Distribution graph and table. 5. The human readable Test Run Results File produced by the Workload Generator (may be included in an appendix). 6. A listing or screen image of all input parameters supplied to the Workload Generator (may be included in an appendix). 7. The Measured Intensity Multiplier for each I/O stream. 8. The variability of the Measured Intensity Multiplier, as defined in Clause SPC-1 Workload Generator Input Parameters The SPC-1 Workload Generator input parameters for the Sustainability, IOPS, Response Time Ramp, Repeatability, and Persistence Test Runs are documented in Appendix E: SPC-1 Workload Generator Input Parameters on Page 80. Sustainability Test Results File A link to the test results file generated from the Sustainability Test Run is listed below. Sustainability Test Results File

24 SPC-1 BENCHMARK EXECUTION RESULTS Page 24 of 83 PRIMARY METRICS TEST SUSTAINABILITY TESTPHASE Sustainability Data Rate Distribution Data (MB/second) The Sustainability Data Rate table of data is not embedded in this document due to its size. The table is available via the following URL: Sustainability Data Rate Table Sustainability Data Rate Distribution Graph Data Rate Distribution BSUs) All ASUs ASU1 ASU2 ASU3 500 Startup Data Rate MBytes/Sec Test Run Minutes

25 SPC-1 BENCHMARK EXECUTION RESULTS Page 25 of 83 PRIMARY METRICS TEST SUSTAINABILITY TESTPHASE Sustainability I/O Request Throughput Distribution Data The Sustainability I/O Request Throughput table of data is not embedded in this document due to its size. The table is available via the following URL: Sustainability I/O Request Throughput Table Sustainability I/O Request Throughput Distribution Graph I/O Request Throughput Distribution BSUs) All ASUs ASU1 ASU2 ASU Startup I/O Requests per Second Test Run Minutes

26 SPC-1 BENCHMARK EXECUTION RESULTS Page 26 of 83 PRIMARY METRICS TEST SUSTAINABILITY TESTPHASE Sustainability Average Response Time (ms) Distribution Data The Sustainability Average Response Time table of data is not embedded in this document due to its size. The table is available via the following URL: Sustainability Average Response Time Table Sustainability Average Response Time (ms) Distribution Graph Average Response Time Distribution BSUs) All ASUs ASU1 ASU2 ASU3 50 Startup Average Response Time (ms) Test Run Minutes

27 SPC-1 BENCHMARK EXECUTION RESULTS Page 27 of 83 PRIMARY METRICS TEST SUSTAINABILITY TESTPHASE Sustainability Response Time Frequency Distribution Data Response Time (ms) > > > > > > > Read 22,436,197 26,131,727 8,377,398 13,267,702 12,791,448 10,823,537 6,953,400 3,856,668 Write 264, ,075,303 34,773,858 10,674,357 3,295, , ,321 80,645 All ASUs 22,700, ,207,030 43,151,256 23,942,059 16,087,252 11,794,168 7,233,721 3,937,313 ASU1 18,643,897 97,035,042 18,214,524 15,666,244 12,845,810 10,411,281 6,614,999 3,652,375 ASU2 4,003,000 23,848,860 4,071,500 2,288,814 1,417, , , ,456 ASU3 53,930 70,323,128 20,865,232 5,987,001 1,824, , ,539 43,482 Response Time (ms) > > > > > > > > Read 3,442,660 1,536,706 1,028, , , ,654 1,521,037 1,547,669 Write 32,169 6,196 3,742 3,341 3,019 2,882 5,597 5,222 All ASUs 3,474,829 1,542,902 1,032, , , ,536 1,526,634 1,552,891 ASU1 3,247,526 1,448, , , , ,480 1,438,352 1,447,047 ASU2 210,099 90,967 58,338 45,565 40,656 39,303 84, ,281 ASU3 17,204 3,160 1,907 1,880 1,705 1,753 3,546 3,563 Response Time (ms) > > > > > > > >30.0 Read 1,582,129 1,514,505 1,575,848 8,867,789 2,885,675 1,160, ,829 5,976,214 Write 5,281 5,128 4,833 21,140 16,437 16,030 15, ,659 All ASUs 1,587,410 1,519,633 1,580,681 8,888,929 2,902,112 1,176, ,406 6,248,873 ASU1 1,459,481 1,367,106 1,394,578 7,573,771 1,939, , ,201 3,514,937 ASU2 124, , ,554 1,298, , , ,296 2,470,314 ASU3 3,728 3,725 3,549 16,716 15,246 15,231 14, ,622 Sustainability Response Time Frequency Distribution Graph Response Time Frequency Distribution BSUs) 200,000,000 Read Write All ASUs 180,000, ,000,000 Number of Occurrances 140,000, ,000, ,000,000 80,000,000 60,000,000 40,000,000 20,000, > > > > > > > > > > Response Time in Milliseconds > > > > > > > > > > > > >30.0

28 SPC-1 BENCHMARK EXECUTION RESULTS Page 28 of 83 PRIMARY METRICS TEST SUSTAINABILITY TESTPHASE Sustainability Measured Intensity Multiplier and Coefficient of Variation Clause IM Intensity Multiplier: The ratio of I/Os for each I/O stream relative to the total I/Os for all I/O streams (ASU1-1 ASU3-1) as required by the benchmark specification. Clauses and MIM Measured Intensity Multiplier: The Measured Intensity Multiplier represents the ratio of measured I/Os for each I/O stream relative to the total I/Os measured for all I/O streams (ASU1-1 ASU3-1). This value may differ from the corresponding Expected Intensity Multiplier by no more than 5%. Clause COV Coefficient of Variation: This measure of variation for the Measured Intensity Multiplier cannot exceed 0.2. ASU1-1 ASU1-2 ASU1-3 ASU1-4 ASU2-1 ASU2-2 ASU2-3 ASU3-1 IM MIM COV

29 SPC-1 BENCHMARK EXECUTION RESULTS Page 29 of 83 PRIMARY METRICS TEST IOPSTESTPHASE Primary Metrics Test IOPS Test Phase Clause The IOPS Test Phase consists of one Test Run at the 100% load point with a Measurement Interval of ten (10) minutes. The IOPS Test Phase immediately follows the Sustainability Test Phase without any interruption or manual intervention. The IOPS Test Run generates the SPC-1 IOPS primary metric, which is computed as the I/O Request Throughput for the Measurement Interval of the IOPS Test Run. The Average Response Time is computed for the IOPS Test Run and cannot exceed 30 milliseconds. If the Average Response Time exceeds the 30 millisecond constraint, the measurement is invalid. Clause For the IOPS Test Phase the FDR shall contain: 1. I/O Request Throughput Distribution (data and graph). 2. A Response Time Frequency Distribution. 3. An Average Response Time Distribution. 4. The human readable Test Run Results File produced by the Workload Generator. 5. A listing or screen image of all input parameters supplied t the Workload Generator. 6. The total number of I/O Requests completed in the Measurement Interval as well as the number of I/O Requests with a Response Time less than or equal to 30 milliseconds and the number of I/O Requests with a Response Time greater than 30 milliseconds. SPC-1 Workload Generator Input Parameters The SPC-1 Workload Generator input parameters for the Sustainability, IOPS, Response Time Ramp, Repeatability, and Persistence Test Runs are documented in Appendix E: SPC-1 Workload Generator Input Parameters on Page 80. IOPS Test Results File A link to the test results file generated from the IOPS Test Run is listed below. IOPS Test Results File

30 SPC-1 BENCHMARK EXECUTION RESULTS Page 30 of 83 PRIMARY METRICS TEST IOPSTESTPHASE IOPS Test Run I/O Request Throughput Distribution Data 660 BSUs Start Stop Interval Duration Start-Up/Ramp-Up 20:05:24 20:08: :03:00 Measurement Interval 20:08:24 20:18: :10:00 60 second intervals All ASUs ASU1 ASU2 ASU3 0 32, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Average 32, , , , IOPS Test Run I/O Request Throughput Distribution Graph I/O Request Throughput Distribution BSUs) All ASUs ASU1 ASU2 ASU Startup Measurement Interval I/O Requests per Second Test Run Minutes

31 SPC-1 BENCHMARK EXECUTION RESULTS Page 31 of 83 PRIMARY METRICS TEST IOPSTESTPHASE IOPS Test Run Average Response Time (ms) Distribution Data 660 BSUs Start Stop Interval Duration Start-Up/Ramp-Up 20:05:24 20:08: :03:00 Measurement Interval 20:08:24 20:18: :10:00 60 second intervals All ASUs ASU1 ASU2 ASU Average IOPS Test Run Average Response Time (ms) Distribution Graph Average Response Time Distribution BSUs) All ASUs ASU1 ASU2 ASU3 50 Startup Measurement Interval Average Response Time (ms) Test Run Minutes

32 SPC-1 BENCHMARK EXECUTION RESULTS Page 32 of 83 PRIMARY METRICS TEST IOPSTESTPHASE IOPS Test Run Response Time Frequency Distribution Data Response Time (ms) > > > > > > > Read 1,117,920 1,528, , , , , , ,722 Write 9,562 8,787,711 2,161, , ,567 69,671 21,362 6,642 All ASUs 1,127,482 10,316,335 2,682,380 1,437, , , , ,364 ASU1 931,967 5,306,669 1,147, , , , , ,789 ASU2 193,597 1,296, , ,426 84,361 49,495 26,240 13,075 ASU3 1,918 3,713,031 1,273, , ,297 37,668 11,392 3,500 Response Time (ms) > > > > > > > > Read 177,566 80,054 57,049 50,987 48,556 47,343 94,151 94,592 Write 2, All ASUs 180,185 80,493 57,262 51,194 48,726 47,503 94,455 94,883 ASU1 167,782 75,786 54,408 48,781 46,613 45,397 89,953 89,398 ASU2 11,011 4,483 2,746 2,295 2,010 1,994 4,293 5,274 ASU3 1, Response Time (ms) > > > > > > > >30.0 Read 95,031 90,547 92, , ,273 64,455 47, ,868 Write ,225 1,024 1, ,978 All ASUs 95,333 90,870 92, , ,297 65,491 48, ,846 ASU1 88,548 82,818 83, , ,813 39,545 29, ,519 ASU2 6,564 7,840 9,510 67,185 48,540 24,968 18, ,072 ASU ,255 IOPS Test Run Response Time Frequency Distribution Graph Response Time Frequency Distribution BSUs) 12,000,000 Read Write All ASUs 10,000,000 Number of Occurrances 8,000,000 6,000,000 4,000,000 2,000, > > > > > > > > > > > > Response Time in Milliseconds > > > > > > > > > > >30.0

33 SPC-1 BENCHMARK EXECUTION RESULTS Page 33 of 83 PRIMARY METRICS TEST IOPSTESTPHASE IOPS Test Run I/O Request Information I/O Requests Completed in the Measurement Interval I/O Requests Completed with Response Time = or < 30 ms I/O Requests Completed with Response Time > 30 ms 19,798,726 19,408, ,846 IOPS Test Run Measured Intensity Multiplier and Coefficient of Variation Clause IM Intensity Multiplier: The ratio of I/Os for each I/O stream relative to the total I/Os for all I/O streams (ASU1-1 ASU3-1) as required by the benchmark specification. Clauses and MIM Measured Intensity Multiplier: The Measured Intensity Multiplier represents the ratio of measured I/Os for each I/O stream relative to the total I/Os measured for all I/O streams (ASU1-1 ASU3-1). This value may differ from the corresponding Expected Intensity Multiplier by no more than 5%. Clause COV Coefficient of Variation: This measure of variation for the Measured Intensity Multiplier cannot exceed 0.2. ASU1-1 ASU1-2 ASU1-3 ASU1-4 ASU2-1 ASU2-2 ASU2-3 ASU3-1 IM MIM COV

34 SPC-1 BENCHMARK EXECUTION RESULTS Page 34 of 83 PRIMARY METRICS TEST RESPONSE TIME RAMP TEST PHASE Primary Metrics Test Response Time Ramp Test Phase Clause The Response Time Ramp Test Phase consists of five Test Runs, one each at 95%, 90%, 80%, 50%, and 10% of the load point (100%) used to generate the SPC-1 IOPS primary metric. Each of the five Test Runs has a Measurement Interval of ten (10) minutes. The Response Time Ramp Test Phase immediately follows the IOPS Test Phase without any interruption or manual intervention. The five Response Time Ramp Test Runs, in conjunction with the IOPS Test Run (100%), demonstrate the relationship between Average Response Time and I/O Request Throughput for the Tested Storage Configuration (TSC) as illustrated in the response time/throughput curve on page 13. In addition, the Average Response Time measured during the 10% Test Run is the value for the SPC-1 LRT metric. That value represents the Average Response Time of a lightly loaded TSC. Clause The following content shall appear in the FDR for the Response Time Ramp Phase: 1. A Response Time Ramp Distribution. 2. The human readable Test Run Results File produced by the Workload Generator for each Test Run within the Response Time Ramp Test Phase. 3. For the 10% Load Level Test Run (SPC-1 LRT metric) an Average Response Time Distribution. 4. A listing or screen image of all input parameters supplied to the Workload Generator. SPC-1 Workload Generator Input Parameters The SPC-1 Workload Generator input parameters for the Sustainability, IOPS, Response Time Ramp, Repeatability, and Persistence Test Runs are documented in Appendix E: SPC-1 Workload Generator Input Parameters on Page 80. Response Time Ramp Test Results File A link to each test result file generated from each Response Time Ramp Test Run list listed below. 95% Load Level 90% Load Level 80% Load Level 50% Load Level 10% Load Level

35 SPC-1 BENCHMARK EXECUTION RESULTS Page 35 of 83 PRIMARY METRICS TEST RESPONSE TIME RAMP TEST PHASE TEST PHASE Response Time Ramp Distribution (IOPS) Data The five Test Runs that comprise the Response Time Ramp Phase are executed at 95%, 90%, 80%, 50%, and 10% of the Business Scaling Unit (BSU) load level used to produce the SPC-1 IOPS primary metric. The 100% BSU load level is included in the following Response Time Ramp data tables and graphs for completeness. 100% Load Level BSUs Start Stop Interval Duration 95% Load Level BSUs Start Stop Interval Duration Start-Up/Ramp-Up 20:05:24 20:08: :03:00 Start-Up/Ramp-Up 20:18:42 20:21: :03:00 Measurement Interval 20:08:24 20:18: :10:00 Measurement Interval 20:21:42 20:31: :10:00 (60 second intervals) All ASUs ASU-1 ASU-2 ASU-3 (60 second intervals) All ASUs ASU-1 ASU-2 ASU , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Average 32, , , , Average 31, , , , % Load Level BSUs Start Stop Interval Duration 80% Load Level BSUs Start Stop Interval Duration Start-Up/Ramp-Up 20:31:59 20:34: :03:00 Start-Up/Ramp-Up 20:45:17 20:48: :03:00 Measurement Interval 20:34:59 20:44: :10:00 Measurement Interval 20:48:17 20:58: :10:00 (60 second intervals) All ASUs ASU-1 ASU-2 ASU-3 (60 second intervals) All ASUs ASU-1 ASU-2 ASU , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Average 29, , , , Average 26, , , , % Load Level BSUs Start Stop Interval Duration 10% Load Level - 66 BSUs Start Stop Interval Duration Start-Up/Ramp-Up 20:58:34 21:01: :03:00 Start-Up/Ramp-Up 17:27:52 17:30: :03:00 Measurement Interval 21:01:34 21:11: :10:00 Measurement Interval 17:30:52 17:40: :10:03 (60 second intervals) All ASUs ASU-1 ASU-2 ASU-3 (60 second intervals) All ASUs ASU-1 ASU-2 ASU , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Average 16, , , , Average 3, ,