Microsoft SQL Server 2014 in a Flash

Transcription

1 Microsoft SQL Server 2014 in a Flash Combine Violin s enterprise- class all- flash storage with the ease and capabilities of Windows Storage Server in an integrated solution so you can achieve higher performance and efficiency for your SQL Server 2014 databases and business critical Microsoft applications January 2015 Abstract Microsoft SQL Server 2014 scales to support large workloads in the data center. Yet legacy storage solutions are keeping SQL Server databases and related applications from reaching their full potential, which means expensive server investments are underutilized. The ideal solution delivers enterprise- class high- performance high- capability storage that is tightly integrated with the Windows environment, holistically managed by native Windows tools and utilities, and delivers state of the art efficiency as part of the journey to the All Flash Data Center. This increases the value of existing human and technology investments while transforming SQL Server performance and economics. The result is enhanced business efficiency, resiliency, and ROI for all of IT.

2 Table of Contents 1. Introduction Performance challenges in SQL Server 2014 environments Higher performance storage for SQL Server Violin Windows Flash Array + Windows Storage Server 2012 R Microsoft ecosystem friendly What is SMB Direct? Performance that transforms business Enhance performance for SQL Server Resiliency for continuous availability Scalability to grow efficiently Non- disruptive volume extensibility Pay- as- you- grow scalability Efficient scalability for SQL Server Relevant data services when you need them Case Study: Australian Department of Defence The challenge The solution Conclusion Pg. 2

3 1. Introduction Microsoft SQL Server 2014 has grown far beyond that of a simple workgroup database. It is a mature and sophisticated platform suitable for large enterprise workloads including scalable online transaction processing (OLTP), parallel data warehouses (DW), business intelligence (BI), and cloud computing. Many organizations have grown up around SQL Server; it is an integral part of their operations. Likewise, businesses expect their SQL Server environment will grow with them; however, limitations in the current IT infrastructure have made achieving this goal difficult, if not impossible. The scalability of SQL Server has increased dramatically with the 2012 and 2014 versions of the product. SQL Server Enterprise edition running on Windows Server 2012 can support up to 64 physical processors and 640 logical processors. The Microsoft SQL Server 2012 and 2014 Fast Track Data Warehouse reference architectures illustrate how much SQL Server s capabilities have matured to be able to deliver proven performance and scalability for data warehouse workloads. At the same time, Microsoft s self- service Business Intelligence initiative is empowering Excel and SharePoint users with new ways to access SQL Server data, which in turns raises demand for SQL Server resources. In many organizations, SQL Server is being deployed within and outside of centralized IT control. Any tactically deployed SQL Server resources are most likely not optimized for performance nor manageability, including safeguards against outages. Windows Server Hyper- V virtualization is mature and scalable and offers a way to recentralize many SQL Server deployments and ensure their availability without taking away local or line of business control over the application. In a nod to future possibilities, many are investigating how Microsoft Azure SQL Database may prove to be an integration consideration later. 2. Performance challenges in SQL Server 2014 environments Although CPU performance and network connectivity have grown dramatically over the past several years, improvements in storage have largely been limited to increased capacity, not performance and capabilities. Legacy storage solutions (disk drives) perform best when read and write operations are linear. Yet modern workloads such as SQL Server databases and Hyper- V virtual machines (VMs) are filled with random I/O requests, which results in high latency and poor application performance. Various striping techniques, solid- state drives, and optimization algorithms seek to improve operations yet I/O performance remains lacking. To compensate, IT managers over provision storage and/or add cache to controllers in an attempt to garner enough I/O capacity; however, this approach is inefficient and enforces continued spending on capacity when the need is actually for lower latency and higher IOPS. Both CAPEX and OPEX are negatively affected by the status quo. Poor storage performance leads to poor CPU utilization. CPUs spend more time waiting than executing, which means servers are expensive and mostly idle investments. This reality has not escaped the attention of CIOs, CFOs, and other senior management. In an attempt to scale beyond the workgroup level, some IT directors have ported Windows applications to another storage platform such as a SAN, or different application altogether. For example, former SQL Server data may have graduated to an Oracle implementation. One way to raise server utilization is virtualization. Yet ironically, virtualization drives random I/O even higher and some virtualization solutions raise server overhead. Organizations are faced with an unpleasant reality that their only choice in responding to growth has meant scaling the inherent inefficiency and expense of legacy storage solutions. Storage area networks (SANs) typically require vendor- specific tools and skill sets. Organizations cannot easily leverage the management skills of their existing Microsoft- savvy staff, which reduces flexibility and increases operational expense. Even some direct attached storage (DAS) solutions may require special handling or expertise in their set up and operation. This is just one shortcoming of bifurcated storage where Ethernet- based storage is the norm for file services (the bulk of the Microsoft universe), and dedicated block- based storage supports high- performance applications such as SQL Server and Hyper- V. Pg. 3

4 Ensuring continuous availability is a challenge, but essential. For example, AlwaysOn Availability Groups are one approach to maintain SQL Server availability; however, there are limitations. AlwaysOn is host- based and does not fully mirror all data such as system databases, login data, scheduler tasks, etc. Setting up availability groups is time consuming and requires ongoing administration. Replication is an alternative but asynchronous replication can suffer data loss if the primary array goes down. Synchronous mirroring is more resilient, but it is expensive and has a slow two- phase commit process. In either case, close coordination of application and storage administrators is needed during scheduled maintenance. From these challenges, it is clear that a new approach is needed to deliver the performance and scalability required for your SQL Server environment to achieve its full potential without breaking the IT budget. 3. Higher performance storage for SQL Server 2014 To address the shortcomings of legacy storage, Violin Memory and Microsoft have co- developed the Violin Windows Flash Array (WFA). The WFA combines Violin Memory s patented Flash Fabric Architecture (FFA), Microsoft s fast SMB Direct protocol, and Microsoft Windows Storage Server 2012 R2 to deliver storage that easily keeps up with your needs. With the WFA, you can start the process of transforming your data center economics from legacy expense levels to that of the All Flash Data Center Violin Windows Flash Array + Windows Storage Server 2012 R2 The WFA is based upon the Violin All Flash Array Dual blades run Windows Storage Server 2012 R2 with support for SMB Direct through RDMA- enabled network cards. The WFA architecture features sub- millisecond latency (as illustrated in Figure 3-1 on page 5) and wide stripe vraid accelerated switched flash for maximum performance. With SMB Direct, the WFA delivers the features of SAN, the performance of DAS, at the price of NAS. You get maximum performance for every byte every time. Table 3-1: Violin Windows Flash Arrays by model Windows Flash Array Model WFA- 64 WFA- 48 WFA- 32 WFA- 24 WFA- 16 Form Factor / Flash Type 3U / MLC 3U / MLC 3U / MLC 3U / MLC 3U / MLC Raw Capacity (TB) Usable Capacity (TB) at 84% format level I/O Connectivity 10GbE, 56Gb IB 10GbE, 56Gb IB 10GbE, 56Gb IB 10GbE, 56Gb IB 10GbE, 56Gb IB Max. 4KB IOPS 1.1M IOPS 1.1M IOPS 800k IOPS 800k IOPS 800k IOPS Max. Bandwidth 4GB/s 4GB/s 4GB/s 4GB/s 4GB/s Nominal Latency <500 µsec <500 µsec <500 µsec <500 µsec <500 µsec Windows Storage Sever 2012 R2 delivers enterprise data services including deduplication, live migration, thin provisioning, compression, replication, Scale- out File Server (SOFS), and data encryption. The SMB 3.0 protocol boosts network storage performance. In particular, SMB Direct (SMB 3.0 over RDMA) enables remotely stored data on the WFA to be read directly into application servers memory. The result is remote storage performing like blazing fast local storage. In fact, Microsoft and Violin collaborated extensively to optimize Windows Storage Server 2012 R2 at the kernel level to take full advantage of our FFA to delivery enterprise- class performance for the Violin Flash Array. Pg. 4

5 Figure 3-1: WFA- 64 4K Random Read Latency over Increasing Queue Depth The WFA helps you attain 24x7 operations for SQL Server databases and related applications through multi- layered hardware and software resiliency. Active- Active clustering delivers continuous uptime and SMB Multichannel ensures that cluster node connectivity is maintained. With the WFA, you can get higher utilization and performance with a smaller, less expensive IT footprint and you can scale as business conditions warrant. You may find that you no longer need dedicated storage to support your high- performance Windows applications Microsoft ecosystem friendly Figure 3-2: Wait State Reduction Pg. 5

6 Violin Memory and Microsoft co- developed and optimized the WFA to leverage its unique performance profile. SMB 3.0 and SMB Direct are part of the Microsoft strategic roadmap for network storage and the WFA fully supports these protocols to deliver performance and scalability not available elsewhere. There is no need for third- party storage management tools or skill sets. Your existing Microsoft admin staff can manage all of your servers, storage, and applications with Microsoft System Center and PowerShell. The tight integration of the WFA with Microsoft management tools delivers a holistic view of your Microsoft environment for simplified and cost- effective management of physical and virtual servers What is SMB Direct? The WFA addresses the traditional bottlenecks that limit performance by delivering a state- of- the- art storage solution where speed, scale, and resiliency combine to deliver operational and financial efficiency. The use of SMB Direct can improve the performance of most vendors storage and we expect that over time most will offer this support. However, the combination of SMB Direct with the Violin FFA and kernel- level optimization of Windows Storage Server 2012 R2 uniquely takes performance to the next level. With the WFA, you get ultra- fast read and write access with sustained throughput of up to 1.1 million 4K IOPS with greater than 4 GB/s of bandwidth. SMB Direct recoups up to 30% of your SQL Server s CPU utilization, which provides you with more flexibility in how you allocate server resources and can have a dramatic effect on your CAPEX and OPEX requirements. As illustrated in Figure 3-2, your servers will spend more time doing than waiting so you can support additional workloads with existing resources and achieve faster response times. Our support for SMB 3.0 along with a suite of enterprise data services delivers a high- performance feature- rich storage solution that will help ensure you can deliver business in a flash Performance that transforms business Meeting growing demand has strained databases as legacy storage solutions hit their design limits. The WFA is particularly suited for transaction- oriented workloads driven by SQL Server and others including Hyper- V virtualization that have a high degree of random I/O activity. By raising the available I/O to meet processor demand, the WFA enables SQL Server to spend more time processing data as opposed to waiting for it. This means you can optimize CPU investments across SQL workloads to meet your performance objectives. In addition, you can run multiple mixed workloads with consistent performance and linear scalability. With the WFA, you can improve SQL Server performance today while aligning with Microsoft s technology roadmap for networked storage. You can benefit from full support for the SMB 3.0 feature set operating on a low- latency (~500 microseconds) all- flash SMB Direct solution. Tight integration with Windows Storage Server 2012 R2 ensures support for large sector (4K) storage as well as other performance optimizations collaboratively developed between Microsoft and Violin. Pg. 6

7 Figure 3-3: WFA- 64 4K Random Read Performance (SQLIO) As illustrated in Figure 3-3, the 4K random read performance achieved by our scale- out reference architecture reveals that the WFA delivers over 1 million IOPS and 4 GB/s of bandwidth at queue depth of 64 in a single array. By adding a second WFA to the cluster, performance and capacity scales linearly. Likewise, Figure 3-4 shows the corresponding 4K random write performance of over 810k IOPS and 3 GB/s of bandwidth at queue depth of 64 in a single array. Figure 3-4: WFA- 64 4K Random Write Performance (SQLIO) The WFA is well suited to the short block random read/write workloads associated with OLTP, which is a critical application for most any enterprise. Pg. 7

8 Our parallel read/write across all of the flash is excellent for parallel data warehouses where there are many simultaneous related queries. Figure 3-5 highlights the sustained bandwidth in excess of 4 GB/s at queue depth of one that a single WFA can deliver in data warehouse or other large block- size sequential access environments. Figure 3-5: WFA K Sequential Read Performance Similarly, Figure 3-6 shows the matching 512K sequential write performance of over 810k IOPS and 3 GB/s of bandwidth at queue depth of one in a single array. Figure 3-6: WFA K Sequential Write Performance Pg. 8

9 Connecting through SMB Direct can free up to 30% of CPU usage; these cycles could be put towards BI initiatives or other value added activities. Likewise, if consolidation or virtualization of SQL Server instances is your preferred approach, the WFA can enable you to increase virtualization density. Unlike legacy solutions, our all- flash arrays do not suffer from write cliff due to the non- blocking nature of our distributed garbage collection. The WFA s architecture delivers maximum performance for every byte every time regardless of block- size or the random or sequential nature of access. The bottom line is that you can experience higher SQL Server performance through an optimized storage platform that is smaller and with a less expensive IT footprint Enhance performance for SQL Server 2014 One of the selling points of SQL Server 2014 is its new in- memory processing capability for OLTP. However, for most users this very specialized tool will not apply to the bulk of their SQL Server workloads. Nevertheless, the high performance delivered by the WFA can dramatically improve most any SQL Server implementation. While in- memory processing can boost specific transaction processing speeds, to maintain overall high performance you need persistent storage that is up to the task of supporting the rest of the DBMS, otherwise database performance will be constrained. When SQL Server 2014 workloads access the WFA through SMB Direct, you get ultra- fast read and write commits. Customer performance testing has revealed that the WFA delivers up to twice the SQL writes throughput and up to 50% higher SQL read performance and compared with an industry standard all- flash array. Therefore, you can achieve consistent high performance for SQL Server 2014 overall. In addition, SMB Direct can reduce application CPU demand by up to 30% thus giving you greater flexibility in how your structure your SQL Server resources. With Hyper- V virtualization, you can optimize CPU cycles without the performance penalty of other virtualization solutions Resiliency for continuous availability With the WFA, you benefit from continuous availability of Microsoft SharePoint and other in- house developed applications using SQL Server data stores along with reduced risk to operations through multiple layers of hardware- and software- based protection. Windows Failover Clustering in conjunction with the WFA hardware can sustain any single point of failure to ensure continuous operations. SMB Multichannel I/O provides bandwidth trunking as well as enhanced node- to- node network connectivity and availability. Should you desire a high availability configuration, deployment is option- rich and as simple as software configuration. In addition, in- flight and at- rest encryption guards against eavesdropping or unauthorized data disclosure. Pg. 9

10 3.7. Scalability to grow efficiently Figure 3-7: Performance Improvements WFA vs. Industry Standard All Flash Array With the WFA, you can optimize CPU investments across SQL Server workloads to achieve greater scale and flexibility in how you maintain consistent performance while adding users, threads, tables, etc. As illustrated in Figure 3-7, the superior performance and linear scalability of the WFA will enable your organization to support a greater number or larger size of SQL Server workloads with existing infrastructure. With increased I/O capacity, freed up CPU cycles, and decreased latency, the WFA scales predictability by delivering consistent performance even as your mix of workloads change Non-disruptive volume extensibility The WFA allows you to scale up to meet growing demand. SOFS enables linear scalability of capacity and performance. You can scale out to four arrays (eight nodes) per cluster. Ensuring continuous availability is especially important as SQL Server is the back end for many applications and their data growth is driving SQL Server demand. For example, the Microsoft self- service Business Intelligence (BI) initiative interconnects SharePoint Server and Microsoft Excel with the Microsoft SQL Server BI platform. Through Excel, self- service BI empowers end users to process millions of rows of data, and provide customer reporting without involving the IT department Pay-as-you-grow scalability Having the resources and flexibility to support growth is essential for a successful enterprise; however, the timing of this investment expense is equally important. With WFA s pay- as- you- grow pricing, you can non- disruptively scale capacity to meet your business needs without having to order and install new capacity in advance. This enables you to more closely align CAPEX with the benefit received. As a result, you can non- disruptively scale your WFA raw capacity in 8.8 TB increments within one of two ranges, TB or TB, based upon your timing. You can scale capacity without full expenditure at the onset, but maintain the operational flexibility to license- up the capacity to support your business growth over time. Pg. 10

11 Efficient scalability for SQL Server Consolidation or virtualization of SQL Server workloads is unleashed with the WFA. SQL Server can now scale to enterprise workloads while delivering consistent service levels so you can eliminate application growth barriers, which increases your investment protection. As your workloads change, Storage Live Migration streamlines virtual machine (VM) migration through direct WFA to WFA communication that bypasses the VM host to complete the process more rapidly. This delivers increased flexibility and simplicity in addressing potential service interruptions due to sudden workload spikes or seasonal peaks. In addition, Windows Server Hyper- V does not suffer from the up to the 20% overhead of competitive virtualization solutions so you can consolidate workloads such as SQL Server with little, if any, performance penalty. Since you can consolidate and scale SQL Server to meet your performance needs, there is no imperative to migrate to another platform. In fact, you might find that the combination of SQL Server and the WFA yields better performance than competitive solutions, even DAS. Developers can easily establish new test beds, which closely match the speed and architecture of the production environment, for development, testing or quality assurance for your strategic applications. Roll out and roll up is quick and simple, with minimal waiting on IT administrators. What s more, the WFA is a very efficient private cloud platform. As you consider the role private and/or public clouds might play in your IT infrastructure, be assured that the WFA is a future- proof technology investment with a defined transition or interaction path for Microsoft Azure SQL Database. Should Microsoft Azure be part of your data center strategy, you can easily move files between the WFA and Azure Data Services through backup, replication, or custom developed applications to maximize flexibility in your choice of onsite and offsite cloud storage Relevant data services when you need them Table 3-2 highlights some of the many data services offered by the WFA to increase the efficiency of your data center. Table 3-2: Enterprise Data Services Provided by the Windows Flash Array Storage and File System File and Block Access Networking Data Deduplication Compression NTFS Availability Offloaded Data Transfer (ODX) Thin Provisioning Encryption SMB 3.0 NFS 3.0 and NFS 4.1 Support for VMware VMs over NFS Scale- out File Server (SOFS) VSS for Remote SMB File Shares (snaps) SMB Direct (RDMA) SMB Multichannel Encryption Transparent Failover Clustering Virtualisation Management Cluster Shared Volumes v2 DFS Replication Live Storage Migration New VHDX standard Microsoft System Center PowerShell Unlike many competitive offerings, the WFA lets you choose which of the data services to deploy, and on which of the WFA nodes. For example, this granularity means that you can enable deduplication only on the shares that would benefit, such as those supporting virtual desktops. Windows Storage Sever 2012 R2 delivers many enterprise data services that help drive the efficiency of your storage solution. Most relevant to SQL Server environments are Windows Cluster Failover, live migration, and Scale- out File Server (SOFS). With the WFA, you are always protected against a single point of failure as the active- active node configuration ensures that data remains accessible even if an array controller should become inoperative. In addition, SMB Multipathing not only aggregates bandwidth for higher performance but can also detect connectivity failures and reroute traffic accordingly. Live migration of VMs delivers flexibility in resource allocation without impacting application servers or requiring taking applications or databases Pg. 11

12 offline. Logical organization of multiple large databases is streamlined through SOFS, which can deliver a single namespace across multiple WFAs thus simplifying the management of large amounts of data. 4. Case Study: Australian Department of Defence The Australian Department of Defence was an early adopter of the WFA. The HMAS Harman is the Royal Australian Navy communications and logistics facility that provides communications, network operations, and information systems. Located in Canberra, New South Wales, the base undertook a massive consolidation of the SQL Server environment located at HMAS Harman The challenge Originally, HMAS Harman was seeking to take advantage of high availability and virtualization that their current storage solution could not support. The combination of latency sensitive applications, massive IOPS requirements, and space limitations within the data center made the existing solution untenable. The workloads involved monitoring and dashboard reporting on network performance, Hyper- V applications, SQL Server, corporate and commercial off- the- shelf applications, and 94 software services. These supported over 110,000 network users spanning 450+ sites with 500+ data collection points. They needed to achieve a higher rate of server and storage consolidation while enabling the requisite performance to support their activities. The existing storage solution was not up to the task, so the department had to reconsider how it would meet its present objectives as well as support the inevitable data growth The solution The workloads in question had specific service level requirements that mandated all processing must be completed within the following data load windows: Software Services: 5 minute window for data feeds from 200+ devices in 15 SQL Server instances Business Intelligence: 15 minute data load window Network Performance and Systems Analysis: data is loaded every 6 and 24 hours In order to meet these service levels, the Australian DoD deployed two WFA- 64s and undertook a massive SQL Server consolidation initiative that was enabled by the low latency and high performance of the WFA. Pg. 12

13 Figure 4-1: Server and Licensing Reduction Achieved by the Australian DoD Using the WFA As shown in Figure 4-1, the number of physical application servers was reduced from 45 to two. In addition, the number of CPU sockets licensed for SQL Server was decreased from 90 to eight. The impact on the data center facilities requirement was equally significant with a 90% reduction in floor space and cooling requirements. Operationally, the new solution allowed HMAS Harman to simplify its data center architecture, which enabled additional savings in both IT and human resources. Since each WFA is two Windows Server Cluster nodes, it was possible to leverage the existing Microsoft admin staff to manage all of its physical and virtual servers, storage, and applications with from within the Microsoft System Center and PowerShell toolkit. Since Hyper- V is included at no cost with Windows Server 2012, license costs for virtual machine technology were eliminated. By moving away from the traditional SAN implementation, there was no longer a need for SAN hardware, third- party SAN storage management tools or related employee skill sets. This resulted in direct savings for the SAN hardware and management software but also reduced employee training costs since the pool of Microsoft- equipped admins could leverage their skill sets across a larger set of IT assets without incurring additional training expense. Pg. 13

14 Figure 4-2: Simplified Architecture Achieved with the WFA By updating its storage environment, the Royal Australian Navy communications and logistics facility at HMAS Harman was able to transform its data center by dramatically reducing the number of application servers, SQL Server licenses, floor space allocated to storage, and overall operational expense while simultaneously meeting all of its service level targets. The shift to a modern high- performance storage solution as enabled by the Violin Windows Flash Array fundamentally changed OPEX and CAPEX while delivering greater performance and efficiency. The Windows Flash Array has exceeded our expectations within the data warehouse and business reporting environments. The combination of the Windows 2012 technologies and Flash Array as a platform enables virtualized environments to be truly scalable and resilient with a performance level equal to or greater than direct attached flash storage. This translates to a data warehouse and reporting environment returning immediate value, outperforming traditional vendor offerings and with the lowest TCO compared to traditional storage arrays. Justin Gasparre, Director Situational Awareness, Service Integration Branch, CIO Group, Dept. of Defence, Australia Pg. 14

15 5. Conclusion Legacy storage solutions are built for capacity, not I/O performance. Yet today s Microsoft environments are performance oriented and this results in complex, expensive, and hard- to- scale solutions being deployed to meet the needs of business critical SQL Server databases and applications. The Violin Windows Flash Array is a fundamentally different and better storage solution for Microsoft SQL Server 2014 environments that addresses the performance shortcomings of legacy storage while enhancing the performance of in- memory database objects. The WFA is an all- flash active- active HA multi- node cluster platform with native Windows Storage Server 2012 R2 that delivers a rich suite of enterprise data services. The expense of a legacy data center based upon magnetic storage will become, if it has not already, a competitive disadvantage for any enterprise. The superior economics of the All Flash Data Center will cause executive staff such as the CIO, CTO, and CFO to take note. Even if your executive staff does not embrace the All Flash Data Center for performance and agility reasons, the financial benefits are too significant to pass up. It is simply a matter of time. The WFA is an enterprise- class storage solution that is at home in the All Flash Data Center. You can unleash your SQL Server database and application performance and free up existing resources to do more. In fact, the performance of the WFA is so high that you should choose to eliminate your existing dedicated storage infrastructure that is supporting your SQL Server databases. The WFA can help reduce CAPEX both in the short and long term. You can recoup existing server and facilities investments while delaying future expenditures, which will free up resources to invest in innovation and growing your business success. With the WFA, you can simplify your management paradigm with Microsoft System Center and PowerShell to achieve holistic control and flexibility of your Windows- based physical and virtual infrastructure. You can consolidate and virtualize a mix of workloads while maximizing your Windows performance on a storage architecture that is simpler and faster, and on Microsoft s strategic roadmap at substantial cost savings. If you would like to learn more about how the Violin Windows Flash Array can dramatically improve your SQL Server performance and transform your data center, please contact your Violin Memory representative today. Pg. 15

16 About Violin Memory Business in a Flash. Violin Memory transforms the speed of business with high performance, always available, low cost management of critical business information and applications. Violin s All Flash optimized solutions accelerate breakthrough CAPEX and OPEX savings for building the next generation data center. Violin s Flash Fabric Architecture (FFA) speeds data delivery with chip- to- chassis performance optimization that achieves lower consistent latency and cost per transaction for Cloud, Enterprise and Virtualized mission- critical applications. Violin's All Flash Arrays and Appliances, and enterprise data management software solutions enhance agility and mobility while revolutionizing data center economics. Founded in 2005, Violin Memory is headquartered in Santa Clara, California. For more information, visit memory.com. Follow us on Twitter at twitter.com/violinmemory Pg. 16