Virtualization: Current Benefits and Future Potential

Virtualization: Current Benefits and Future Potential Technology Concepts and Business Considerations Abstract This white paper reviews server and storage (file and network) virtualization technologies from EMC and how virtualization can address the problems of massive data growth and increased infrastructure complexity while supporting cost and performance goals of the enterprise. This paper also discusses the future of virtualization technology. January 2008

Copyright 2008 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com All other trademarks used herein are the property of their respective owners. Part number H3495 Technology Concepts and Business Considerations 2

Table of Contents Executive summary...4 Introduction...4 Audience... 5 What is virtualization?...5 Virtual memory... 5 Virtual networks... 6 Server virtualization...7 Server virtualization with VMware ESX... 7 Storage virtualization...12 Block-level virtualization... 13 File-level storage virtualization... 16 The future of virtualization...19 The EMC vision: A component-based infrastructure... 21 Managing a virtual infrastructure...23 Conclusion...24 Technology Concepts and Business Considerations 3

Executive summary Virtualization encompasses a powerful set of technologies. In the mainframe world, server virtualization has been in use for decades. Storage and network virtualization are also proven technologies. Rapid developments in IT and the need to reduce power consumption are causing the concepts surrounding virtualization to be examined in a new light and with new urgency. Enterprises are turning to virtualization to solve a multitude of vexing challenges. Storage virtualization can help cope with the consequences of enormous data growth and its impact on the corporate IT infrastructure, such as constant, resource-draining data migrations and storage reconfigurations. Server proliferation and excessive power consumption can be addressed by server virtualization while network virtualization consolidates communication infrastructure without compromising access controls that protect sensitive information. Virtualization achieves these benefits by providing logical representations of physical resources, removing resource limits, and improving utilization. Applied to servers, networks, and storage, virtualization supports the performance goals of the enterprise while controlling costs and simplifying infrastructure management. These benefits make virtualization an important strategic and tactical decision. It can be deployed incrementally starting with the data center s primary pain point whether that involves servers, networks, or storage. Virtualization s benefits increase as its deployment is extended across enterprise infrastructure, making the use of virtualization a strategic technology decision that pays off again and again. Introduction Off-the-charts digital data growth is a phenomenon in every area of the extended enterprise. In 2006, the amount of digital information created, captured, and replicated was 1,288 x 10 18 bits. That s 161 million terabytes enough to hold 3 million copies of every book ever written. Between 2006 and 2010, the information added annually to the digital universe will increase more than six-fold from 161 to 988 million terabytes. The largest component of this information is images, captured by more than 1 billion devices, ranging from small camera phones to large medical scanners. Still and moving images are replicated and rendered over the Internet, on private organizational networks, by PCs and servers, in data centers, in digital TV broadcasts, and on digital projection screens. 1 For enterprise IT, this growth means that data migrations and storage reconfigurations are constant, timeconsuming, and resource-draining activities. They delay the use of newly purchased storage, require staff to come in on weekends to oversee migrations, and take mission-critical business systems offline. In addition, the effects of data migrations frequently include hectic days for the corporate help desk when, despite everyone s best efforts, users can t find the same data on Monday that they could the previous Friday. In addition to massive data growth, IT must also manage increased infrastructure complexity the result of an ever greater reliance on enterprise information systems, including legacy and custom applications that support business goals. Two symptoms of this complexity are server sprawl and underutilized servers. For example, network services such as domain controllers often demand their own servers but may use less than 10 percent of a server s computational capability. 1 The Expanding Digital Universe, A Forecast of Worldwide Information Growth Through 2010; IDC; March 2007 Technology Concepts and Business Considerations 4

This white paper explains how virtualization can address these problems while supporting the performance goals of the enterprise controlling costs and simplifying infrastructure management. It covers the components of virtualization, illustrates the application of virtualization through various EMC products, and describes EMC s future vision of virtualization. Audience The intended audience for this white paper is the business IT professional who wants to gain a solid understanding of current virtualization technologies and a look at the future of virtualization the component-based, service-oriented virtual infrastructure. What is virtualization? Virtualization provides logical representations of physical resources while preserving the usage interfaces of those resources. Virtualization techniques can remove resource limits while improving utilization. Virtual memory and virtual networks have existed in the technology industry for years so we have included a quick recap of those technologies and their benefits to provide context for the discussion of virtual server and virtual storage technologies. Virtual memory The first computers were originally programmed via physical memory. This memory was based on individual magnetic cores, making it expensive and scarce. There was only one address zero, which was a problem because every application started at address zero, potentially limiting a computer to running one application at a time. Virtual memory broke the connection between the virtual memory addresses used by applications and the underlying physical memory addresses, as shown in Figure 1. This created more than one address zero for applications, making it possible to run multiple applications at once. It also enabled portions of applications to be paged out of physical memory to swap space, removing the limit of physical memory size on the maximum size of an application. Technology Concepts and Business Considerations 5

Figure 1. Virtual memory removes physical memory limits Today, these benefits are taken for granted. Any laptop can run multiple applications simultaneously. The amount of physical memory is no longer an absolute barrier to application size. Virtual networks A virtual network, also known as a virtual LAN or VLAN, creates independent logical networks within a single physical network. This administratively separates logical entities or portions of the network (such as company departments) that should not exchange data without requiring separate physical networks. For example, in Figure 2 there are three clients, each of which needs to be exclusively connected to its own server. But there is only one network link between the two switches. VLAN technology enables each client to share the network link while limiting communication to the appropriate servers. The result is a common link with the access control properties of separate links. Technology Concepts and Business Considerations 6

Figure 2. Client A accesses Server A through VLAN A VLANs provide a flexible and less costly way to manage logical groups in changing environments. Network managers can manage connectivity at the logical level instead of at the physical level, centrally configuring access to devices located in physically diverse locations. This class of virtualization has also been applied to storage area networks in the form of virtual fabrics or virtual SANs. As you can see, virtualization is not new and its value is easily understood. Now, let s focus on how virtualization technology can be applied to servers and storage. Server virtualization Server virtualization spans multiple technologies. IBM introduced server virtualization to the mainframe world in the 1970s with the VM operating system for IBM System/370 mainframes. More recently, UNIX multiprocessor systems have used logical partitions to provide a physical separation form of virtualization. The latest development for Intel and compatible systems is VMware virtualization of server, memory, storage, and networking resources. VMware creates virtual machines (VMs) that share physical resources, making mainframe-class virtualization available on an open systems platform. Let s review server virtualization technology on VMware ESX in more detail. Server virtualization with VMware ESX Before server virtualization, the operating system software and hardware were tightly coupled. Running multiple applications on a machine could create problems, such as DLL conflicts and registry issues. Avoiding these problems often required giving each application a dedicated physical machine resulting in underutilized resources. Server virtualization enables one server to run multiple applications even when an application requires its own server in order to function correctly. As Figure 3 illustrates, VMware ESX enables a single physical server to contain several virtual machines, each of which combines an operating system with one or more applications. This is accomplished by the Technology Concepts and Business Considerations 7

ESX Server s virtualization software layer that removes the dependencies between the operating systems and hardware. Figure 3. Server virtualization The benefits of server virtualization include strong fault and security isolation. An application or operating system can crash without affecting other operating systems on the same server or taking down the virtualization software. I/O virtualization for virtual machines When one thinks of server virtualization, one initially thinks about processors and computation. VMware virtualizes much more than that. To convince a virtual machine s OS that it has exclusive use of the virtual server hardware, VMware virtualizes the I/O each virtual machine has its own virtual I/O interfaces. For example, the network interface presented to guest operating systems is a virtual network interface card (NIC) with its own MAC address. Inside VMware ESX, an Ethernet switch is implemented in software. As far as the network is concerned, the server hardware has one Ethernet port; that port acts as a switch port with multiple virtual NICs behind it, each of which has its own MAC address. Each OS can have its own IP address, and the network operates transparently with the switch handled entirely in software. Technology Concepts and Business Considerations 8

Figure 4. I/O virtualization with VMware ESX The storage interface for virtual machines is likewise virtualized through virtual disk controllers that provide access to storage resources. Server pooling VMware ESX, in combination with VMware VMotion, also enables server pooling the aggregation of servers into a unified resource pool for provisioning virtual machines. A virtual pool supports dynamic resource scheduling, including on-the-fly movement of virtual machines without interruption to their activities. VMotion moves virtual machines from one physical server to another quickly enough that clients aren t affected when a server application is moved. VMotion knows that each operating system has its own virtual I/O interfaces. So VMotion moves the virtual machine, its operating system, and virtual I/O resources. When the virtual machine moves, its MAC address and the IP address move as well. The network understands where the addresses have moved; clients continue to execute; and the state of operations in progress is preserved. This eliminates downtime and provides continuous service while movement occurs. Along with provisioning, VMotion can balance workloads across the data center and dramatically consolidate hardware resources, often by a factor of 10 or more, even as demands for application usage change. Technology Concepts and Business Considerations 9

Figure 5 shows server and resource pooling with VMware ESX. Figure 5. Server and resource pooling with VMware ESX Server and resource pooling has multiple benefits that include: Consolidating servers Supporting consolidation of direct-attached storage into a centrally managed, highly available SAN Leveraging shared storage for high availability Migrating virtual machines online VMware in action: The data center In the data center, VMware server virtualization enables resource optimization, server consolidation, and rapid provisioning. Resource optimization VMware Distributed Resource Scheduler (DRS) is software that takes advantage of ESX Server and VMotion to dynamically allocate and balance computing capacity across a pooled collection of hardware resources. DRS continuously monitors utilization and intelligently allocates resources based on predefined Technology Concepts and Business Considerations 10

rules that support business needs and changing priorities. When a virtual machine experiences an increased load, VMware DRS automatically redistributes the virtual machines among the pool of physical servers, rebalancing the load. VMware DRS allows IT organizations to: Assign resources to the highest value applications, aligning resources with business goals Automatically optimize hardware utilization and continuously respond to changing conditions Provide dedicated (virtual) infrastructure to business units while retaining IT control over hardware Conduct zero-downtime server maintenance Server consolidation Server consolidation reduces the number of physical servers required to handle the same workload. By bundling applications and their operating systems as virtual machines, each physical server can run multiple VMs and fewer physical servers are needed. In x86 environments a 10:1 consolidation ratio is common and ratios of 20:1 and 30:1 are not unheard of. The result is a cost savings of 30 percent to 70 percent, simplified management, and improved responsiveness. Consolidation also creates latent capacity to handle unexpected changes and planned growth. The cost savings from server consolidation includes dramatic reductions in power consumption for cooling and routine operation. In at least one case, this impact caught the attention of the local electric utility, which called a VMware customer to express concern that the data center s power consumption had dropped dramatically. The customer assured the utility that everything in the data center was fine, and was pleased to receive this confirmation of the power savings. Rapid provisioning VMware ESX supports a template-based approach to rapid provisioning storing standardized VM configurations (operating system, application, and I/O) for repeat deployment such as web or database servers. When it s time to deploy a server, a wizard displays the appropriate template, the administrator selects the physical server on which the VM will run, and with a few more clicks it s up and running. We can deploy servers in a pinch. With EMC VirtualCenter, it takes 10 minutes. It s absolutely fantastic to be able to deliver a server and have the applications up, have them tested, and then put them in production in a matter of four or five hours. Robert Buchwald, Technical Lead, Systems Assurance Team, Moen Technology Concepts and Business Considerations 11

Figure 6. VMware ESX can provision servers in minutes We have reviewed server virtualization using technology from VMware ESX. Server virtualization not only enables one server to run multiple applications, it also provides benefits such as resource optimization, server consolidation, and rapid provisioning. Now let s examine storage virtualization. Storage virtualization Storage virtualization separates logical and physical storage resources in a fashion analogous to server virtualization s separation of logical and physical servers. Storage can be virtualized at the block level (SAN storage) and file level (NAS storage). Storage virtualization can extend volumes online, resolve application growth issues, and consolidate storage resources all important benefits. Nevertheless, as data stores continue to grow, migrating data without production downtime is perhaps its most compelling benefit. Data growth has reached the point where using weekends to perform data migrations is no longer adequate. For example, a typical rate for a server-based migration is 4 GB per hour with production systems down. At that rate, the maximum data migration over a weekend would be 192 GB (4 GB x 48 hrs). Of course, the true maximum depends on the number of servers moving data, but the point is that with many enterprise organizations routinely storing terabytes of information those migration rates are simply too slow. With storage virtualization, data migrations can be part of daily infrastructure optimization techniques requiring little or no downtime. There are two types of storage virtualization: block level and file level. Let s first review storage virtualization at the block level. Technology Concepts and Business Considerations 12

Block-level virtualization Block-level storage virtualization enables a storage area network (SAN) to present multiple storage arrays as a single virtual array. Mapping within the storage network redirects I/Os to the underlying physical arrays, providing nondisruptive data mobility and data migration. One product that provides block-level virtualization is EMC Invista. EMC Invista: Network-based, block-level storage virtualization Block-level storage virtualization simplifies data access for the user and provides nondisruptive data mobility to replace old storage, add additional storage, or leverage multiple tiers of storage without affecting service levels. Without virtualization, migrating storage requires storage to be taken offline; with virtualization it can remain online. Figure 7. Block-level storage virtualization with Invista As shown in Figure 7, EMC block-level storage virtualization technology relies on switches. When scaling is a concern, switches are the preferred architecture because they are designed to maximize I/O throughput and support a direct path from host to storage. Switches avoid caching data and can perform I/O at wire speed. EMC Invista is built on a high-performance, split-path architecture that leverages the processing capabilities of intelligent switches, alleviating concerns about data integrity, latency, and bandwidth. Invista handles I/O redirection at the port level, so that wire-speed performance can be achieved with negligible latency even in large enterprise deployments. Block storage virtualization that relies on general purpose processors requires the appliance or controller to store and forward I/O operations to attached storage. These systems cache data, acknowledging writes to the host prior to data being written to attached storage. In contrast, Invista processes I/O requests in real Technology Concepts and Business Considerations 13

time and acknowledges writes only after data has been written to the attached storage. This provides an end-to-end check that the data has been correctly written to the storage. Invista allows servers to access data from anywhere on the SAN by presenting virtual storage resources that are abstracted from physical storage devices. Invista is based on intelligent Fibre Channel switches with custom hardware for enhanced processing, capable of handling data operations at line speed (Figure 8). An intelligent switch looks inside the I/O to examine and change the destination port, logical volume, and offset. This is transparent to hosts, so the first principle of Invista block-level storage virtualization is that the intelligent switch becomes the storage target. Figure 8. Invista intelligent switches handle data operations at line speed The host obtains storage resources from the intelligent switch, with no knowledge of the I/O mapping operations and redirection that occur behind the scenes. When a host requests I/O from a virtual storage resource, it is unaware of the physical location at which the data resides. The I/O data request might access EMC Symmetrix, HDS, or IBM storage. The physical-volume location or composition can change without any impact to the virtual volume seen by the host. The second important principle of switch-based, block-level storage virtualization is the separation of data and control (Figure 9). With Invista, there is a highly available control path cluster that makes changes to storage allocation, issues commands for the migrations, and handles uncommon I/O operations such as SCSI inquiries. In the event that a new storage array is added, the control path discovers the new array and with input from the network administrator issues instructions to the switch that updates I/O mapping. The control path cluster can be separated within the data center to provide tolerance for local disruptions such as partial power outages, leaking pipes, or minor storm damage. Technology Concepts and Business Considerations 14

Figure 9. Invista separates data and control Invista provides two important functional capabilities that enable mapping and redirection. The first is network-based volume management, which provides a single point of control for creation and deletion of and changes to storage volumes. The resulting centralized management and control simplify operations and provide better insight into host utilization of storage resources, regardless of the storage vendor. Networkbased volume management also provides front-end LUN masking and mapping of storage volume to hosts. The second capability is volume mobility or data migration, as shown in Figure 10. Migration is performed by mirroring data to its new location but only releasing the original location for reuse after the new location has all the data. This preserves data access in the event migration is interrupted. Migration can be performed while I/O is in progress this reduces application downtime and improves the ability to meet service level agreements (SLAs) and match storage resources to business requirements. Volumes can be moved across heterogeneous storage systems, while the host views Invista as a large array, unaware of the physical address for the volumes it accesses. Technology Concepts and Business Considerations 15

Figure 10. Data mirroring supports volume mobility We have just reviewed block-level storage virtualization using EMC Invista as an example. Next, we ll look at file-level virtualization. File-level storage virtualization File-level virtualization applies the general principles of virtualization to file storage, specifically networkattached storage (NAS). Without file-level virtualization, each NAS storage device and file server is physically and logically isolated, with the result that files are bound to a specific file server. This causes capacity problems and underutilized storage. Migrating files to deal with these issues entails downtime because hosts, applications, and storage devices must be reconfigured to use new filenames and server locations. When file-level virtualization is deployed, the dependencies are broken between data and its physical location. File-level virtualization optimizes storage utilization, enables nondisruptive migrations, and supports file sharing across multiple organizational boundaries. Figure 11 compares NAS devices before and after file virtualization. Technology Concepts and Business Considerations 16

Figure 11. Before and after file-level virtualization We ll use EMC Rainfinity to illustrate the benefits of file-level virtualization.. EMC Rainfinity: Moving files online with global file virtualization With EMC Rainfinity Global File Virtualization, file systems can be moved online and namespaces reorganized without disruption to clients. Because clients are insulated from the locations of files, they have continuous access to their files while they are being moved. Files can be read from the old location and written to the new location without the client realizing that the physical location has changed. In contrast, when each client has direct knowledge of file locations, it is difficult to move files to other NAS systems. Figure 12 illustrates the benefits of global file virtualization with Rainfinity. Technology Concepts and Business Considerations 17

Figure 12. File-level virtualization with Rainfinity Rainfinity is inactive until a file needs to be moved. A typical file migration is illustrated in Figure 13. Rainfinity is moving the blue file system from the yellow file server to the orange file server. During the migration, the Rainfinity appliance is triggered and inserts itself into the I/O stream, making the target (yellow) file server visible only through the appliance as a network bridge. Clients retain access to files, but all traffic passes through and is controlled by Rainfinity so that it can move the file system to a new location (orange file server). Technology Concepts and Business Considerations 18

Figure 13. Rainfinity Global File Virtualization Rainfinity uses data mirroring to ensure that all files remain accessible while migration occurs. Initially after the move, all of the I/O continues through Rainfinity and only Rainfinity knows the new location of the file system. Rainfinity Global Namespace Manager then updates the client namespaces, letting clients know the new location of the file system. As clients learn the new location, their I/O goes directly to the new location and Rainfinity is removed from the I/O path when all clients are updated. Global namespace management When there are millions of files and hundreds or thousands of clients, manual namespace updates can be risky. For example, if there are 1,000 clients to update over a weekend and 95 percent are successful, that still means 50 mistakes and 50 employees who can t access their files. The resulting problems could easily consume Monday and Tuesday with help desk calls. On Wednesday, the planning begins for the next set of migrations. With Rainfinity, global namespace management automates these updates and eliminates this scenario, enabling migrations to be performed during normal working hours. The future of virtualization So far, we ve looked briefly at memory and network virtualization, and provided a more detailed view of server and storage virtualization through VMware ESX Server, EMC Invista, and EMC Rainfinity. While each technology independently delivers substantial benefits, bringing the technologies together under a single virtualization management structure provides additional value. Now, we ll examine the potential benefits of combining server and storage virtualization in a data center within a service-oriented infrastructure and how that may influence the future direction of virtualization. Technology Concepts and Business Considerations 19

At EMC, we believe that information technology infrastructure is moving in a service-oriented direction where the entire stack from storage to applications is loosely coupled to provide an architectural focus on functionality. To put these changes into context, it s helpful to examine how IT has arrived at this point. The first generation of information technology was based on monolithic computing: purpose-built, turn-key applications tightly integrated from the user interface to the data residing on hardware. While some airline reservation and financial transaction systems still employ this approach, it is now the exception. Figure 14. Tightly integrated purpose-built applications Today, most systems are built on a tiered infrastructure where the client, server, applications, and storage are combined into a functional stack. The various layers are separated and for the most part can be changed independently in order to use best-of-breed technology. But the downside of this independence is that applications have to cope with an increasing variety of platform configurations. That s why integration and maintenance have become a large part of what IT does. Figure 15. Second-generation tiered infrastructure Technology Concepts and Business Considerations 20

In Figure 15, the horizontal enterprise application integration (EAI) is the only access across the tiered implementation stacks. There is little, if any, integration in the lower layers. This is a target of opportunity where EMC foresees the deployment of server, network, and storage virtualization across the infrastructure to make it more efficient and eliminate downtime. The EMC vision: A component-based infrastructure The creation of this next-generation infrastructure starts with the separation of services from the underlying hardware on which they reside, as shown in Figure 16. This creates standard service interfaces across the IT stack. Figure 16. Flattening the IT stack Having made that separation, virtualization allows hardware resources to be grouped in pools of capacity, computation, and connectivity that are coherently managed across an entire infrastructure. When the entire infrastructure is virtualized, significant changes can be handled online without disruption to servers, networks, and storage. In contrast, individual point virtualization technologies leave the infrastructure vulnerable to downtime from components that are not virtualized. For example, consider application provisioning when only the servers in the IT stack are virtualized, as shown in Figure 17. The first step includes installing operating systems, authenticating users or applications, and allocating server resources. This can be done through the virtual server infrastructure while servers remain online an obvious benefit. Technology Concepts and Business Considerations 21

Figure 17. Application provisioning with only servers virtualized Nevertheless, when provisioning an application, there is much more to the task than working within the server domain. There may be network changes, and storage changes are certain. Server provisioning enables us to do the server portion of the work online, but changes to the network and storage may force those systems offline. On the other hand, virtualization of all resources enables dynamic online operation, as shown in Figure 18. Figure 18. A completely virtualized infrastructure Technology Concepts and Business Considerations 22

Managing a virtual infrastructure But now the question is how to manage this virtual infrastructure. EMC envisions management and orchestration through an abstract model that enables end-to-end dynamic provisioning, load balancing, and optimization. A useful analogy for this abstract model is an automobile, a complex system that (like IT infrastructure) has multiple subsystems that need to be consistently managed. The car s primary subsystems engine, transmission, brakes, steering, and their many supporting components have to be coordinated effectively for the car to function. Figure 19. The concept of a car is an abstract model for a complex system Regardless of this complexity, almost everyone can drive a car. The underlying complexity is transparent to the driver because the concept of car is a common, abstract model that has only a few important elements. The gas and brake pedals, the gear shift, the steering wheel, the ignition these are five control elements that if understood can help one drive almost any car. The details of what s under the hood don t matter. The common model defines how the car responds when the control elements are manipulated. And that s all it takes. This common, abstract model is supported by a functional model that includes detailed information about how all the components of a car work. A mechanic understands both models, making it possible for him to understand a driver s description of a problem every time I put my foot on the brake, it squeaks and fix the component that s causing it. The driver and the mechanic use the same frame of reference to communicate: the common abstract model. Technology Concepts and Business Considerations 23

Figure 20. An abstract model provides a uniform frame of reference This common abstract model describes how EMC envisions infrastructure will be managed in the future. Just like the driver and the mechanic, administrators will have the same frame of reference to manage the environment. For example, in the server environment, a server will exhibit the same basic behavior when a storage administrator is provisioning it for storage or the network admin is setting up network connectivity. But beyond that, each of these administrators is an expert. They understand functional models for their particular areas server, storage, network and they have tools with task-specific interfaces to the infrastructure based on the common abstract model and the area s supporting functional model. The result shown in Figure 20 is a consistent set of tools that provides complete coverage of the IT infrastructure. Conclusion In EMC s vision of the future, all IT resources and their management will be virtualized based on a common abstract model and a supporting functional model. Over time, EMC and its partners will deliver a complete set of tools to realize this orchestrated environment. Today, virtualization is an important strategic decision with benefits that increase with the extent of deployment. Nevertheless, it s important to keep in mind that virtualization can be incrementally deployed. Start with your primary pain point, whether it s server, network, or storage issues. Finally, pick your partners wisely, as virtualization is a technology for the long haul. To learn about EMC virtualization solutions, please visit us at http://www.emc.com/products/category/virtualization.htm Technology Concepts and Business Considerations 24