White Paper Kelvin Lum Fee Foon Kong Platform Application Engineer, ECG Penang Intel Corporation Kam Boon Hee (Thomas) Marketing Development Manager, ECG Penang Intel Corporation Intel Embedded Virtualization Manager High Availability Single Embedded System with Intel Virtualization Technology November 2011 326438
Executive Summary Virtualization is commonly used to consolidate the software running on different systems onto a single system in order to reduce the cost, size and power consumption. This white paper introduces a new usage model enabled by virtualization that achieves high availability (HA) on a single embedded system. HA is a major requirement in many embedded market segments, such as digital signage, retail and industrial automation, where systems are expected to run 24/7 without fail. The return on investment for highly-reliable systems is even greater for unmanned systems in remote areas, where onsite repair visits can be very expensive. A key ingredient of this HA solution is the Intel Embedded Virtualization Manager, which provides a management daemon (e.g., computer program running in the background) on the host operating system (OS) that monitors the health and status of two identical virtual machines (VMs). If the primary operating VM hangs, the management daemon immediately initiates a failover procedure by activating a backup VM to resume system operation. A variation to this approach involves a single VM, which can be rebooted in the event of a failure without having to reboot the entire system. In both cases, system recovery can be accomplished without human intervention or a network connection. This paper explains how developers can implement a virtualization-based HA solution and provides a list of applicable design resources. 2
Virtualization in Embedded Systems Virtualization has been around for many years, most notably used in data centers where many applications are consolidated onto a single server. For example, the combination of robust virtualization technology and high performance Intel processors has enabled companies to run the software previously run on fifteen servers all on a single server platform. 1 Although the adoption of virtualization in embedded applications is still in its infancy, some developers are already taking advantage of the technology to: simplify the porting of legacy applications onto new platforms increase the determinism of time-critical functions improve the security and stability of safety-critical code optimize the processor utilization in a system Once an embedded system is virtualized, multiple usage models, including the HA solution covered in this paper, can co-exist on the same platform. Hardware-Assisted Virtualization In recent years, Intel has developed different versions of Intel Virtualization Technology (Intel VT) 2 to improve the fundamental flexibility and robustness of software-based virtualization solutions: Intel Virtualization Technology for IA-32, Intel 64 and Intel Architecture (Intel VT-x) speeds up the transfer of platform control between the virtual machine manager (VMM) and guest operating systems (OSes) by using hardware-assist to trap and execute certain instructions on behalf of guest OSes, relieving the VMM of such duties. These commonly used virtualization operations are very secure because they are performed in hardware and thus unalterable by hackers. Intel Virtualization Technology for Directed I/O (Intel VT-d) enables the VMM to securely assign specific I/O devices to specific guest OSes, where each device is given a dedicated area in system memory accessible only by the device and the designated guest OS. Hardware assistance speeds up data movement and lowers VMM activity, hence processor load, because the VMM is no longer involved in every I/O transaction. Additional background information about virtualization can be found at http://www.intel.com/technology/advanced_comm/322288.pdf. 3
Intel VT is available on computing platforms with Intel vpro technology, which also incorporates a number of other technologies, including: Intel Active Management Technology (Intel AMT) 3 offers breakthrough remote management and energy saving capabilities Intel Trusted Execution Technology (Intel TXT) 4 supplies additional security protection over software solutions Integrated within many embedded Intel processors and chipsets, Intel vpro technology delivers unprecedented hardware support for management functions, security protection and software integration enabling solutions that are reliable, trusted and cost-effective. High Availability Solution For Embedded The key objective of the Intel high availability embedded solution (Figure 1) is to provide developers with a cost effective and straightforward way to implement a software failover mechanism. The solution uses the royalty-free Xen* 4.0 and KVM (Kernel-based Virtual Machine for Linux*) hypervisors (aka, virtual machine monitor) running on an Intel vpro technology-based platform with Intel VT-x and Intel VT-d enabled. The installation of the Host OS, Xen hypervisor and the Intel Embedded Virtualization Manager takes about an hour. Note: Xen is used for VGA pass-through mode and KVM for non-vga pass-through mode. Figure 1-Figure 3 depict the VGA pass-through mode case. 4
Figure 1. High Availability Solution Most open source hypervisor managers/management software support simple functions such as creating, destroying, powering on, powering off and resetting the VMs. Going beyond this functionality, the Intel Embedded Virtualization Manager provides high availability features, including heart beat monitoring, self-rebuild of the corrupted VM, and system failover, as well as being easy to setup and use. The Intel Embedded Virtualization Manager supports two failover approaches, either with two VMs or one VM. Neither failover approach requires human intervention or a network connection. The two VM alternative is most suitable when using open source OSes, like Linux, which does not incur software license fees. In the event of license fees, the single VM alternative saves cost because just one OS instance is required. Video Graphic Array (VGA) Implications In Figure 1, the VMs communicate with a video graphic array (VGA) display via pass-through mode, which is supported by the Xen hypervisor. VGA passthrough is essential for many embedded segments that require high quality graphics and video performance, such as digital signage devices playing high definition (HD) video. Without VGA pass-through mode, virtualization would likely compromise graphics and video performance. 5
Although the graphics performance with VGA pass-through mode is superb, a drawback is the backup VM must be in shutdown mode (instead of standby), and as a result, the failover time is about a minute. The alternative is to use non-vga-pass-through mode and the KVM hypervisor, which allow the backup VM to be in standby mode; therefore, the resume time is much faster within seconds whenever there is a failover. When VGA pass-through mode is enabled and no network connection is available, the user can view the primary VM from the local display, but not the Intel Embedded Virtualization Manager that reside on the host system. However, with a network connection, the system administrator can configure and control VMs via the Intel Embedded Virtualization Manager VM Manager running on a remote machine. For the non-vga pass-through mode, the user can install the Intel VM Manager on the target platform where the VMs reside. The Intel Embedded Virtualization Manager can also run on the Kernel-based Virtual Machine (KVM) hypervisor from www.linux-kvm.org. At the time this paper was completed, the KVM hypervisor delivered better screen resolution under non-vga pass-through mode than with VGA pass-through mode. There are slight differences in the way Intel Embedded Virtualization Manager manages the system in non-vga pass-through and VGA pass-through modes, which is clarified in the following section. Implementation with two virtual machines: The system implements two identical VMs, meaning they contain the same guest OS and application software. One is the primary (operating) VM, and the other is the backup VM. If the primary VM experiences system failure, the management daemon in the Intel Embedded Virtualization Manager will recover the system by activating the backup VM, which resumes system operation. 1. VGA-pass-through mode: The Intel Embedded Virtualization Manager monitors the heartbeat of the primary VM, but not the backup VM, which is in shutdown mode. The status of the backup VM can be checked using the Intel Embedded Virtualization Manager client GUI dashboard. 2. Non-VGA pass-through: The Intel Embedded Virtualization Manager ensures the backup VM is in paused state, since it is possible for the two VMs to run concurrently without the VGA pass-through feature. The sequence of events is depicted in Figure 2 and described in the following: 6
Figure 2. High Availability Scenario Using Two VMs 1. During normal operation, the primary VM is running and the Intel Embedded Virtualization Manager management daemon monitors the heartbeat of its guest OS. 2. If the primary VM fails, the heartbeat from its guest OS will stop, and the management daemon will shutdown the OS of the failing VM. Immediately following, the management daemon starts the backup VM, which becomes the new primary VM (Figure 2, left side). a. VGA-pass-through mode: In an attempt to recover the system, the failing VM is placed in shutdown mode and becomes the new backup VM. If there is another failover event, it will be rebooted to recover the system, and it becomes the new primary VM. b. Non-VGA pass-through: The Intel Embedded Virtualization Manager reboots and pauses the failing VM, which then becomes the new backup VM while the former backup VM is running. c. If the Intel Embedded Virtualization Manager detects the failing VM is not operating normally after a period of time following a reboot (failed to boot to the OS), then it is considered to have a corrupted image, and the management daemon will delete the image. The image selfrebuild mechanism in the Intel Embedded Virtualization Manager then rebuilds a new VM based on the gold software image, and the repaired VM is set as a new backup VM. This process eliminates the 7
need for manual reinstallation of the OS and application software (Figure 2, right side). Implementation with one virtual machine: In the single VM scenario, there is only a primary VM, which the Intel Embedded Virtualization Manager closely monitors. If the VM fails, the Intel Embedded Virtualization Manager attempts to reboot the VM, as illustrated in Figure 3. Figure 3. High Availability Scenario Using One VM Managing the Virtualization Environment In addition to the previously discussed virtualization management daemon, the Intel Embedded Virtualization Manager includes a VM Manager, which is a subset of the client graphical user interface (GUI) application. It enables the user to view, create, configure and control VMs from a remote machine when VGA pass-through mode is enabled, using the screen shown in Figure 4. For more information about the VM Manager, please refer to the User Guide that accompanies the Intel Embedded Virtualization Manager VM Manager Installation Package. 8
Figure 4. Client Graphical User Interface (GUI) Screenshot The client GUI, which contains the VM Manager, resides on a remote management console that connects to the virtualized target platform via Ethernet, as shown in Figure 5. In this case, the virtualized target platform, which incorporates the high availability feature, is managing a digital signage display. The VM manager can also reside on the target platform for non-vga pass-through mode. Figure 5. Network Example 9
Setup Tools The Intel Embedded Virtualization Manager setup first requires the installation of the Host OS, the Xen/KVM hypervisors and the Intel Embedded Virtualization Manager. Subsequently, the Intel Embedded Virtualization Manager Installation Wizard, shown in Figure 6, guides the user through the guest OS setup. If the user selects VGA pass-through mode, the wizard gives the VMs direct access to the VGA display. For two VM implementations, the wizard assists the user in creating a gold image of the VM, which contains a known good configuration with the OS, application software and installed drivers. The gold image is stored by the Intel Embedded Virtualization Manager and is used to rebuild a corrupted VM, if necessary. Lastly, the client GUI, which manages the system, is installed in one of two ways: 1. On a remote system for VGA pass-through mode, or 2. On the target system for non-vga pass-through mode. Figure 6. Installation Wizard Screenshot For the two VM approach, the installation wizard creates both a primary and backup VM using the gold image, whereas for the single VM approach, just a primary VM is created (i.e., there s no gold image). Applicable Design Resources For access to the Intel Embedded Virtualization Manager, please contact your Intel field sales representative. For technical information about Intel processors, please visit the Intel Embedded Design Center at http://intel.com/embedded/edc. The website 10
provides access to technical resources, Intel confidential design materials, step-by-step product selection guides, application reference solutions, training, Intel s tool loaner program, an e-help desk and the Intel embedded community. Additional links: Intel vpro technology: www.intel.com/p/en_us/embedded/hwsw/technology/vpro Intel VT: www.intel.com/p/en_us/embedded/hwsw/technology/virtualization Authors Kelvin Lum is a Platform Application Engineer with Intel ECG. Fee Foon Kong is a Platform Application Engineer with Intel ECG. Acronyms AMT ATM GUI HA HD Intel EVM KVM OS ROI TCO VGA VM VT Active Management Technology Automated Teller Machine Graphical User Interface High Availability High Definition Intel Embedded Virtualization Manager Kernel-based Virtual Machine Operating System Return On Investment Total Cost of Ownership Video Graphic Array Virtual Machine Virtualization Technology 11
1 Source: http://www.intel.org/my/roadmap/pdf/5intelprocessorserver%20selection%20guide.pdf 2 Intel Virtualization Technology (Intel VT) requires a computer system with an enabled Intel processor, BIOS, virtual machine monitor (VMM), and for some uses, certain platform software enabled for it. Functionality, performance, or other benefits will vary depending on hardware and software configurations and may require a BIOS update. Software applications may not be compatible with all operating systems. Please check with your application vendor. 3 Intel Active Management Technology (Intel AMT) requires the platform to have an Intel AMT-enabled chipset, network hardware and software, as well as connection with a power source and a corporate network connection. With regards to notebooks, Intel AMT may not be available or certain capabilities may be limited over a host OS-based VPN or when connecting wirelessly, on battery power, sleeping, hibernating or powered off. For more information, see http://www.intel.com/technology/manage/iamt. 4 No computer system can provide absolute security under all conditions. Intel Trusted Execution Technology (Intel TXT) requires a computer system with Intel VT, an Intel TXTenabled processor, chipset, BIOS, Authenticated Code Modules and an Intel TXT-compatible measured launched environment (MLE). The MLE could consist of a virtual machine monitor, an OS or an application. In addition, Intel TXT requires the system to contain a TPM v1.2, as defined by the Trusted Computing Group and specific software for some uses. For more information, see http://www.intel.com/technology/security. INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR. Intel may make changes to specifications and product descriptions at any time, without notice. This paper is for informational purposes only. THIS DOCUMENT IS PROVIDED "AS IS" WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY WARRANTY OF MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR ANY PARTICULAR PURPOSE, OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION OR SAMPLE. Intel disclaims all liability, including liability for infringement of any proprietary rights, relating to use of information in this specification. No license, express or implied, by estoppel or otherwise, to any intellectual property rights is granted herein. Copyright 2011 Intel Corporation. All rights reserved. Intel, the Intel logo and Intel vpro are trademarks of Intel Corporation in the United States and/or other countries. *Other names and brands may be claimed as the property of others. 12