June 2012 Toward Exitless and Efficient Paravirtual I/O IBM Research Haifa Abel Gordon, Nadav Har'El, Alex Landau, Muli Ben-Yehuda, Avishay Traeger 2009 IBM Corporation
I/O virtualization models In ELI: Bare-metal Performance for I/O Virtualization we showed how to reach near-optimal (bare-metal) performance for I/O on VMs. But, this assumed the host can assign a device to the guest. Device assignment best-performing I/O virtualization option, but Needs a physical device E.g., cannot assign a virtual disk backed by host file. Host gives up control E.g., cannot process packets going in/out of guest (switch, firewall, etc.). More expensive hardware IOMMU, and to assign one device to multiple guests SRIOV devices. Prevents or complicates VM migration Prevents or complicates memory overcommitment In many cases, paravirtual I/O is used instead of device assignment: 2 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
The research question Emulated I/O: Host emulates a familiar device. E.g., Intel E1000 network card. Guest OS uses its standard driver for this card. Paravirtual I/O: Host emulates a new device. E.g., new type of NIC. Device optimized for more efficient guest-host communication. Guest aware of hypervisor, and uses a new driver hence paravirtual. In many cases, paravirtual I/O is used instead of device assignment Can we also improve paravirtual I/O performance? 3 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Paravirtual I/O implementation, and exits Remember that ELI improved performance by avoiding exits. What exits are there in paravirtual I/O? 4 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Paravirtual I/O implementation, and exits Remember that ELI improved performance by avoiding exits. What exits are there in paravirtual I/O? Guest-to-host Notifications (typically PIO) Host-to-guest Notifications (typically inter-processor interrupts IPIs) We propose ELVIS Exit-Less Virtual I/o System: Continue to use spatial division of cores (guest core, I/O core) Avoid exits caused by the two types of notifications 5 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Host-to-guest notifications Host's I/O core sends an Inter-Processor Interrupt (IPI) to guest core. Normally, this IPI causes an exit from the guest to host, and immediately entry (with virtual interrupt) back to guest. Previous work [BenYehuda12] used guest-side polling to avoid exits. Problems: Overcommitment, power, guest modification. Using ELI, we can cause the IPI to directly deliver the virtual interrupt into the guest, without exit. 6 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Guest-to-host notifications Guest puts new I/O request(s) in shared memory buffers. Normally, it then notifies host of new requests with exit-causing instruction, e.g., PIO. Instead, since we already have a dedicated I/O core, it can poll the shared memory buffer, and the guest doesn't need to exit. 7 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
I/O-Core polling Dedicated host I/O core polls for new work from the guest, replacing guest-to-host notifications. [BenYehuda12, Liu09] I/O core also handles the I/O, of course (not just polls). Usually we don't need a full I/O core for each VM One I/O core polls multiple VMs Consider fairness and performance (latency and throughput) For non I/O-intensive workloads, excessive polling may outweigh benefits of exit-less notification Dynamically switch between polling and traditional exit-based guest-tohost notifications. [Mogul97, Salim01, VMWare05] 8 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Exits with and without ELVIS 9 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Proof of concept Initial implementation On KVM hypervisor (open source, part of Linux kernel) Used ELI patches to KVM with new code for delivering virtual interrupts with IPIs and ELI. Modified vhost, an in-kernel implementation of the virtio paravirtual I/O framework Single thread to handle multiple devices (VMs) for better I/O scheduling Poll for new guest work, instead of guest-to-host notifications. 10 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Preliminary results We tested ELVIS's impact on paravirtual network I/O (vhost-net). Test machine with 8 cores, 10 Gbps NIC And another similar machine directly connected. Measured throughput and latency: 11 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Preliminary throughput results Measured Netperf TCP-stream To verify that ELVIS can serve multiple VMs using a single I/O core, ran multiple VMs (1..7). Each VM opens single TCP connection to the remote machine. Repeatedly write()s 64 bytes. 64 was chosen to saturate test machine's CPUs (not the 10Gbps line) so throughput can be compared. 12 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Preliminary throughput results With one VM, ELVIS improved throughput by 45% The single vhost-net I/O core (used by both vanilla vhost-net and ELVIS) saturated at 3100 Mbps, leading to the plateau for 2 VMs and more. This would not be a problem for less I/O intensive workloads. Can be avoided with more than one I/O core. With further research, I/O capacity of single I/O core can be increased. 13 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Preliminary throughput results As expected, ELVIS eliminated most exits: 1 VM: from 120,000 exits/sec to 800 7 VMs: from 32,000 exits/sec to 800 (per VM) Most remaining exits unrelated to I/O (e.g., 500 related to timer interrupts) 14 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Preliminary latency results Measured Netperf UDP Request-Response Again, 1 7 VMs, using one I/O core. Each VM sends a UDP packet, and waits for a reply before sending the next. Latency reduced by 22-25 microsec - Exitless notifications Single vhost thread I/O core not saturated 15 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Summary and future work We proposed ELVIS, a new model for exit-less paravirtual I/O. Our prototype showed good potential: 45% improvement in throughput. Unharmed (and even improved) latency. Need further research to be efficient for more multi-vm workloads: Increase the number of VMs that a single I/O core can handle. Dynamically create and destroy I/O threads depending on workload. Dynamically switch between polling and exit-based notifications. Support a mixture of latency and throughput sensitive VMs. Also need to: Demonstrate ELVIS potential for other paravirtual I/O devices, viz. Disk. 16 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation
Th-Th-Th-That's all folks! Questions? 17 June 2012 Towards Exitless and Efficient Paravirtual I/O 2012 IBM Corporation