Advanced Computer Networks. Network I/O Virtualization

Transcription

1 Advanced Computer Networks Network I/O Virtualization Patrick Stuedi Spring Semester 2014 Oriana Riva, Department of Computer Science ETH Zürich 1

2 Outline Last week: Today: Software Defined Networking OpenFlow Network I/O Virtualization Paravirtualization SR-IOV 2

3 Data Transfer Non-Virtualized syscall OS TCP/IP Driver privileged instructions NIC 1) : syscall, e.g., socket.write() 2) OS driver: issue PCI commands Set up DMA operation 3) NIC: transmit data raise interrupt when done 3

4 Virtualization and Hypervisors VM1 VM2 VM3 Hypervisor Hardware 4

5 Virtualization and Hypervisors VM1 VM2 VM3 How does Hypervisor network access work? Hardware 5

6 Option 1: Full Device Emulation OS unaware that it is being virtualized Hypervisor emulates device at the lowest level Privileged instructions from guest driver trap into hypervisor Advantage: no changes to the guest OS required Disadvantage: Inefficient Complex OS Traps Device Emulation Hypervisor Hardware 6

7 Option 1: Paravirtualization OS aware that it is being virtualized Runs special paravirtual device drivers Hypervisor cooperates with guest OS through paravirtual interfaces Advantage: Better performance Simple Disadvantage: Requires changes to the guest OS OS Paravirtual Driver Interfaces Device Emulation Hypervisor Hardware 7

8 Paravirtualization with VirtIO OS Front-end drivers virtio Back-end drivers Device Emulation KVM Hypervisor Back-end drivers Device Emulation lguest Hypervisor Hardware VirtIO: I/O virtualization framework for Linux Framework for developing paravirtual drivers Split driver model: front-end and back-end driver APIs for front-end and back-end to communicate 8

9 Example: KVM Hypervisor VM1 guest mode (guest user, guest kernel) Single Linux Process Based on Intel VT-x Additional guest execution mode I/O at guest OS trap into KVM (VM Exit) KVM schedules QEMU process to emulate I/O operation user mode QEMU VM Enter & VM Exit KVM module Hypervisor User user mode kernel mode /dev/kvm Hardware Starting new guest = starting QEMU process QEMU process interacts with KVM through ioctl on /dev/kvm to Allocated memory for guest Start guest 9...

10 VirtIO and KVM QEMU VirtIO Back-end 1 tx rx OS VirtIO-Net Driver (Front-end) VirtIO shared memory 3 KVM module 2 4 Hypervisor tap Real NIC 1) VirtIO-Net driver adds packet to shared VirtIO memory 2) VirtIO-Net driver causes trap into KVM 3) KVM schedules QEMU VirtIO Back-end 4) VirtIO back-end gets packet from shared VirtIO memory and emulates I/O (via system call) 5) KVM resumes guest 10

11 Vhost: Improved VirtIO Backend OS QEMU VirtIO-Net Driver tx rx KVM module Hypervisor vhost net tap Real NIC Vhost puts VirtIO emulation code into the kernel Instead of performing system calls from userspace (QEMU) 11

12 Inter-VM communication VM1 VM2 VM3 Hypervisor NIC 12

13 Inter-VM communication VM1 VM2 VM3 Hypervisor How does inter-vm Hardware communication work? 13

14 Switch in Hypervisor Bridge/switch NIC HV 14

15 Switched Vhost in KVM Advantage: low latency (1 software copy) OS OS OS Disadvantage: uses host CPU cycles VirtIO-Net Driver VirtIO-Net Driver VirtIO-Net Driver vhost net vhost net vhost net KVM tap tap tap bridge Real NIC 15

16 Switch Externally......either in External switch: Simplifies configuration: all switching controlled/configured by the network Latency = 2xDMA + 2hops Hypervisor NIC NIC Latency = 2xDMA 16

17 Where are we? Option 1: Full emulation No changes to guest required Complex Inefficient Option 2: Paravirtualization Requires special guest drivers Enhanced performance Option 3: Passthrough No hypervisor involvement: best performance Problems: 17

18 Where are we? Option 1: Full emulation No changes to guest required Complex Inefficient Option 2: Paravirtualization Requires special guest drivers Enhanced performance Not good enough! Still requires hypervisor involvement, e.g., interrupt relaying Option 3: Passthrough No hypervisor involvement: best performance Problems: 18

19 Where are we? Option 1: Full emulation No changes to guest required Complex Inefficient Option 2: Paravirtualization Requires special guest drivers Enhanced performance Not good enough! Still requires hypervisor involvement, e.g., interrupt relaying Option 3: Passthrough Directly assign NIC to VM No hypervisor involvement: best performance 19

20 Paravirtual vs Passthrough in KVM VM1 OS VM2 OS VirtIO-Net Driver Physical Driver tx rx KVM module Hypervisor vhost net tap NIC exclusively assigned to VM2 Real NIC Real NIC 20

21 Challenges with Passthrough / Direct Assignment VM tied to specific NIC hardware Makes VM migration more difficult VM driver issues DMA requests using VM addresses Incorrect: VM physical addresses are host virtual addresses (!) Security concern: addresses may belong to other VM Potential solution: let VM translate it's physical addresses to real DMA addresses - Still safety problem: exposes driver details to hypervisor, bugs in driver could result in incorrect translations Solution: Use an IOMMU to translate/validate DMA requests from the device Need a different NIC for each VM Solution: SR-IOV, emulate multiple NICs at hardware level 21

22 Challenges with Passthrough / Direct Assignment VM tied to specific NIC hardware Makes VM migration more difficult VM driver issues DMA requests using VM addresses Incorrect: VM physical addresses are host virtual addresses (!) Security concern: addresses may belong to other VM Potential solution: let VM translate it's physical addresses to real DMA addresses - Still safety problem: exposes driver details to hypervisor, bugs in driver could result in incorrect translations Solution: Use an IOMMU to translate/validate DMA requests from the device Need a different NIC for each VM Solution: SR-IOV, emulate multiple NICs at hardware level 22

23 Memory Address Terminology Virtual Address Address in some virtual address space in a process running in the guest OS Physical Address: Hardware address as seen by the guest OS, i.e., physical address in the virtual machine Machine address: Real hardware address on the physical machine as seen by the Hypervisor 23

24 IOMMU IOMMU Memory controller Main memory PCIe function (e.g. NIC) 24

25 IOMMU VMM programs IOMMU with VM-physical to machine address translations IOMMU Memory controller Main memory PCIe function (e.g. NIC) 25

26 IOMMU VMM programs IOMMU with VM-physical to machine address translations IOMMU Memory controller Main memory OS programs NIC with VMphysical address of DMA PCIe function (e.g. NIC) 26

27 IOMMU VMM programs IOMMU with VM-physical to machine address translations IOMMU Memory controller Main memory OS programs NIC with VMphysical address of DMA NIC issues a DMA request to VM physical memory PCIe function (e.g. NIC) 27

28 IOMMU IOMMU checks and translates to machine (real) address for transfer VMM programs IOMMU with VM-physical to machine address translations IOMMU Memory controller Main memory OS programs NIC with VMphysical address of DMA NIC issues a DMA request to VM physical memory PCIe function (e.g. NIC) 28

29 IOMMU IOMMU checks and translates to machine (real) address for transfer VMM programs IOMMU with VM-physical to machine address translations IOMMU Memory controller Main memory OS programs NIC with VMphysical address of DMA NIC issues a DMA request to VM physical memory Memory controller accesses memory PCIe function (e.g. NIC) 29

30 SR-IOV Single-Root I/O Virtualization Key idea: dynamically create new PCI devices Physical Function (PF): original device, full functionality Virtual Function (VF): extra device, limited functionality VFs created/destroyed via PF registers For Networking: Partitions a network card's resources With direct assignment can implement passthrough 30

31 SR-IOV in Action OS Physical Driver OS Physical Driver OS Physical Driver Hypervisor Physical Driver Virtual Function SR-IOV NIC Virtual Function IOMMU PCI Virtual Function Virtual Ethernet Bridge/Switch Physical Function 31

32 References I/O Virtualization, Mendel Rosenblum, ACM Queue, January 2012 Kernel-based Virtual Machine Technology, Yasunori Goto, Fujitsu Technical Journal, July 2011 VirtIO: Towards a De-Facto Standard For Virtual I/O Devices, Rusty Russel, ACM SIGOPS Operating s Review, July