Outline CS 6V81-05: System Security and Malicious Code Analysis Overview of System ization: The most powerful platform for program analysis and system security Zhiqiang Lin Department of Computer Science University of Texas at Dallas April 4 th, 2012 1 Background 2 ization Types 3 Key Techniques 4 Summary Outline Why virtualization? Why not virtualize? 1 Background 2 ization Types Today s data center Cloud computing 3 Key Techniques 4 Summary resource pool Fault resist Cost Effect Easy to maintain
What is ization What is ization ization Properties Linux Linux (devel) XP Vista MacOS Machine Monitor Isolation Encapsulation Interposition Isolation Encapsulation Fault Isolation Fundamental property of virtualization Performance Isolation Accomplished through scheduling and resource allocation All state can be captured into a file Operate on by operating on file mv, cp, rm Complexity Proportional to virtual HW model Independent of guest software configuration
Interposition Why Not the OS? All guest actions go through monitor Monitor can inspect, modify, deny operations Example: Compression Encryption Profiling Translation It about interfaces Ms operate at the hardware interface interface are typically smaller, better defined than software interfaces Disadvantages of being in the monitor Low visibility into what the guest is doing ization benefits ization: server consolidation ization: server consolidation Increased resource utilization: Mobility Server consolidation Multiplexing Enhanced Security Test and Deployment... Physical Servers Server Host Machines s
Mobility: load balance (Migration) Mobility: load balance Server 1 CPU Utilization = Server 2 CPU Utilization = 90% 50% Enhanced Security Enhanced Security Conventional Secure Firewall 1 None 2 3 4 5 6 Mobility: load balance Server 1 CPU Utilization = Server 2 70% CPU Utilization = 70% 7 M Testing and Deployment Outline 1 None 2 3 4 5 None 2 Production 6 7 1 Background Production 2 ization Types Development QA Production 3 Key Techniques Production 4 Summary Develop Test Deploy
Types of ization Process ization Language construction Java,.NET Cross-ISA emulation Apple s 68000-PowerPC to Intel X86 Transition Application virtualization Sandboxing, mobility Taxonomy Process s Same ISA Multiprogrammed Systems Different ISA Dynamic Translators Taxonomy Same ISA Classic-System s System s Different ISA Whole-System s System ization ware Dynamic Binary Optimizers HLL s Hosted s Co-designed s Xen Microsoft s Viridian System Machine Monitor Architectures Traditional What is ization Traditional Hosted Hybrid ware Workstation ware ESX Xen Hypervisor Linux Linux (devel) XP Vista MacOS Machine Monitor Examples: IBM /370, Stanford DISCO
Hosted Machines Hosted Monitor Architecture (Box) Hosted Monitor Architecture Goal: Why Run Machines as an application on an existing Operating System (QEMU, PC, Box, ware-workstation) Application continuity Reuse existing device drivers Leverage OS support File system CPU Scheduler management platform User App Kernel Module Host OS (Window XP) OS (Linux) World Switch Machine Monitor Hosted Monitor Architecture Hosted Monitor Scheduling (Box) Hosted Monitor Scheduling User App Host OS (Window XP) User App Kernel Module OS (Linux) Machine Monitor OS (Vista) Machine Monitor Hosted Monitor Scheduling 3 Hosted Architecture User App Tradeoffs Positives Installs like an application Kernel Module OS (Linux) No Host disk OS partitioning needed (Window XP) disk is a file on host file system No host reboot needed Machine Monitor CPU / Memory ization Hosted Monitor Architecture Runs like an application Negatives Uses host schedulers I/O path is slow Requires world User Appswitch Kernel Module OS (Linux) Relies on host scheduling Device I/O May not be suitable for intensive Network, workloads Disk, Display, Keyboard, Timer, USB Host OS (Window XP) Machine Monitor User App User App OS (Linux) OS (Vista) Hosted Monitor Architecture
ware ESX 2.0 Hybrid Ex 2 - Xen 3.0 Para -virtualization Linux -supported virtualization Unmodified Windows Isolated Device Drivers Source: Ottawa Linux Symposium 2006 presentation. http://www.cl.cam.ac.uk/netos/papers/ Source: http://www.vmware.com/pdf/esx2_performance_implications.pdf Hypervisor Outline -supported -supported single-use single-use monitor monitor Characteristics Characteristics Small size Small size Runs in a special Runs in a special hardware mode hardware mode OS runs in OSnormal runs in privileged level normal Uses privileged level Uses Security System management Security Fault tolerance System management Fault tolerance Hypervisor Operating System Hypervisor User Mode Kernel Mode Monitor Mode 1 Background 2 ization Types 3 Key Techniques 4 Summary
Key Techniques to implement ization I Instruction Interpretation Instruction Interpretation Binary Translation Trap-and-emulate Para-virtualization (hardware virtualization) Emulate Fetch/Decode/Execute pipeline in software Positives Easy to implement Minimal complexity Negatives Slow! Bochs: http://bochs.sourceforge.net/ Example: CPUState Example: izing the Interrupt Flag w/ Instruction Interpreter static struct uint32 GPR[16]; uint32 LR; uint32 PC; int IE; int IRQ; CPUState; void CPU_CLI(void) CPUState.IE = 0; void CPU_STI(void) CPUState.IE = 1; void CPU_Run(void) while (1) inst = Fetch(CPUState.PC); CPUState.PC += 4; switch (inst) case ADD: CPUState.GPR[rd] = GPR[rn] + GPR[rm]; break;... case CLI: CPU_CLI(); break; case STI: CPU_STI(); break; if (CPUState.IRQ && CPUState.IE) CPUState.IE = 0; CPU_Vector(EXC_INT); void CPU_CLI(void) CPUState.IE = 0; void CPU_STI(void) CPUState.IE = 1; void CPU_Vector(int exc) CPUState.LR = CPUState.PC; CPUState.PC = distab[exc];
Privileged Unprivileged II. Binary Translator Binary Translator Code Basic Blocks Code Basic Blocks Translator vpc mov ebx, eax cli and ebx, ~0xfff mov ebx, cr3 Straight-line code Basic Block sti TC Index Translation Cache Callouts CPU Emulation Routines ret Control flow Binary Translator Binary Translation III. Trap and Emulate Trap and Emulate OS + Applications Code Translation Cache vpc mov ebx, eax cli and ebx, ~0xfff mov ebx, cr3 mov ebx, eax call HANDLE_CLI and ebx, ~0xfff mov [CO_ARG], ebx start Page Fault Undef Instr virq sti call HANDLE_CR3 ret call HANDLE_STI jmp HANDLE_RET MMU Emulation CPU Emulation I/O Emulation Machine Monitor
Software M Software M Direct Exec (user) IV. virtualization, Para-virtualization (e.g., Xen) Faults, syscalls, interrupts Kernel Execution M Translated Code (guest kernel) IRET, sysret Traces, faults, interrupts, I/O Modify operating systems to let OS to cooperate with M Let OS runs in Ring1 and user apps run in Ring3 Change sensitive instructions to explicit calls to the M No need to trap and emulate Traditional Address Spaces Traditional Address Spaces s Sol: s 0 4GB Process Address Space Background Process Background Process Operating System Operating System 0 4GB Current Process Operating System Address Space CR3 Real CR3 0 4GB RAM Frame Buffer Devices ROM Physical Address Space
Write to CR3 Outline Write to CR3 Write to CR3 1 Background 2 ization Types CR3 CR3 3 Key Techniques Real CR3 Real CR3 4 Summary Summary References ization (or virtualisation), in computing, is the creation of a virtual (rather than actual) version of something, such as a hardware platform, operating system, storage device, or network resources Impacts Cloud computing Data centers Provides greater opportunities to security Xen and the art of virtualization Memory resource management for Vmware Esx Server http://en.wikipedia.org/wiki/ization http://labs.vmware.com/courseware (great virtualization resources) CPU ization- Basic ization http://labs.vmware.com/download/76/ Memory ization http://labs.vmware.com/download/77/