CON9577 Performance Optimizations for Cloud Infrastructure as a Service

CON9577 Performance Optimizations for Cloud Infrastructure as a Service John Falkenthal, Software Development Sr. Director - Oracle VM SPARC and VirtualBox Jeff Savit, Senior Principal Technical Product Manager October 26, 2015 Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted

Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 3

Program Agenda 1 2 3 4 5 Oracle VM performance in cloud Performance concepts for the cloud and VM environments Oracle VM Server for x86: optimizing CPU, Memory, I/O Oracle VM Server for SPARC: optimizing CPU, Memory, I/O Recommendations and Resources Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 4

Oracle VM performance in cloud 1 2 3 4 5 Oracle VM performance in cloud Performance concepts for the cloud and VM environments Oracle VM Server for x86: optimizing CPU, Memory, I/O Oracle VM Server for SPARC: optimizing CPU, Memory, I/O Recommendations and Resources Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 5

Oracle VM performance in cloud Not just about performance of running VMs Also performance of management at scale Multi-tenancy, many clients, can t give individual attention to all Oracle VM is designed to engineer in best practices Prevent problems through design features and default configuration Reduce manual configuration for good out of the box performance Best practices where configuration settings are necessary Oracle VM product family members address different markets Entire stack tested to find limits, and enhanced in each release Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 6

Oracle VM Portfolio From Development, Fully Configurable Software to Turn Key Systems Desktop Oracle VM VirtualBox Oracle VM x86 Private Cloud On Premise Oracle VM SPARC Oracle VM x86 Public Cloud Oracle VM SPARC Copyright 2015, Oracle and/or its affiliates. All rights reserved. 7

Oracle VM Family: Choice and Flexibility From Development, Fully Configurable Software to Turn Key Systems Oracle VM VirtualBox Development platform with multiple OS support Oracle VM Server for x86 & SPARC Zero cost license: pay only for support Full functionality including all management software Your hardware infrastructure Private Cloud Appliance Enterprise class, turn key converged infrastructure system containing servers, storage, network, and software From power-on to production in about an hour Automated installation and configuration Copyright 2015, Oracle and/or its affiliates. All rights reserved. 8

Performance concepts for the cloud and VM environments Evaluating and performance: books-worth of material, but in short: Target service level objectives, not load factors Response time: in terms of X% of a specified transaction complete in Y seconds at a rate of N transactions per second Throughput: in terms of units of work (jobs) accomplished per time interval The simplified performance management loop: Proactively measure performance and resource utilization for baseline values Measure: are we meeting service level objectives? If yes, take a break Identify bottlenecks getting in the way Address those bottlenecks Re-measure. Lather, rinse, repeat Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 10

Performance goals for the cloud and VM environments Run workloads as close to bare metal performance as possible Support large servers efficiently: >100 CPUs, multi-tb memory Typical: 2 (very common) to 8 socket, 256GB to 1TB RAM Support large VMs: >100 vcpus, 512GB/1TB/2TB RAM Typical: 4 to 32 vcpus, 16GB to 128GB RAM. Typical 4-8GB per vcpu Drive 10GbE networking near line rate Scale to dozens of servers, hundreds of VMs under single-pane control Scale to hundreds of LUNs, paths Do all the above without much tuning or manual intervention Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 11

Performance concepts for the cloud and VM environments Rules are sometimes different in the cloud. Sometimes less is more Virtual CPUs Excess CPUs can increase VM s SMP overhead (true on bare metal too) Many applications don t scale with more CPUs, so no point in adding more Can over-subscribe on x86 subject to contention and service requirements Jitter and performance can worsen if aggregate vcpus > pcpus, especially under load No individual VM should have its vcpus > pcpus. Ideally fits in single server socket Memory On bare metal: assign it all or it just goes to waste In cloud: excess memory assigned to a VM reduces number of VMs you can host In a VM: excess memory increases NUMA latency since spreads over sockets Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 12

Oracle VM for x86 Architecture Oracle VM Manager Centralized management server Web browser-based: No client required Manage hundreds or thousands of VMs centrally Support x86 and SPARC from same interface Advanced virtualization management included Oracle VM Server Installs on bare-metal servers in about a minute Linux, Solaris and Windows guests: Supports paravirtualizedand hardware-virtualized kernels and drivers High performance at scale: Up to 320 physical CPUs and 6TB memory; Up to 128 vcpu and 2TB memory per VM Copyright 2015, Oracle and/or its affiliates. All rights reserved. 14

Oracle VM Server for x86: a little bit of architecture Xen thin hypervisor in software Dom0 for platform control and virtual I/O CPUs are time-sliced in software with Xen s priority-based credit scheduler Memory is not overcommitted or shared between VMs. Hypervisor does not swap/page VM memory Virtual network and disk I/O are served through dom0 Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 15

Oracle VM Server for x86: optimizing CPU and memory Most important tuning options are automated Dom0 CPUs are pinned to first server socket, uses memory on that socket Guest vcpus have node affinity: memory is assigned from a socket and CPUs are assigned to the same socket if possible. Reduces remote memory access Administrator guidance: Avoid guest #vcpu>#cpus_per_socket to avoid NUMA effect Avoid excessive over-subscription of active virtual CPUs (with high %CPU) In guest, watch iostat %steal or vmstat %st to see if was dispatchable but preempted If hyperthreading, count in units of threads (the effects are closely aligned) Use Dynamic Resource Scheduling to load balance VMs across servers Can pin CPUs to ensure placement for NUMA and cache latency Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 16

Oracle VM Server for x86: optimizing CPU: hyperthreading Hyperthreading improves some cases, degrades others Workload dependent, so must be tested Linux tries to spread work over cores, but guest doesn t know if virtual CPUs are on the same core or socket. Real topology data isn t available Can help scalable apps when #vcpus > cores/socket but < threads/socket when keeps allocation on same socket Best for multi-threaded applications that can drive multiple vcpus Can reduce per-vcpu performance and affect single-thread dominated applications Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 17

Oracle VM Server for x86: optimizing CPU, Memory, I/O Virtual machine domain type and weights HVM + PV drivers gives best performance on current systems, 64-bit guests Drivers available for Oracle Linux, Solaris, Windows PVM mode may still outperform with 32-bit guests, and on old systems Credit scheduler useful to reflect importance when CPU constrained but never make a guest (domu) weight higher than dom0 Hugepages help large memory guests Especially for Oracle DB with large SGA to reduce TLB footprint and thrashing Must be enabled for PVM guests There can be trade-offs for flexibility Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 18

Oracle VM Server for x86: optimizing I/O Storage Physical disk (FC or iscsi LUN) outperforms virtual disks from repositories Physical disk is needed for Oracle RAC Virtual disks easier to manage, more common in multitenant and cloud usage NFS vs. OCFS2 on iscsi or FC LUN repositories No clear winner (IMO) between SAN and NAS Ease of use and administration for NFS OCFS2-based repository provides thin clone, which is very handy vdisk performance may not be much different. YMMV Storage Repository Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 19

Oracle VM Server for x86: optimizing network I/O NUMA sensitive on fast networks (10GbE) Main tuning actions (dom0 sizing and pinning) are now set by default Substantial benefit on network performance with this one setting Should review if running old versions of Oracle VM Can pin key guests to same socket as dom0 (not a typical cloud use case) NUMA is top item, other tuning options listed in whitepaper (see references at end of presentation) Balance IRQs if individual CPUs are saturated Set TCP windows Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 20

Oracle VM Server for SPARC Evolution of Logical Domains technology Isolated OS and applications in each logical domain Oracle Solaris 11 S11 zone S10 zone Oracle Solaris 10 Oracle Solaris 11 Oracle Solaris 11 Domain with zones Solaris 10 Domain GP Domain Domain with DB zones Oracle Solaris 11 Oracle Solaris 11 Oracle Solaris 11 Oracle Solaris 11 Database Domain Database Domain Database Domain Database Domain Firmware-based hypervisor Each logical domain runs in dedicated CPU threads and RAM for bare-metal CPU performance SPARC Hypervisor SPARC Server Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Oracle VM Server for SPARC: a little bit of architecture Thin hypervisor in SPARC firmware Control domain used for platform control Acts as a service domain to provide virtual I/O Optional additional service domains for resiliency and non-disruptive upgrade CPUs multiplexed in hardware, and cores/threads dedicated to domains Memory is not over-committed. Hypervisor does not swap/page VM memory. No memory side-channel security vulnerabilities like other vendors, no performance anomalies due to sharing memory Physical I/O available via SR-IOV and PCIe bus assignment to domains Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 23

Oracle VM Server for SPARC Virtualized I/O Architecture Guest Domain Guest Domain Service Domain App App App App kernel kernel Solaris ` I/O stack Virtual Device Driver LDC LDC kernel Solaris I/O stack Virtual Device Driver LDC LDC Hypervisor LDC Virtual Device Service LDC LDC Solaris I/O stack Device Driver Physical Devices Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Oracle VM Server for SPARC: optimizing CPU, Memory Every domain has dedicated cores/threads Logical Domains manager assigns CPUs and memory to reduce NUMA latency CPU and memory tuning measures Size domains for their expected load, especially service domains Best practices whitepaper gives guidance for initial values Measure utilization (vmstat, mpstat, ldm list) and adjust suitably. Can add/remove CPUs on the fly Allocate whole cores to domains rather than threads Chip multi-threading, hardware-assisted cryptography, large pages all available by default. Solaris knows how to dedicate cores to critical threads Really, that s pretty much it Policy management can automate resource distribution on server or pool. Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 25

Oracle VM Server for SPARC: optimizing I/O Many I/O options. First choice is virtual I/O vs. physical Virtual I/O is proxied through service domains Physical I/O is directly performed by the application domain Virtual I/O can provide near-native network and disk performance Most flexible (permits live migration, easiest provisioning, most device sharing) Preferred in general case, and especially for cloud environment Very important: size service domains to drive I/O (based on load) Virtual disk: best performance with FC or iscsi LUN backends Virtual network: Can use aggregations, jumbo frames, just as on bare metal Virtual HBA (NEW!) Improves on vdisk: better multi-pathing, any SCSI device type (disk, tape, cd/dvd), improved LUN management, with similar performance Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 26

Oracle VM Server for SPARC: optimizing I/O Physical I/O for extreme performance use case Native performance and scaled down service domain requirements Additional considerations when using physical I/O: RAS, live migration, scaling, device sharing Can assign entire PCIe bus acts like bare metal SR-IOV with qualified Ethernet, IB or FC cards Useful for datacenter maximum performance, but not generally a cloud deployment method Virtual I/O the typical choice Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 27

Recommendations / Summary Oracle VM is designed for out of the box performance wherever possible low touch performance important for cloud Apply best practices (which are often the defaults) Stay with current releases to benefit from enhancements Tuning may be needed to achieve service levels in some cases Measure and tune to target workloads Measure baseline performance before there is a problem Use baseline figures so you can see what s different when there is a problem Use metrics and tests that are meaningful to application and service level requirements. Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 29

Oracle VM Foundation for Cloud At the core of Oracle s Cloud strategy Integrated VM lifecycle and cloud management solution with Oracle Enterprise Manager Supports both x86 and SPARC Certified for Oracle products Ready for OpenStack Cloud platform for Oracle & Non-Oracle applications Supports Oracle Linux, Oracle Solaris, Microsoft Windows, and other Linux distributions Copyright 2015, Oracle and/or its affiliates. All rights reserved. 30

Stay Connected Join our communities, visit our websites for more information Oracle.com/virtualization @ORCL_Virtualize Facebook.com/OracleVirtualization Oracle VM Group Blogs.oracle.com/virtualization Youtube.com/OracleVirtualization Download: www.oracle.com/technetwork/server-storage/vm/downloads/index.html Training: education.oracle.com/virtualization Copyright 2015, Oracle and/or its affiliates. All rights reserved. 31

Learn More Oracle VM Documentation landing page http://www.oracle.com/technetwork/server-storage/vm/documentation/index.html Oracle VM whitepapers http://www.oracle.com/technetwork/serverstorage/vm/overview/index-160875.html Oracle VM Server for SPARC Best Practices: http://www.oracle.com/technetwork/serverstorage/vm/ovmsparc-best-practices-2334546.pdf Oracle Virtualization blog: https://blogs.oracle.com/virtualization/ MOS notes: Oracle VM Server for SPARC (LDoms) Document Index (Doc ID 1367098.1) CPU Over Commitment and Scheduling on Oracle VM Server 3.x (Doc ID 1640862.1) How to detect an Hyper-Threading or/and a Multi-Core CPU on Oracle Linux (Doc ID 462410.1 Oracle VM 3: 10GbE Network Performance Tuning (Doc ID 1519875.1) Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 32