Design Considerations for Large Scale Deployment of Oracle VM in Oracle s Managed Cloud Service Jose Fernando Niño Higuera
Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 3
Program Agenda 1 2 3 4 5 What is Oracle Managed Cloud Services (OMCS) OMCS Design Criteria How We Use Oracle VM Lessons Learned and Best Practices Benefits of Oracle VM and OMCS 4
Oracle Managed Cloud Services (OMCS) ( ) Three Ways for Customers to use Oracle Software 1. Purchase Software Product and self-host 2. Subscribe to Software as a Service (SaaS) 3. Have OMCS host the customer s software 5
Cloud Marketplace Oracle SaaS, PaaS, IaaS Cloud Offerings Infrastructure as a Service Platform as a Service Software as a Service Infrastructure Compute Storage Messaging Platform Database Java Database Backup Developer Documents Business Intelligence Mobile Customers Marketing Sales Service People Global Human Resources Talent Management Business Financials Procurement Project Portfolio Management Supply Chain Value Chain execution Product Value Chain Enterprise Performance Enterprise Planning Financial Planning Social Social Network Social Marketing Social Engagement & Monitoring Social Data & Insight Multi-Tenant Shared Machines* Oracle owns Hardware and Software Customer pays for usage * typically 6
Oracle Managed Cloud Services (OMCS) Applications Managed Cloud Service Technology Managed Cloud Service Extended Managed Cloud Service E-Business Suite Fusion Applications PeopleSoft Demand Management Siebel Markdown Optimization J D Edwards Information Discovery Hyperion Project Management Business Intelligence Beehive Collaboration Commerce Transportation Management Agile User Productivity Kit Retail Retail Predictive Application Governance Risk & Compliance Oracle Database Fusion Middleware Web Center Engineered Systems Identity Management Backup Refresh Upgrade Migration CEMLI Management Business Transaction Monitoring Security PCI & HIPAA Compliance Disaster Recovery Non-Production Environment Service Other Extended Services Single-Tenant Dedicated Machines Customer owns Software - Oracle owns Hardware and manages everything Typically in the Oracle Data Center but sometimes @customer/partner 7
Oracle Managed Cloud Services (OMCS) Why use OMCS Single Provider: Hardware, Software, Network, Storage, Interoperability If there s a problem, it s Oracle s problem Expertise: Let Oracle manage Oracle Large expert pool available around the clock Direct Access to Product Development Groups Leverage: Design & Optimize once, Repeat often 8
Oracle Virtualization Strategy At the core of Oracle s cloud strategy Integrated VM lifecycle & cloud management solution with Oracle Enterprise Manager Supports both x86 and SPARC Integrated with OpenStack Cloud platform for Oracle & Non-Oracle applications Supports Oracle Linux, Oracle Solaris, Microsoft Windows 9
About Oracle VM Oracle VM Templates Oracle Real Application Clusters(RAC) Oracle E-Business Suites Oracle JD Edward EnterpriseOne. Oracle Linux Oracle Solaris Integrated for scale & ease of deployment Oracle Virtual Networking Boosts Performance by 30%. X86 and SPARC Oracle Enterprise Manager End-to-end Management- Physical to Virtual To Cloud 10
OMCS Design Criteria 1. Security 2. Isolation of Users (Customers) 3. Stability 4. Disaster Containment 5. Large Scale 6. Performance 7. Cost 11
OMCS Design Criteria: Security Protect the Customer s Data under all circumstances Rigorous Security Review Process Deployment Architecture Need-to-know access to data Authentication, Authorization & Audit for all Activities Guard against unauthorized Access Intrusion Detection Perimeter Security around the Deployment Cloud Additional Segregation & Firewalls within Security Patches via Routine Maintenance Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost Virtualization allows us the luxury of having separate Machines for each Tier within each Instance 12
OMCS Design Criteria: Isolation of Users (Customers) Strong Separation between Customers is mandatory One customer must never see another s data or activities After years of testing and Operation Oracle VM Virtual Machines are proven just as safe as physical machines Virtual disks with backing store from a storage pool are safe Flat Network alone doesn t isolate enough for OMCS iptables + ebtables + Perimeter Firewall provide Security vlan + Internal Firewall provide Isolation Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost Oracle VM Virtual Machines plus vlans meet our Segregation Requirements 13
OMCS Design Criteria: Stability Preservation of Data must never be in question What matters is the Customer s App Infrastructure components have to support this goal Avoid Single Points of Failure Redundancy wherever possible Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost Choose proven, stable Infrastructure Components with active/active or active/passive Failover Capabilities 14
OMCS Design Criteria: Disaster Containment In spite of the best prevention, high-impact infrastructure breakdowns can happen Backups, of course Online snapshots, on-site storage, off-site Limit the number of VMs that can be affected Fully segregated Zones We accept certain limitations e.g. limited migration mobility between zones Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost Optional Disaster Recovery at a different Data Center We partition each data center into smaller self-contained Zones 15
OMCS Design Criteria: Large Scale Design for 100,000 Virtual Machines Actual number of deployed VMs is approaching 20,000 Allow very large VMs 100+ vcpus and multi-tb memory Typical today: 4 32 vcpu and 16-128 GB Typically 4-8 GB per vcpu for Oracle Applications Accommodate Multiple concurrent Operators Start, stop, resize, clone, etc Service 1,000+ Customers Each Customer is a Corporation Quick turn-around time at scale Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost Oracle VM 3 with multiple Oracle VM Managers and lots of Server Pools 16
OMCS Design Criteria: Performance The Deployment Architecture must not substantially limit performance provided by the underlying raw hardware The Networking Stack in VMs must run at GigE+ speeds Use the best Virtualization Method available for each use case Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost Select Paravirtualization wherever possible. Otherwise Hardware Virtualization with PV Drivers 17
OMCS Design Criteria: Cost Reduce Cost through standard building blocks and repeatable Process Standard Hardware Over-provisioning in the interest of Uniformity is acceptable Certified Configuration images for all Applications and Databases Pre-Design a library of standard building blocks Invest in Tuning and Testing Re-use these for every customer Repeatable Standard Process Cookbook Automation wherever possible Share Infrastructure where Possible Segregate where Necessary Standard Hardware stays fixed for one model year Certified Configurations with Periodic Updates Process Cookbooks Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost 18
Oracle VM in OMCS All Server Virtualization in OMCS uses Oracle VM. All Server Deployments are virtualized by default 15,000 + VMs in operation Oracle VM 3.2.4 in use since 2012 Evangelize, Certify, Prove Security Experiment with HVM vs. ParaVirt vs. other Introduce Windows VM Performance Optimization Work Oracle VM 2.2.2 in use in some Legacy Zones Introduced first Xen based Virtual Machine Oracle VM Pre-release beta Oracle VM 0.9 (pre-release and joint Beta) Linux Paravirtualized only Oracle VM Initial Release 2005 2006 2007 2007 2008 2014 19
Oracle VM Deployment in OMCS Self-contained Zones Even a catastrophic zone failure of one zone cannot affect the other zones Zone Oracle VM Manager All required networking equipment Switch/Router, Load Balancer, Firewall, Security Single switch hop from any to any node in zone (full 10GigE bandwidth, no shared uplinks) Server Pool Storage Network Redundant Storage NAS per Server Pool SAN per Zone Four Oracle VM Server Pools with 12 physical servers each (48 total) One Oracle VM Manager Instance Also: Legacy Zones Giant Zones, being migrated / converted 20
Global Deployment Global Data Center 1 Data Center 2 Enterprise Manager Zone 1 Zone 2 Zone 3 OVM Manager OVM Manager Oracle VM Manager 7 Data Centers 4 Primary 3 Disaster Recovery Server Pool Server Pool Server Pool Multiple Zones per Data Center Target 100 zones total Stor Stor age Stor age age Network Network Network Shared Service Zone Redundant or non-critical Enterprise Manager One global Instance Redundant Shared Services 21
What do OMCS Customers share? Shared Data Center Real Estate Power, Cooling Generic Network Internet, WAN, LAN Storage Pool Physical Server Pool Dedicated Customer Network WAN and/or VPN DNS Name Space LAN Subnets and VLAN Customer specific Gear Customer Storage Shares, Projects, LUNs Machines Virtual and Physical 22
OMCS Server Hardware Standard Building Block Sun Server X4-2 with 24 cores (48 Threads) and 512 GB memory CPU oversubscription yields approx 50-100 vcpus Bonded dual 10gigE NIC Specialty Configurations Sun Server X4-4 for high performance applications Sun Server X4-2L with SSD for low latency transient storage 128GB memory configuration for certain 32-bit VMs FibreChannel and Infiniband 23
Networking Bonded dual 10gigE NIC per physical Server vlans in the Hypervisor Administrative vlans Dom-0 login Cluster Heartbeat Live Migration NFS Network Separate vlans per Customer Public and Private Middle Tier Access Privileged and Restricted Database Access Database Cluster Interconnect PVLAN and XVLAN 24
Engineered Systems Engineered Systems are an integral part of OMCS Deployments Exadata Database Machine Physical Machines only Oracle Exalogic Elastic Cloud With Server Virtualization, IB Partitioning Oracle SuperCluster With Hardware Virtualization Oracle Big Data Appliance Oracle Exalytics In-Memory Machine Physical Machines only 25
Standard and Custom Automation Automate Provisioning of Server, Network, Storage at once Custom Scripting against the Oracle VM Manager and Oracle Enterprise Manager APIs Library of Partial and complete Workflows Build abstract composite objects ( Instance ) in one command Infrastructure Provisioning Subnet, IP, DNS, vlan, Firewall, LoadBalancer Application Provisioning 26
Do we use Physical Machines? No, not normally. Yes, sometimes. Virtualization is the Default We even virtualize single VMs using the whole machine Third-party Applications which are not certified as VMs Specialty Applications Appliances, some Infiniband, FC, DAS 27
Lessons Learned Don t hesitate to virtualize machines Oracle VM is mature, stable, enterprise-proven Don t put all your eggs in one basket Compartmentalize large domains into smaller zones Oracle VM Managers can themselves be hosted in VMs Just no circular References! Virtualization is also a great tool to right-size machines for License Compliance 28
Re-Sizing VMs We always stop / start the VM after vcpu or Memory resizing Reasons: 1. Most Applications choke on resource reductions 2. Most Applications ignore resource increases 3. Those that can deal with it require re-tuning 29
Live Migration and Ksplice We cannot always impose a reboot (downtime) on our customers Ksplice Allows us to patch the kernel in the running VM Live Migration Used occasionally for Server (HW) Maintenance Used occasionally for Capacity Rebalancing 30
How We Migrate Machines Our Data Center safety zones prevent Live Migration across Zones Occasional Live Migrations within a Zone The majority of moves are Cold Migrations a) Shutdown b) Image move/copy across zone via Router c) Restart Special Case: LiveMigration-to-self Useful to re-initialize certain driver functions 31
The VM Tetris Problem 32-bit VMs have to reside in lowest 128GB of physical Memory Start 32-bit VMs first, then 64-bit Repeated start-stop cycles of mixed 32-bit and 64-bit VMs lead to fragmentation Eventually, no low memory can be found, and VMs fail to start or live migrate Our Solution: Limit 32-bit VMs to small physical machines (128GB memory) 32
Core Dumps Core Dumps can become large and take a long time Our Environment: Hypervisor core dumps entire memory to local disk VMs: Core dump to local disk NetDump no longer used (too slow) 33
Performance Considerations We found that paravirtualized VMs run at nearly the native speed of the physical machine HVM imposes some performance decrease, can be greatly reduced through PV drivers The Network stack is plenty fast enough for most applications We We treat treat VMs VMs as as fully equivalent to to Physical physical Machines machines The Advantages of OVM far outweigh some negligible performance impact The Advantages of Oracle VM far outweigh some negligible performance loss 34
Designed & Tested Together Oracle Develops & Uses The Stack Internally Oracle Managed Cloud Services 15,000+ VMs Internal Testing 22,700 Oracle x86 servers supporting 182,400 Oracle Virtual Machines 26,700,000 test and production hours per week Workloads: software/hardware development, corporate infrastructure Test Environments Oracle x86 Server Hardware Oracle Storage Oracle Operating Systems (Oracle Solaris and Oracle Linux) Oracle VM Oracle Database, Oracle Middleware, Oracle Applications 35
Stay Connected Join the Oracle VM and Oracle Cloud Communities @ORCL_Virtualize Facebook.com/ OracleVirtualization Blogs.oracle.com /virtualization Oracle VM Group YouTube.com/ OracleVirtualization Download: edelivery.oracle.com/oraclevm Visit us: oracle.com/virtualization @OracleCloudZone, #OracleCloud Facebook.com/ OracleCloudComputing Blogs.oracle.com/cloud Learn more: oracle.com/cloud Try now: cloud.oracle.com 36
What OMCS Customers Are Saying: We are a diverse business with a multitude of internet and media properties growing at varying rates, as well as frequent acquisitions and divestures with a global reach. Oracle Managed Cloud Services provides us with the scalability and flexibility to successfully manage an expansive and complex organization that supports several finance and accounting management teams. Paul Scribano, Vice President, Finance, Mindspark Interactive Inc., part of IAC Search and Media Inc., IAC We have a small IT staff, so we must work very efficiently and optimize our resources. Oracle Managed Cloud Services is critical to our ability to run and optimize our Oracle E- Business Suite environment. It ensures extremely high availability, timely patches and maintenance, industry-leading 24/7 support, and world-class system backup and recovery. Cindy Shieh, Information Systems Manager, Greenball Corp. Oracle E-Business Suite running through Oracle Managed Cloud Services provides a compelling value proposition for Genworth Financial. It allows us to take advantage of industry-leading enterprise applications and gain the expertise of Oracle managing the applications on Oracle technology. JP Raffenot, Director of IT/Applications, Genworth Financial Inc. 37
38