Building and Managing a Standard Operating Environment Dirk Herrmann Head of Strategic Consulting Central Europe, Red Hat Todd Warner Satellite Product Manager, Red Hat Milan Zázrivec Satellite Software Engineer, Red Hat June 13th, 2013
Part 1: Red Hat Satellite
Systems Management?...Systems!
Systems Management?...Systems! A machine? A definition of a machine? All the pieces and parts that make up that definition? All that... x100? x1000? x10,000?
Systems Management?...Systems! A machine? A definition of a machine? All the pieces and parts that make up that definition? All that... x100? x1000? x10,000?
Systems Management?...Management!
Systems Management?...Management!
Systems Management?...Management!
Systems Management?...Management!
Red Hat DMZ Managed Systems x86 Red Hat Enterprise Virtualization Other Red Hat Network & CDN Internet Firewall Mainframe Workstation
Reporting System inventory Subscriptions Users Errata SCAP PCI-DSS...and more...
A bit of History Since 2002 First, Patch Management Then... Configuration and Provisioning APIs, Multiple Organizations and Federation Reporting and Compliance Bare-Metal Virtualization Cloud Consulting services to Standardize, Integrate and Streamline
We Are Here 2012 2013 Satellite v5.5 Satellite v5.6 2014 2015 2016 Satellite Satellite v5.7 v5.x 2017 Satellite v5.x... 2018
v5.5 Available Now! Themes Features Benefits Modernization / Compliance IPv6 Enablement Better positioning to manage large datacenters OpenSCAP support Ensure those systems adhere to internal & external compliance reqs Content Management Refinement Clone-By-Date IT Admins can control precisely by datestamp and content streams (channels) to adhere to application support requirements Infrastructure Complexity Management Provisioning over Bonded Network Interfaces We now address high-bandwidth systems more completely and can more readily manage multi-network IT deployments Support for Multi-NIC RHN Proxies IT Ecosystem (APIs) API Expansion Continuing to expand the ability for Satellite to integrate into 3rd party and automation tooling
v5.6 Fall 2013 Themes Features Benefits Reporting & Compliance Granular Customer Usage & Deeper / Richer client knowledge and Consumption Reporting reporting. Report per system expiration, SLA or contract. Client Troubleshooting ABRT (Automatic Bug Reporting Tool) Collect and report on critical crash information via the Satellite UI Uptime & Disaster Support for Zero Downtime Recovery Hot Backups Simple backup of the RHN Satellite does not require a scheduled outage window Choice of External Oracle or PostgreSQL Database Backend Freedom of choice between Oracle or PostgreSQL databases for those who require an external DBA-managed database for scaling or compliance Satellite Scaling Satellite across Two Servers Deploy Satellite & embedded DB across Installation Option two servers for significant scaling improvement. No DBA required!
v5 and v6 We Are Here v5 v6 2012 2013 2014 Satellite v5.5 Satellite v5.6 2015 2016 Satellite Satellite v5.7 v5.x Satellite Satellite v6.0 v6.x 2017 Satellite v5.x... 2018
v5.7, v5.8, and beyond Prioritization... Scaling Reporting and Compliance Feature refinement 3rd party integration best practice Transition to Satellite6 tooling, process, best practice
v6 [Past session: Using Red Hat Satellite Today & Into the Future Wed June 12th 1:20pm] A tease about v6 GA in 2014! New architecture Vastly improved workflow Dramatic content and configuration management improvements Fast, efficient, scaling out... forever big Recipe-based configuration CloudForms integration best practice...and much much more...
Part 2: Demo
Standard Operating Environment Effective application lifecycle management: Operating system Custom system configuration Software updates Applications used within an organization Standard implementation of an operating system, its configuration and related software Reduction in costs and time to deploy, configure, manage and support computers within an organization
An example SOE Development QA / Testing Production Custom Custom Applications Applications Custom Custom Applications Applications Custom Custom Applications Applications Configuration Configuration Configuration Configuration Configuration Configuration Updates Updates Updates Updates Updates Updates OS OS OS OS OS OS
System Groups A system may belong to one or more system groups Work with unions and intersections
Software Content Channels as containers for packages (RPMS) and errata advisories (individual software updates)
Software Content: Custom Channels Cloning: spacewalk-clone-by-date Custom RPM pushes, spacewalk-repo-sync
Software Content: Custom Channels spacewalk-clone-by-date
Configuration Management Configuration Channels: containers for config. files Channel rankings Configuration Files Static or templated
Activation Keys Key to bootstrap a system to a Satellite
Activation Keys System Group link
Activation Keys Configuration Channel link
Provisioning Kickstart Profiles: executable system definitions Software content, configuration, custom applications linked together
Provisioning New system to be created in Development group
Provisioning New system created in Development group
Development QA Production Re-iterate procedure with spacewalk-clone-by-date, configuration channels, activation keys, kickstart profiles Automation using API
Part 3: Real Life Examples
4
Yearly Patch Cycles Many customer are not updating their servers more often than 4 times per year.
72
Compliance Requirements Most compliance guidelines demand that critical security issues will be fixed within 72 hours.
Software Lifecycle 72 hours
Major Customer in Financial Services * Emergency Change Handling in less than Release Testing (automated): Release Deployment (automated): Releases every 2 weeks in average 48 hours 8h 6h (1000 servers) Admin2Server Ratio: 1:250 13 RHEL server: 23 min Restore SAP server: 4 min Solaris2RHEL Migration of 74 systems: 4 days (1 admin) Provisioning of a full parallel banking environment: 3d What do they differently? * Public reference story will be published soon!
IDC Study Standardize on RHEL Platform A standardized environment is way more efficient Standardize on RHEL is the best option 82% less time dealing with downtime issues 92% less time dealing with Help Desk activities 20% of downtime of Mixed & Free Distributions 53% less hardware and software costs Source: Understanding Linux Deployment Strategies: The Business Case for Standardizing on Red Hat Enterprise Linux, IDC #227903, April 2011, Sponsored by Red Hat
Standardization 72
Standardization interfaces & risk Data Data Application Application Middleware Middleware Management ManagementTools Tools Operating OperatingSystem System Standardization does not mean homogenization! Standardization means Defined components Defined interfaces Defined processes HW HW//Virt VirtDriver Driver Virtualization Virtualization Hardware Hardware Knowing them and detecting deviations!
Standardize with RHEL Data Data 10 year Lifecycle Application Application Middleware Middleware ISV Certifications Management ManagementTools Tools Operating OperatingSystem System RHEL RHEL HW HW//Virt VirtDriver Driver ABI / API Stability Virtualization Virtualization Hardware Hardware OEM Certifications
Standardization - Core Build Data Data Application Application smallest common denominator Operating OperatingSystem System HW HW//Virt VirtDriver Driver Core Build Middleware Middleware Management ManagementTools Tools Virtualization Virtualization Hardware Hardware Defined core for all RHEL systems Integrates required RHEL, 3rd party and custom components (backup, monitoring, scheduling,...) typically based on minimal installation defines basic security & compliance configuration Flexible and complete to run in all target environments
Critical Success Factors Platform
72 Architecture
Architecture App Platforms / Messaging / Cache / Grid / SOA / Mobile PaaS Red Hat Enterprise Linux Management Systems Foundational Storage for Big Data Applications Physical Servers Virtual Servers Network & Storage Infrastructure Cloud Servers Open Hybrid Cloud Management IaaS
Red Hat Security Management 95 1
Red Hat Security Management 95 percent of all critical security issues in Red Hat Enterprise Linux are fixed within https://www.redhat.com/security/data/metrics/ 1 day.
Software Lifecycle 24 hours 48 hours
Critical Success Factors To ol s Platform
72 Automation
Automated Server Provisioning Provision a server within a few minutes including OS (IaaS) and application platform (PaaS) No manual intervention required No manual postprocessing required Hardware / virtualization specific adaptions handled automatically System is fully integrated into ITSM tool chain After reboot the system is tested and production ready
I don't need this automated provisioning. Initial Provisioning Disaster Recovery Data Restore System Cloning Env specific adaptions Dynamic Scale-Out Load Balancer Mgt
One technology core multi-purpose usage Initial Provisioning Disaster Recovery Data Restore System Cloning Env specific adaptions Dynamic Scale-Out Load Balancer Mgt
Software Lifecycle 24 hours Remaining: 48 hours
Software Lifecycle 24 hours 4 hours Remaining: 44 hours
Fully automated QA cycles 1 2 + 3 4 5
Software Lifecycle 24 hours 4 hours 8 hours Remaining: 36 hours
Deployment Automation via Satellite Customer Example from a large financial customer in Germany Fallback Update Success Wave 0 (Test) no Success Criteria: 100 % within 60 min STOP yes Fallback STOP Wave 1 no Success Success Criteria: 95 % within 60 min Update yes Fallback Wave N Update Success no Actually < 3 % error rate Success Criteria: 95 % within 60 min STOP
Software Lifecycle 6h 24 hours 4 hours 8 hours Remaining: 30 hours
Control 72
Security & Compliance Management CVE Errata System Mapping Customized Security Reporting Daily / nightly verification checks OpenSCAP integration for compliance reporting Integration of existing compliance checking scripts Vulnerability Scanner Integration (Nessus) Software lifecycle tracking Documentation / Incident & Problem Mgt Integration
Software Lifecycle 6h 24 hours 4 hours 8 hours 2h Remaining: 28 hours
~ 100 SPARC Solaris 25 unmanaged RHEL 25 % virtualized High operational costs Limited scalability Little automation Timeframe between: 1 year Major Customer in Financial Services * 40 HP x86 RHEL 850 fully manag. RHEL 99% virtualized low operational costs unlimited scalability Fully automation Release Creation QA Automation Deployment Automation How did they get there? * Public reference story will be published soon!
Critical Success Factors To ol s Platform
Red Hat Standard Operating Environment
Standard Operating Environment Implementation
Key Finding IDC Study Standardize on RHEL IDC Standardize on RHEL **: IT Productivity Key Performance Metrics IT productivity Metric Standardized on RHEL Mixed Primarily Nonpaid Servers per admin (virtual and physical) 174 115 97 IT users per admin 422 373 358 Hours to deploy a virtual server 0.4 1.0 1.0 Days to deploy an application 11 19 23 Higher server counts, higher user counts per admin when standardized on RHEL Deployment consistency, tools and lack of diversity as drivers for productivity * Public reference story will be published soon! ** Source: IDC White Paper sponsored by Red Hat, IDC #227903, April 2011
Major Customer in Financial Services * IDC Standardize on RHEL **: IT Productivity Key Performance Metrics IT productivity Metric Standardized on RHEL Mixed Primarily SOE Customer Nonpaid 250 Servers per admin (virtual and physical) 174 115 IT users per admin 422 373 Hours to deploy a virtual server 0.4 1.0 0.16 1.0 Days to deploy an application 11 19 1 23 Benefits 97 358 Standardization hardware and software (no vendor lock-in) Significant license and HW maintenance cost reduction High operation reliability and flexibility Acceptance and trust in RHEL as a mission critical platform * Public reference story will be published soon! ** Source: IDC White Paper sponsored by Red Hat, IDC #227903, April 2011
Key Components Control Standardization Knowledge Automation Architecture
Other Red Hat Satellite-related Sessions Wednesday, June 12 1:20 pm - 2:20 pm Using Red Hat Satellite Today & Into the Future 1:20 pm - 2:20 pm Controlling Clouds Beyond Safety 3:40 pm - 4:40 pm Red Hat Satellite Power User Tips & Tricks: System Deployment 4:50 pm - 5:50 pm Managing SELinux in the Enterprise https://www.redhat.com/summit/sessions/topics/red-hat-satellite.html
Other Red Hat Satellite-related Sessions Thursday, June 13 3:40 pm - 4:40 pm War Stories from the Cloud: Lessons from US Defense Agencies 4:50 pm - 5:50 pm Building & Managing a Standard Operating Environment 4:50 pm - 5:50 pm Managing Updates on Red Hat Enterprise Linux 4:50 pm - 5:50 pm Real-world Perspectives: Gaining Competitive Advantages with Red Hat Solutions https://www.redhat.com/summit/sessions/topics/red-hat-satellite.html
Other Red Hat Satellite-related Sessions Friday, June 14 9:45 am - 10:45 am Real-world Perspectives: Optimizing Infrastructures with Red Hat Satellite 11:00 am - 12:00 pm Begin Programming Your Red Hat Satellite Server https://www.redhat.com/summit/sessions/topics/red-hat-satellite.html
28 hours remaining what to do with it? Efficiency means doings things right. Effectiveness means doing the right things.