Concurrency CHICAGOLAND LOCAL CHAPTER DevOpsand The Service Desk Don t Let The Developers Hijack The Discussion! Randy Steinberg RSteinberg@concurrency.com
The Issue Here You Go! Dev Great! Ops
This Is A Billion Dollar Problem! Major Healthcare Deployment Failure HHS Secretary Resigns Financial Firm $457M Trading Error Riles Investors Customer Order Failures Result In IPO Withdrawal Airline IT Woes -Lost Customers, Airport Fines, Stock Drops 2.4% Federal Disaster -High volume of background check transaction volumes halts gun sales
What is the return on investment of an IT solution that cannot be deployed and operated at acceptable cost and risk?
Amount negative equal to your development investment plus lost opportunity costs and possible fines and penalties.
When does the business actually see and realize value from IT?
At the point the SERVICE is delivered!
Scary Things People Say About DevOps With DevOps we can deploy up to 15 times a week You can bypass your IT organization Infrastructure engineers often have little incentive to change anything -they are used to being ignored until something goes wrong Just co-locate dev and ops desks and place them under the same management It s all about continuous delivery of services
DevOps Defined A cultural and professional movement that stresses communication, collaboration, and integration between software developers, quality assurance and IT operations professionals with these perspectives: A service perspective (people, process, technology, data) An agileperspective (continuous delivery, faster deployment) A risk perspective (safety, security, reliability)
DevOpsOperating Principles Flow increase the flow (speed) of work from left(dev) to right (Ops) Feedback continuously improve, work together (extend operations into development) and recognize that mistakes are more expensive to deal with once in production Continuous Experimentation and Learning create a culture that fosters experimentation, and quickly learns from failures
DevOpsLingo Continuous Integration requirement that developers integrate code into a shared repository on a daily basis Continuous Delivery make sure that software is ALWAYS in a releasable state throughout it s lifecycle -not to be confused with Continuous Deployment every change that passes automated tests gets automatically deployed to production
Key DevOpsBenefits Allows IT to respond effectively to accelerated demand for quality software products and services Recognizes the need for cultural and organizational improvements to communicate and collaborate across IT engineering silos Supports and leverages industry best practices (IT Service Management, Agile, Lean) Encourages use of automation wherever appropriate Shifts labor costs from unplanned, low value activities to productive value activities Claimed Benefits*: Code shipped 30 times faster Deployments completed 8,000 times faster 50% fewer failures Failures restored 12 times faster *2013 State Of DevOpsReport Puppet Labs
Melding Together Extend delivery to production Ensure delivered solutions can be operated at acceptable cost and risk Dev Ops Offer IT support services in a manner that can easily be consumed at acceptable speed and cost Extend operations into development
Ensuring solutions can be operated at Utility Fit For Purpose Let s put kiosks in shopping malls to sell our products! acceptable cost and risk Warranty Fit For Use Is there enough space and power? Can the floor handle the weight of the kiosk? Does it pass fire and safety inspection? Is there a pathway to move kiosk into position? Who unpacks and disposes of the shipping crate? Do malls restrict when you can install the kiosks? What repair service will we use to fix them? What does customer do if kiosk is not working? Who approves content to be displayed on kiosk? Do we have adequate skills to support them? Can support resources quickly get on site if needed? etc.
Working Together A Win-Win Story
Offer IT support services in a manner easily consumed at acceptable speed and cost Lean Operations Hybrid Cloud Virtualization Containerization Converged Infrastructure Declarative Configuration Management Software Defined Infrastructure/Networks/Data Center
Organization Challenges Developer View Of Operations Operations View Of Development Architecture As Viewed By Development and Operations Out Of My Way! Here You Go! Great! My Service Levels! Neither Of You Truly Understand IT!
A Typical IT Organization No one is accountable for service just capability CIO IT Development IT Operations Network Management Server Management Storage Management What organization is best positioned to work across all these silos? Etc.
Extending The Service Desk May Be The Easiest Approach To All This Organizationally Service Desk IT Development IT Architecture and Operational Planning IT Operations
Developer View Of IT What they say about themselves We are king we own the CIO relationship We are closest to the business and know what s best for them No functionality = No solution What they say about Architecture Important to us for Data and Applications Could care less about rest of enterprise they should be using what we develop Gets in the way and stifles our creativity What they say about Operations Slooooowwwww!!! We ll talk to them when we re ready to go live (usually 1-2 weeks before cutover) We ve already built solutions for important things like backup, job scheduling and monitoring They ll get it done they always do
Operations View Of IT What they say about themselves We re always dumped on at the last minute We are closest to how things work and know what s best for everyone We re the ones that make it all work What they say about Architecture They speak gobbilty-gook Why bother? Just put the tools in What the hell is an architecture anyway? Best job in IT all theory and don t have to produce anything. What they say about Development They want to install what??? Why weren t we told about this earlier? These guys have no clue how to make things really work There s no budget for this
Architect View Of IT What they say about themselves No one in IT appreciates what we do Most in IT don t understand important concepts like layering, abstraction, normalization What they say about Development Developers don t listen to us They think we re just a tick mark on the QA checklist Reuse is not in their vocabulary What they say about Operations Never heard of us These guys are so tactical we can t relate
Break In The Service Delivery Chain Typical Developer/Architect Concerns... The job is done when the system is built Architect Design Build Test Convert The job ain't done 'til we can run this thing safely...typical Things They Wish To Avoid Until Later Rollout Operate Support Monitor
DEV Dev and Ops Working Together Throughout The Entire Development Lifecycle! Requirements Design Build & Test Deploy Maintain & Support OPS Service Strategy (Plan) Service Design (Do) Service Transition (Do) Service Operation (Check) Service Improvement (Act) Determine Service Scope, Offerings, Service Metrics, Service Level, Requirements, Sourcing And Delivery Strategy Design Service And Support Organization, Processes, Technologies, Data And Reporting Infrastructure Build, Test And Deploy IT Service And Support Infrastructure Validating That IT And Business Requirements Can Be Met Execute IT Service And Support Activities To Support Users and Customers Review IT Service And Management Metrics To Proactively Identify Opportunities For Improving IT Services
So What Happens When You Put Development and Operations Together? We re in conceptual data design and we just did a logical database architecture so how much storage you need in GB? What size processor? How about network traffic volumes? SLA objectives?
Service Management Is Now Critical!! Ops needs to provide development with standard services easily consumed Developers Customers or Business End Users Service Catalog IT Services IT Service Management Bridge (Coordinates delivery of services across IT technology silos) Support Services Servers Applications Applications Networks Operations Common Delivery Tools Let s build a bridge to our developers! Common Delivery Processes Service-Based Organization External Suppliers 1 External Suppliers
DevOps Examples Of Services Service Catalog/Portfolio Examples Service Operations Services Service Transition Services Service Design Services Control Services Service Desk Service Monitoring Incident Response Problem Control Request Fulfillment Backup/Restore Mgt Job Schedule Mgt Clock Mgt Startup/Shutdown Mgt File Transfer and Control Mgt Archive Mgt Data Entry Support Report Distribution Request Fulfillment Release Planning and Packaging Service Deployment and Decommission Site Preparation Support Service Validation and Testing Support Training Support Organizational Change Support Knowledge Management Test Lab Management Security Testing Service Catalog Mgt Service Level Mgt Supplier Mgt Operational Planning and Consulting Solution Planning and Development Development Support Operations Capacity Mgt Availability Mgt IT Service Continuity Mgt Process Design Support Lease and License Mgt Service Audit and Reporting Process Mgt Risk Mgt and Audit Support Security Design and Consulting Enterprise Architecture Mgt and Research IT Financial Mgt IT Project Mgt Change Control Configuration and Asset Mgt
Developers Work With Operations To Pick From Sets Of Services Reuse Work To Do Spend Money Outsource It Complexity Design Decision Leverage Existing Support Services Enhance Existing Support Services Purchase New Tools And Support Solutions Provide Support Services Via An Outside Vendor Support Service Cannot Be Addressed By Any Vendor or Tooling Solution Design Impact Almost None - But Must Ensure Standards Are Being Adhered To May Have To Build Additional Scripts, Log Files And Event Handlers To Better Leverage Existing Infrastructure Varies - Depending On Service Solution Scope Varies - But May Need To Provide Input For Service Requirements May Require Design, Build And Development Efforts To Provide Needed Services
DevOps-Impact On Operations Change Management Slow!!! Most changes average 8-15 business days (mostly waiting for approvals!) New Focus: Quickly determine risk and then increase or decrease flow e.g. if the risk is low, let the Change Manager approve it utilize CAB only for higher risk changes Aggressively drive Standard Changes if the change doesn t involve human hands and can be automated, then risk is drastically lower Virtualization, Declarative Configuration Management, and Software Defined Infrastructure is a 2 edge sword changes to virtualized/software defined components is much lower risk while changes to core components (e.g. Hypervisors, infrastructure as code elements) carries much higher risk than normal
DevOps-Impact On Operations New Focus: Release Management Automate Push Of Code From QA To Production Release Units = Total Solution Units no longer dealing with different timetables and releases by technology silo
DevOps-Impact On Operations New Focus: Configuration Management Automate discovery and tracking of configuration changes this is getting too complex to do manually and too critical to ignore Tie configurations to the applications and services they support can t effectively manage applications and services when you don t know the pieces and parts Software naming and configuration management is critical especially with agile practices for continuous integration, release and deployment
DevOps Impact On Operations Supplier Management Consider leveraging suppliers to provide common support services Need to ensure that suppliers do not adversely introduce changes into the production environment The role of IT is changing to focus more on service integration versus engineering and building everything that may be needed!
DEV Dev and Ops Working Together More than just pushing code into production! Requirements Design Build & Test Deploy Maintain & Support OPS Service Strategy (Plan) Service Design (Do) Service Transition (Do) Service Operation (Check) Service Improvement (Act) Determine Service Scope, Offerings, Service Metrics, Service Level, Requirements, Sourcing And Delivery Strategy Design Service And Support Organization, Processes, Technologies, Data And Reporting Infrastructure Build, Test And Deploy IT Service And Support Infrastructure Validating That IT And Business Requirements Can Be Met Execute IT Service And Support Activities To Support Users and Customers Review IT Service And Management Metrics To Proactively Identify Opportunities For Improving IT Services
Strategy Phase Considerations Strategy Design Transition Operation Service Improvement Development Considerations Architecture Considerations ITSM Consideration Solution Model Business Volumes Specialized Support Needs Implementation Timeframes High Level Functional Requirements Development Organization Working Standards Sourcing Partners Solution Architecture Overall Enterprise Fit Standards To Be Used Frameworks To Be Used Opportunities For Reuse High Level Service Requirements Support Capabilities Fit Hosting Partners Delivery Strategy Timeframes Costs: - One Time - Ongoing
Strategy Phase Step By Step Identify Service Ownership and Steering Identify Service Stakeholders Identify Service Development Team Analyze Market, Customers and Needs Develop Perspective, Position, Plans and Patterns Identify Service Value And Objectives Identify Service Critical Success Factors Develop Business Demand Forecasts Develop High Level Service Requirements Identify Service Products and Outcomes Develop Service Descriptions and Features Develop Service Model(s) Define Service Units and Packages Assess Service Delivery Capabilities Identify Standards and Methods Identify Key Service Assets Identify Constraints Identify Sourcing Strategy and Partners Procure and Assign Service Resources Develop High Level Work Plan and Schedule Conduct Service Business Impact Assessment Conduct Service Risk Analysis Identify Organizational Change Strategy Develop Preliminary Cost Estimates Develop Service Pricing and Cost Recovery Strategy Identify Funding Strategy Prepare Service Specification Charter Develop Service Portfolio Entry Develop And Present Business Case Conduct Service Strategy Review Gate
Strategy Phase -Deliverables Support Strategy High Level Solution Architecture Service Requirements Service Demand Forecasts/Estimates Cost Estimates Process Assessment Technology Assessment Organizational Assessment Training Strategy Working Standards Sourcing Partners Strategic Tooling Architecture High Level Deployment Strategy High Level Project Plans
Design Phase Considerations Strategy Design Transition Operation Service Improvement Development Considerations Architecture Considerations ITSM Considerations Application Designs Software Package Customizations Testing Requirements Component Configurations Procurement Requirements Operational Detailed Requirements (i.e.): - Required Alerts - Backup Requirements - Solution Configurations Data Architecture Functional Architecture Technical Architecture (i.e.) - Application - Middleware/Messaging - Network - Transaction - Database Operational Architecture Data Models/Data Design Availability Designs Capacity Designs Service Continuity Designs Security Designs Deployment Designs/Strategy Process Designs Support Roles/Responsibilities Operational Control Tasks Facilities Designs Monitoring Designs Procurement Requirements
Design Phase Step By Step Assemble Design Team, Plans and Schedules Select Design Approaches and Strategies Procure Service Delivery Suppliers and Partners Implement Design Knowledge Bases and Tools Develop Detailed Design Requirements Design Business and Management Process Workflows Design Service Data and Information Elements Design Application Elements Design Request Fulfillment Models Design Quality Measurements and Indicators Design Service Delivery Agreements Design Supplier Interfaces Design Service Delivery Organization Design Service Technical Assets and Configurations Design Networking Elements Design Service Availability Design Service Capacity Design Security Design Management Tooling Elements Design Monitors, Events, Alerts and Thresholds Design Physical Site and Environmental Elements Design Service Operability Elements Design Service Documentation Elements Design Service Continuity Elements Design Service Billing and Chargeback Design Service Communications and Training Elements Design Service Testing Strategies and Plans Design High Level Transition Strategies and Schedules Develop Service Design Package Conduct Service Design Review Gate
Design Phase -Deliverables Technical Architectures Data Architectures Service Descriptions/Catalogs Designed Processes Designed Operational Procedures Designed Tooling Infrastructure Designed Roles/Responsibilities Designed Measurements/Reports Designed Monitoring SLA/OLA Agreements Availability Designs Capacity Estimates Security Designs Physical Facility Designs Service Continuity Designs Supplier Integration Component Configurations Testing Requirements Detailed Transition Plans Detailed Training Plans
Transition Phase -Considerations Strategy Design Transition Operation Service Improvement Development Considerations Architecture Considerations ITSM Considerations Application Coding Package Implementation Migration/Code Libraries Rollout Schedules/Volumes Maintenance Needs Testing and Validation User Acceptance Support Call Lists Early Life Support Validated Architecture Ensured Integration With Rest Of Enterprise Updated Architecture Documentation Migrations/Releases Operational Testing Site Surveys/Preparation User/Support Training Operational Integration: - Service Desk - Technical Support - Operation Control - Facilities Management - Application Management
Transition Phase Step By Step Assemble Transition Team, Plans and Schedules Establish Transition Governance Develop Detailed Transition Plans Conduct Transition Risk Assessment Establish Service Release Policies Finalize Service Acceptance Criteria Build Physical Site Infrastructure Elements Build Service Application and Utility Elements Build Service Management Warranty Elements Build Service Billing and Chargeback Elements Build Service Quality Metrics and Reporting Configure Service For Production Operation Build Service Continuity Elements Implement Service Delivery and Support Organization Finalize Internal and External Support Agreements Document Service Configurations and Knowledge Conduct Service Component Testing Conduct Service System Integration Testing Conduct Service Availability and Security Testing Conduct Service Capacity and Performance Testing Conduct Service Operational Testing Conduct Service Quality and Compliance Reviews Conduct Service User Acceptance Testing Conduct Service and Support Training and Communications Deploy Service Elements Finalize Service Cost Models and Budgets Evaluate Service and Known Errors Implement Service Desk and Early Life Support Conduct Service Transition Review Gate Update Service Catalog and Portfolio
Transition Phase -Deliverables Detailed Transition Plans Operational Scripts Job Schedules Run Books Solution Documentation Configured Security Configured Components Operational Test Results Trained Users Trained Support Staff Solution Known Errors Installed Monitoring Agents Installed Processes Assigned Support Staff Installed Management Tools Site Surveys Facility Build Outs Installed Releases Early Life Support Installed Applications User Acceptance Results Post Implementation Reviews
Operations Phase -Considerations Strategy Design Operation Transition Service Improvement Development Considerations Architecture Considerations ITSM Considerations None at this point Application Management takes over for ongoing application maintenance and support None. Delivering Services Ongoing Support Monitoring/Event Management Maintenance - Applications - IT Infrastructure Execution of planned operational control tasks Incident,/Problem Management Request Fulfillment Access Management
Operations Phase -Deliverables Incident Logs Problem Logs Operational Results Fulfilled Requests Escalated Alerts Shift Turnover Reports Escalated Operational Issues Delivered Services Executed Operational Controls Workforce Management Service Issues Log
Improvement Phase -Considerations Strategy Design Operation Transition Service Improvement Development Considerations Architecture Considerations ITSM Considerations None At This Point Application Management Takes Over For Ongoing Application Improvement Review Of Service And Operational Results To Identify Further Architecture Improvement Opportunities Maintain Architecture While Improvements Are Being Made Collecting Service Measurements Service Quality Reporting Review Of Service And Operational Results To Identify Further Opportunities For Improvements
Improvement Phase -Deliverables CSI Register Service Measurements Process Measurements Technology Measurements Service Quality Reports Service Dashboards Service Improvement Recommendations Service Improvement Plans Service Reviews
Service Operation Service Transition Service Design Service Strategy Continual Service Improvement ITSM Implementation ITSM Organization ITSM Tooling Technologies ITSM Governance Tools and Aids Operations Management ITSM Metrics and Reporting ITSM Books and Articles ITSMLib Games and Simulations Write RandyASteinberg@gmail.com for free access
Rsteinberg@concurrency.com