Hope is Not A Strategy

Transcription

1 Hope is Not A Strategy Avoiding 5 Common Network Management Mistakes Tim Connelly Director, Systems Engineers

2 In the News. February 18, 2010 TechCrunch crippled for 4 hours Cause: a unscheduled router change December 17, 2009 Blackberry loses ability to send Cause: a routine router maintenance caused unplanned result August 3, Paypal goes down payment services are lost for over 6 hours! Cause: a "back-end router" complicated by a change that took place August 9, Cisco s web site goes down Cause: A human error September 1, Gmail goes down worldwide Cause: A change by an admin intended to improve traffic flow

3 A Few of The Responses I hope it will be much longer than four years before we face a problem like this again. Matt Mullenweg, WordPress Founder on blog outage impacting TechCrunch Something went wrong.these things happen. Crystal Davis, Sprint on the Blackberry outage Is Hope a strategy? Can IT get away with these things happen?

4 Mistake 1 Manually Dealing with Network Change

5 Facts on the Network Infrastructure 60-70% of all network issues are tied to network change Over 70% of day is spent on unplanned work Up to 80% of all IT resources are consumed just to maintain the status quo Over 90% of organizations use significant manual processes for network infrastructure

6 The IT Capability Gap Importance of network/ risk of downtime IT capability gap 0% IT resource/staff

7 The Growing Pain of Manual Processes The misnomer of A quick & easy manual change Easy change is 3-5 minutes per device Access device, log in, check current setting, change, save, verify, hope nothing breaks Doing the math 4 minutes per device X 2.3 changes per week X 275 devices = 2,530 minutes (over 42 hours) Doesn t include human error risk

8 Mistake 2 Don t Understand the Difference between Change Management and Managing Change

9 Change Management Process Request a change Make change/close ticket Review change Schedule ticket Receive approval

10 Change Management Process Successful? Request a change Review change Within policy? Approval Impact neighbors? Schedule ticket Good change? Change/ close ticket

11 Mistake 3 Believing Managing Performance and Managing Change Should Be Separate

12 Performance and Change Are Separate Performance Team Track up/down status Monitor response time Measure usage and utilization Verify service level agreements Change Team Manage network devices Implement changes and configurations Maintain consistency and standards Manage ticket system

13 Only Meet When Problems Occur Only get together when troubleshooting starts Who caused the problem What happened When did the cause start Where is the impact Why is it going on

14 Mistake 4 Inconsistent/Non-Standard Networks

15 Key Driver for Compliance Internal Internal best practices gold standard Designed by IT/networking staff Typically defined on paper and taught Generally accepted by IT ISO, ITIL and custom standards External mandates - governance Required by government or sector Generally disliked by IT Extensive audit and reporting needs PCI, HIPAA, SOX and many more

16 Common Issue Today for Compliance Define Standards on Paper Wait Until Auditor Report Is Needed

17 Five Critical Steps for Compliance Define Standards & Policies Implementation to Device Set Proactive Monitoring & Alerting Violation Remediation & Verification Reporting Up-to-the Minute & Scheduled

18 Mistake 5 Device Centric Views

19 Change on Network Neighbors Typically, changes are focused on individual devices Reviews should consider impact on neighboring devices or along the path Challenges occur with unintended consequences on device neighbors Limited view on impact of change increases risk along multiple devices

20 How to Avoid the 5 Common Mistakes

21 Understanding the Impact of Change Cause & Effect Help user identify hard to find issues See if a change had a positive or negative impact on health Verify if change impacts policy compliance View impact on device neighbors

22 Enforce Compliance and Standardization Build Consistency Over 140 pre-packaged rules Wizard encoding of complex rule logic Proactive alerts for policy violations Built-in remediation Live and historical status, trends and reports

23 Improve Staff Efficiency and Productivity Empower Staff Automate data collection & analysis Reduce manual time and effort Become proactive Remediation options Multi-user roles Views based on individual needs

24 Meeting New Technology Requirements Build Foundation Understand impact of virtualization, cloud computing, data center consolidation Visibility into dynamic world Build consistency/ standardization Scalability/ distributed networks

25 Don t Fall Into the Same Traps as Others Manage Change Take control of #1 cause of network outages Understand impact of change Use automation to achieve goals Control access Automate policy management

26 Hope is Not A Strategy Avoiding 5 Common Network Management Mistakes Tim Connelly Director, Systems, Engineers