Best Practices in Network and System Administration

Transcription

1 Best Practices in Network and System Administration Dónal Cunningham

2 Outline Some notes on Infrastructure 10 rules to live by Network Administration System Administration Reading list/training Info

3 Infrastructure I Space Cabinets? Cages? Shared keys? Power AC only? 220V only? UPS area or in-cab? Testing? UPS death? Air Conditioning Fire supression Should people be here?

4 Infrastructure II Cable routing Label EVERYTHING (but don t trust labels blindly) Access (diversity see later)

5 Rule 1 - Be Good Citizens Visibility Ticketing system Updates must propagate outside your group Know your metrics User perception (quick response) CTO perception Partner perception Remember! System administrators and network engineers manage systems and networks on behalf of other people

6 Rule 2 - Monitor your Systems Status Establish baselines Watch trends Use the right tools for the jobs Use the right tools for your team Start in a known state!

7 Routers Link traffic Capacity CPU Memory Environmental ACL hits, BGP routes...

8 Networks Reachability Ping Traceroute Routing loops Latency Directly affects end-user perception

9 Systems Disk CPU Memory Environment Services

10 Rule 3 Perform Disaster Recovery Planning Things break. All the time. Quis custodiet ipsos custodes? If your monitoring system breaks, who will notice? Who will care? Timestamps are essential for correlation NTP is your friend NO SINGLE POINTS OF FAILURE

11 3a - Networks Redundant paths Dynamic routing minimal human intervention Spares (GBICs and cables, too ) Know your S.L.A.

12 3b - Systems Load-balancing DNS round-robin F5/Cisco Director/Resonate Global Dispatch Redundancy of service MX backups Leaf nodes should cache

13 3c - Backups No, really. That s it.

14 3d - Backups USELESS unless you have tested restores!

15 3e - Backups Remember no single points of failure? This goes for backups, too! Media fails Media devices fail Networks fail Try restoring on a different system

16 Rule 4 - It s not done until it s documented YOU are the single point of failure! You do want to go on holiday sometime, right? If not, see Rule 9

17 4a - Change Control Peer review Show others how you think Shows people what s coming Catches typos Revision Control Systems Roll-back. Say it again. Doesn t it sound good?

18 Rule 5 Establish Procedures Consistency Reproducibility ISO 9001 is all about procedures Helps to implement Rule 4 Peer review

19 Rule 6 Practise Defence in Depth Not all eggs in one basket Heterogeneity quis custodiet ipsos custodes? (Monitoring systems can fail too) You get time to react between layers Some are more important that others

20 6a - Defence in Depth Software updates OS and applications All software is buggy. Get over it. Firewall Can give false sense of security Misconfigured? Worse than no firewall. Monitor your network, too (IDS, honeypot) Internal more likely than external

21 Rule 7 - It s not done until it s tested Software installation is a risk Yes, patches too! Test systems Must the software updates be applied right now? Automate your testing, if possible

22 Rule 8 - Learn from Others Don t re-invent the wheel Save yourself time Save yourself money Mailing lists SAGE and local groups NANOG Conferences

23 8a - Other sources Vendors Sometimes they hire smart people FAQs Search engines White papers Books Articles

24 Rule 9 - Learn to Relax The Big Blue Room The most important metric - your family! DELEGATE! Go for a pint (beer, blackcurrant, whatever) Nothing messier than an exploding sysadmin...

25 Rule 10 - Non scriptum, non est If it ain t written down, it never happened Acceptable Use Policies Have all users signed them? ALL users? Including the sysadmins? Can t have perception of multi-tier system Sometimes you have to fire technical people, too

26 Rule 11 Learn how to count to 10 Thank you. Questions?