A New Approach to Network Visibility at UBC. Presented by the Network Management Centre and Wireless Infrastructure Teams

Transcription

1 A New Approach to Network Visibility at UBC Presented by the Network Management Centre and Wireless Infrastructure Teams

2 Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 2

3 Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 3

4 University Services 4

5 Business Requirements High availability High performance Virtualized Secure 5

6 Why is Visibility Needed? Client Experience Application Performance NEED VISIBILITY Data Centre Security 6

7 Challenges Life cycling needed Complex network with multiple paths Highly virtualized infrastructure Budget $$$$ 7

8 Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 8

9 Overview of UBCNet 9

10 Visibility of Physical and Virtual Networks A virtual network spans multiple network devices Collect network information from multiple sources 10

11 Multiple Sources of Network Information Netflow Many devices Collector (Netflow Analyzer, StealthWatch) SNMP Many devices Tool (Statseeker, Intermapper) Logs Many devices Tool (Kibana-Elasticsearch) Real network traffic One or many Tool (WireShark, Cisco NAM, WildPacket, IDS/IPS) 11

12 Traditional Approach Manage large number of tools and span sessions Separated tools and information make analysis difficult 12

13 Need a New Approach Manage much less number of tools and span sessions Single tool and information make analysis much easier 13

14 Why Network Packet Brokers? Many-to-many port mapping for real-time adjustments of packet flow. Filtering of packet data based on the characteristics found in the packet headers. Packet slicing and de-duplication that allows a subset of the full packet data to be passed to the monitoring device. Aggregating multiple packet stream inputs into one larger stream, or balancing one large stream into several smaller streams. Insertion of hardware-based time stamps that monitoring tools can use to take more accurate measurements. (Gartner Analyst Jonah Kowall, April 2012) 14

15 Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 15

16 Current Visibility at UBC 16

17 Future Visibility at UBC 17

18 APCON: Main Panel Network ports (Ingress) Tool ports (Egress) 18

19 APCON: Port Mapping 19

20 APCON: Traffic Filtering 20

21 APCON: Advanced Features 21

22 APCON: Protocol Stripping 22

23 APCON: Blade/Port Status 23

24 Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 24

25 Network Monitoring Tool: Agenda Challenges What is Statseeker? Advantages Use Cases Troubleshooting Proactive Alerting Baseline Traffic Aggregation 25

26 Challenges Limited visibility What s happening on this part of the network? Troubleshooting Is it a network issue? No baseline What is normal? 26

27 What is Statseeker? Statseeker - Commercial product - Charts network statistics including bandwidth, latency, utilization, errors, discards, CPU, memory and temperature. - Threshold and alerting - Syslog 27

28 Statseeker: Advantages Fast! Small footprint 1 VM monitoring over 1000 switch stacks, and 100,000 ports Polls every 60 seconds Keeps data indefinitely with original granularity 28

29 Use Cases Troubleshooting Proactive alerting Baseline 29

30 Troubleshooting Troubleshooting with Network Statistics Does the time of the issue correlate with traffic dips / spikes? Are other ports experiencing the same issue? How about other switches? Track down source of traffic dip / spike Any errors or discards on the ports? 30

31 Example 1 Unicast Storm 31

32 Example 2 High Utilization 32

33 Example 3 Compromised Server 33

34 Example 4 DOS Attack 34

35 Example 5 High Errors 35

36 Proactive Alerting High CPU, interface down, syslog matches 36

37 Baseline Do we need to increase bandwidth on any interfaces? Someone wants to upgrade their uplink from 1 Gbps to 10 Gbps. Do the traffic patterns justify the upgrade? Able to see historical trends, and anticipate growth requirements 37

38 Example: Traffic Utilization Traffic Utilization over 30 days 38

39 Traffic Aggregation Total traffic of multiple interfaces 39

40 Traffic Aggregation 40

41 Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 41

42 Data Analytics with ELK = 42

43 ELK: How? By using free and open source software Elasticsearch database optimized for search Logstash Parse any data Kibana HTML visualization frontend All components are horizontally scalable Current deployment has about 20 Virtual Machines 5 VM s for central syslog, 2 REDIS queues, 2 logstash parsers, and 11 Elasticsearch database servers These VM s handle 100 million log events daily, about 100 gig per day on disk 43

44 ELK: Architecture Logstash Forwarder (LSF) is a lightweight daemon that forwards logs from your application/server to Logstash Logstash gets the log from LSF, or acts as a Central Syslog receiver (udp/514) from other network devices (switches, servers, etc). It sends those logs into a REDIS queue for processing Logstash Parser pulls the logs from REDIS and parses/converts them into a format that can easily be searched by Elasticsearch The Elasticsearch cluster contains dedicated master nodes (esm1-3), client load balancer (esc1-2) and data nodes (es1-6). Each data node has 32GB Ram and 2TB disk. The Kibana3 GUI and Kibana4 beta provide user access to the log data 44

45 ELK: Logstash Input: File, syslog, udp (netflow) Filters: grok, mutate, GeoIP, replace, split, clone Output: Elasticsearch, REDIS, file Many, many Common timestamp format Easy to convert timestamps from various applications, devices, and servers into one standard format Data manipulation All MAC Addresses have the same format. Any MAC s that come in as aabb.ccdd.eeff or aa:bb:cc:dd:ee:ff are converted to aa-bbcc-dd-ee-ff to make searching the different datasets easier 45

46 ELK Dashboard: Wireless user

47 ELK Dashboard: Wireless Overview

48 ELK Dashboard: TACACS

49 Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 49

50 Q&A Any questions? 50

51 Contacts Amy Osman Network Analyst, Network Management Centre Solomon Huang Network Analyst, Network Management Centre Jeremy Cohoe Network Analyst, Wireless Infrastructure Sean Wang Network Architect, Network Management Centre Miranda Chiu Manager, Network Management Centre 51

52 Photo Credits Slide 4: University Services 1. Erhardt, Don, The multi-purpose Franklin Lew Forum at Allard Hall., 2. Baer, Rhoda, Researcher Looking Through Microscope, g/product_data_sheet0900aecd802ff012.html Grigoryan, Arthur, Videoconference classroom, TedxVancouver c/o Maurice Li, TEDxVancouver 2011, UBC Chan Centre, 9. Sistoiv, POS device (Italy), Slide 5: Business Requirements 1. Zammit, Jared, Blue fibre, 2. Samollov, Yuri, System Lock, 52

53 The End Thank you for your interest! 53