A New Approach to Network Visibility at UBC Presented by the Network Management Centre and Wireless Infrastructure Teams
Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 2
Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 3
University Services 4
Business Requirements High availability High performance Virtualized Secure 5
Why is Visibility Needed? Client Experience Application Performance NEED VISIBILITY Data Centre Security 6
Challenges Life cycling needed Complex network with multiple paths Highly virtualized infrastructure Budget $$$$ 7
Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 8
Overview of UBCNet 9
Visibility of Physical and Virtual Networks A virtual network spans multiple network devices Collect network information from multiple sources 10
Multiple Sources of Network Information Netflow Many devices Collector (Netflow Analyzer, StealthWatch) SNMP Many devices Tool (Statseeker, Intermapper) Logs Many devices Tool (Kibana-Elasticsearch) Real network traffic One or many Tool (WireShark, Cisco NAM, WildPacket, IDS/IPS) 11
Traditional Approach Manage large number of tools and span sessions Separated tools and information make analysis difficult 12
Need a New Approach Manage much less number of tools and span sessions Single tool and information make analysis much easier 13
Why Network Packet Brokers? Many-to-many port mapping for real-time adjustments of packet flow. Filtering of packet data based on the characteristics found in the packet headers. Packet slicing and de-duplication that allows a subset of the full packet data to be passed to the monitoring device. Aggregating multiple packet stream inputs into one larger stream, or balancing one large stream into several smaller streams. Insertion of hardware-based time stamps that monitoring tools can use to take more accurate measurements. (Gartner Analyst Jonah Kowall, April 2012) 14
Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 15
Current Visibility at UBC 16
Future Visibility at UBC 17
APCON: Main Panel Network ports (Ingress) Tool ports (Egress) 18
APCON: Port Mapping 19
APCON: Traffic Filtering 20
APCON: Advanced Features 21
APCON: Protocol Stripping 22
APCON: Blade/Port Status 23
Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 24
Network Monitoring Tool: Agenda Challenges What is Statseeker? Advantages Use Cases Troubleshooting Proactive Alerting Baseline Traffic Aggregation 25
Challenges Limited visibility What s happening on this part of the network? Troubleshooting Is it a network issue? No baseline What is normal? 26
What is Statseeker? Statseeker - Commercial product - Charts network statistics including bandwidth, latency, utilization, errors, discards, CPU, memory and temperature. - Threshold and alerting - Syslog 27
Statseeker: Advantages Fast! Small footprint 1 VM monitoring over 1000 switch stacks, and 100,000 ports Polls every 60 seconds Keeps data indefinitely with original granularity 28
Use Cases Troubleshooting Proactive alerting Baseline 29
Troubleshooting Troubleshooting with Network Statistics Does the time of the issue correlate with traffic dips / spikes? Are other ports experiencing the same issue? How about other switches? Track down source of traffic dip / spike Any errors or discards on the ports? 30
Example 1 Unicast Storm 31
Example 2 High Utilization 32
Example 3 Compromised Server 33
Example 4 DOS Attack 34
Example 5 High Errors 35
Proactive Alerting High CPU, interface down, syslog matches 36
Baseline Do we need to increase bandwidth on any interfaces? Someone wants to upgrade their uplink from 1 Gbps to 10 Gbps. Do the traffic patterns justify the upgrade? Able to see historical trends, and anticipate growth requirements 37
Example: Traffic Utilization Traffic Utilization over 30 days 38
Traffic Aggregation Total traffic of multiple interfaces 39
Traffic Aggregation 40
Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 41
Data Analytics with ELK = 42
ELK: How? By using free and open source software Elasticsearch database optimized for search Logstash Parse any data Kibana HTML visualization frontend All components are horizontally scalable Current deployment has about 20 Virtual Machines 5 VM s for central syslog, 2 REDIS queues, 2 logstash parsers, and 11 Elasticsearch database servers These VM s handle 100 million log events daily, about 100 gig per day on disk 43
ELK: Architecture Logstash Forwarder (LSF) is a lightweight daemon that forwards logs from your application/server to Logstash Logstash gets the log from LSF, or acts as a Central Syslog receiver (udp/514) from other network devices (switches, servers, etc). It sends those logs into a REDIS queue for processing Logstash Parser pulls the logs from REDIS and parses/converts them into a format that can easily be searched by Elasticsearch The Elasticsearch cluster contains dedicated master nodes (esm1-3), client load balancer (esc1-2) and data nodes (es1-6). Each data node has 32GB Ram and 2TB disk. The Kibana3 GUI and Kibana4 beta provide user access to the log data 44
ELK: Logstash Input: File, syslog, udp (netflow) Filters: grok, mutate, GeoIP, replace, split, clone Output: Elasticsearch, REDIS, file Many, many more @ https://github.com/logstash-plugins Common timestamp format Easy to convert timestamps from various applications, devices, and servers into one standard format Data manipulation All MAC Addresses have the same format. Any MAC s that come in as aabb.ccdd.eeff or aa:bb:cc:dd:ee:ff are converted to aa-bbcc-dd-ee-ff to make searching the different datasets easier 45
ELK Dashboard: Wireless user
ELK Dashboard: Wireless Overview
ELK Dashboard: TACACS
Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring Tool Data Analytics and Visualization Tool Q&A 49
Q&A Any questions? 50
Contacts Amy Osman Network Analyst, Network Management Centre amy.osman@ubc.ca Solomon Huang Network Analyst, Network Management Centre solomon.huang@ubc.ca Jeremy Cohoe Network Analyst, Wireless Infrastructure jeremy.cohoe@ubc.ca Sean Wang Network Architect, Network Management Centre sean.wang@ubc.ca Miranda Chiu Manager, Network Management Centre miranda.chiu@ubc.ca 51
Photo Credits Slide 4: University Services 1. Erhardt, Don, The multi-purpose Franklin Lew Forum at Allard Hall., http://en.wikipedia.org/wiki/allard_hall 2. Baer, Rhoda, Researcher Looking Through Microscope, http://commons.wikimedia.org/wiki/file:researcher_looking_through_microscope.jpg 3. http://www.amsrentsline.com/vancouver-bc/apartment/shared-ubc-campus-gallery26922 4. http://www.cisco.com/c/en/us/products/collateral/collaboration-endpoints/unified-ip-phone- 7941g/product_data_sheet0900aecd802ff012.html 5. http://moementum.com/work-smart-blog/six-thoughts-on-curbing-the-disease-of-being-busy/ 6. Grigoryan, Arthur, Videoconference classroom, http://commons.wikimedia.org/wiki/file:videoconference_classroom.jpg 7. https://it.ubc.ca/projects/new-university-data-centre-udc 8. TedxVancouver c/o Maurice Li, TEDxVancouver 2011, UBC Chan Centre, http://commons.wikimedia.org/wiki/file:tedxvancouver_2011,_ubc_chan_centre.jpg 9. Sistoiv, POS device (Italy), http://commons.wikimedia.org/wiki/file:pos_device.jpg Slide 5: Business Requirements 1. Zammit, Jared, Blue fibre, http://www.jisc.ac.uk/network/connectivity 2. Samollov, Yuri, System Lock, https://www.flickr.com/photos/110751683@n02/13334048894/ 52
The End Thank you for your interest! 53