Scaling Up Your Network Monitoring: From the Garden Hose to the Fire Hose

Transcription

1 UNIVERSITY OF CALIFORNIA Scaling Up Your Network Monitoring: From the Garden Hose to the Fire Hose Vincent Stoffer Cyber Security Engineer Technology Exchange October 28, 2014

2 Agenda Intro / overview The problem Device roundup and review Cool new stuff Discussion / Questions

3 Overview Lawrence Berkeley National Laboratory Located in Berkeley, CA "Bringing science solutions to the world" Unclassified DoE research facility operated by University of California Functions much like a research university

4

5

6 Computing overview ~5000 users ~10,000 hosts Distributed computing resources Many guests and visitors Open network to enable collaboration and research

7 The (scaling) problem Orders of magnitude changes in network speeds/bandwidth create big issues for network monitoring What s driving these changes?

8 Courtesy Greg Bell, ESnet

9 Courtesy Greg Bell, ESnet

10 All of that means transitions <1G to 1G 1G to 10G 10G to 40G/100G These transitions mean changing more than network equipment!

11 From 1G to infinity 1G is easy 1-10G is mostly a solved problem >10G is still evolving

12 Monitoring Pipeline Input Tapping Aggregation & Load-balancing Filtering Output Analysis Bulk packet capture Filtering

13

14 Aggregation/load balancing Commercial appliance vendors High performance Custom ASICs Flexible High cost per port

15 Apcons, 10G monitor devices 2007

16 What is Bro? Not your typical IDS/IPS A monitoring platform A standalone network monitor A programmable framework An ecosystem

17

18

19 Everything running smooth Average traffic 1-3 Gbps Peaks to 6-7 Gbps There will always be some amount of packet loss, try to minimize Then...

20 LBLnet redesign 100G border Science DMZ Redundant border routers New distribution layer routers All dual connected

21 New monitoring diagram

22 100G Berkeley Lab approach Duplicate our setup on 10G Moving from duplication to advanced aggregation New device needed

23 100G Device wish list Filtering at ingress & egress Port speed agnostic Aggregation, symmetric loadbalancing No oversubscription limits API for dynamic filtering/shunting

24 100G Device wish list cont d Filtering for arbitrary IP headers / TCP flags Every port can be input/output Create port groups Send output to load-balanced groups and single ports IPv6 support

25 100G Monitoring device options Commercial / Appliance Commodity network (proprietary / hybrid) Commodity network + SDN Roll your own

26 Appliance vendor roundup Vendor Product 100G? Tested? Pros Cons Gigamon HD series Yes No Good feedback Cost! cpacket cvue No Not at 100G LBL reference Cost Endace/ Emulex Endace Access Yes Yes 2 devices, filtering, cost Form factor Others: VSS, IXIA/Anue/Netopics, Apcon,???

27

28 The new hope...delivered! Commodity network vendors SDN/Openflow or tap aggregation code (distribution, telemetry, DANZ, etc.) Lower cost per port Massively scalable

29 Network vendor roundup Vendor Model 100G support? Covers wish list? Pros Cons Arista 7150 LANZ (7280) Yes, with 2nd device Yes API, GUI, SDN 2 devices, IPv6 Brocade MLXe Yes Telemetry Yes Cost, SDN No GUI or API, lower density Cisco Nexxus? Monitor manager Unknown, Cost? not tested Yes Cisco

30

31

32 SDN / Openflow We have not tested yet Hoping to try on Arista / Brocade Advantages over native feature sets? New apps like...

33 Scipass New project built off lessons learned with IU s Flowscale SciPass is an OpenFlow application designed to help network security scale to 100Gbps Wednesday 1:30 session

34 We chose Arista Flexible interface including GUI High density - 6 port 100G line card! Easy to use API dynamic shunting! Relatively low cost Lots of peers using

35

36

37 Output Filtering Analysis Ethernet cards Bro Packet capture

38 Filtering Elephant flows Control traffic Exclusions (IP pairs, netblocks, ports/protocols) Research networks / affiliates Resnet?

39 Filtering cont d Dynamic via Bro near real time via API (Arista) or scripting holy grail

40 Dumbno Python program for shunting Written by Justin Azoff Uses Arista JSON API to limit to control packets Bro s reaction framework feeds in data Connection details are preserved

41 Dumbno cont d Much more simple than SDN but not as flexible Small amount of code Limited number of ACLs for now Let Bro use the force

42

43 Network cards - Intel pf_ring (LibDNA, zero copy) direct memory access to network hardware high throughput supports multiple tools

44 Network cards - Myricon Sniffer10G Support for Linux, FreeBSD Myricom 10G cards only Supports only one tool in 2.0 (multiple tools in 3.0) Company/IP in some flux

45 Network cards - netmap Framework for high speed packet capture Kernel module for Linux and FreeBSD Will be testing soon as alternative to Myricom

46 Bro Packet bricks Linux/FreeBSD traffic steering daemon based on netmap Load-balancing Duplication Filtering to multiple apps Starting to test

47 Blocking Dynamic blocking via ACLD All our security tools feed data Nullroutes and ACLs on Border routers No interference with science

48

49 Questions / Discussion Thank you! vstoffer@lbl.gov security@lbl.gov

50 References Arista - cpacket - Brocade - Endace - Cisco - SciPass - Dumbno - pf_ring - Myricom - Netmap - Packetbricks - Bro -