Heartbleed....and why yours should, too

Transcription

1 Heartbleed...and why yours should, too

2 You are in the right session _ This is an emergency service announcement _ Due to events that transpired on Tuesday _ I thought it d be good to have some info OSDC

3 About me _ Dr. Christopher Kunz _ Studied CompSci in Hannover, PhD in 2012 _ Works as a hoster for 15 years _ Some admin experience _ Used to do a lot of PHP _ Author, PHP- Sicherheit, ed. 1-3 _ And don t get me started about swords! OSDC

4 About filoo _ hqps:// _ Quickly- growing hosvng company _ Data center in Frankfurt, Germany _ Developed own IaaS middleware _ QEMU/KVM, OVS, Ceph _ Offer hosvng, co- locavon, cloud services _ 100% subsidiary of Thomas- Krenn.AG _ Visit their booth! OSDC

5 Heartbleed in a nutshell _ A bug with a cute name _...and not so cute effects _ Pre- auth, pre- logging universal TLS/SSL bug _ Introduced in OpenSSL 1.0.1a (2012) _ Allows to make 64kb memory dumps of the server s memory OSDC

6 Wait. What? _ Yes, remote memory dumps _ Due to an unchecked buffer length, a TLS enabled server may dump memory contents to the client _ Limit of 64k per reply _ MulVple replies possible _ Memdump may contain... _ URLs and GET / POST variables _ Random excerpts from whatever _ Source code of scripts/whatever else _ SSL cervficate private keys OSDC

7 About DTLS heartbeats _ RFC 6520, Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS) Heartbeat Extension _ Provides a heartbeat for TLS (TCP) and DTLS (mostly UDP) sessions _ Intended to add stability to unstable connecvons and prevent renegovavons _ Implemented in OpenSSL as part of a PhD thesis _ Patch commiqed Dec 15, OSDC

8 What this bug is not _ This is not a crypto bug _ At least not in its primary funcvon _ This is not a fully arbitrary mem disclosure _ Only memory belonging to aqacked daemon can be dumped _ This is not a remote root hole _ Hence the relavvely low CVE score of OSDC

9 Anatomy of the bug 1 struct { HeartbeatMessageType type; uint16 payload_length; opaque payload[heartbeatmessage.payload_length]; opaque padding[padding_length]; } HeartbeatMessage; _ From RFC6520: _ payload_length: The length of the payload. _ payload: The payload consists of arbitrary content OSDC

10 Anatomy of the bug 2 _ ssl/d1_both.c, line 1474+: buffer = OPENSSL_malloc( payload + padding); bp = buffer; [..] memcpy(bp, pl, payload); _ From: d069b4c b02a22116ad75f822b OSDC

11 Anatomy of the bug _ The heartbeat extension allocates payload+19 bytes of memory _ Copies pl bytes of arbitrary user- supplied data payload via memcpy() to construct response _ Client sets pl to _ Client sends only 1 byte of data in payload _ Response contains 1 byte of client- supplied payload _...and 64K of RAM from the memcpy() call _ Analysis in: hqp://blog.existenvalize.com/diagnosis- of- the- openssl- heartbleed- bug.html OSDC

12 Test vulnerability _ Python script at: hqps://gist.github.com/takeshixx/ _ Can test any SSL/TLS enabled TCP service _ Has support for StartTLS (- s opvon) _ Conveniently dumps 64kb of memory for you 00d0: F #...3A1% 00e0: F 6D 6F C%22_mode%22%3A 00f0: A 73 6F 6E %22json%22%2C% : 5F F _id%22%3a%22p_ : F %22%2C%22_ 0120: 63 6F 6E E container%22%3a0 0130: F F 6E %2C%22_action% : %3A%22view%22%2C OSDC

13 Memdump _ From: hqps://twiqer.com/markloman/status/ OSDC

14 Memdump _ Memory contents is non- determinisvc _ SomeVmes excivng, mostly boring _ while true do python hb-test.py yahoo.com grep -C 2 login >> /tmp/out; sleep 1; done" _ Profit! OSDC

15 Detect exploitation _ No logging on the machine _ All exploitavon is pre- logging, pre- applicavon _ IDS vendors are pushing out signatures already OSDC

16 Affected services _ Above all, SSL- enabled web servers _ Any that uses OpenSSL, anyway _ Mail servers _ IMAP over SSL, POP over SSL, SMTP over SSL, StartTLS _ VPN tunnels _ OpenVPN when using cert auth (maybe?) _ PotenVally others _ IRC servers, XMPP, FTP over TLS _ Android is vulnerable _ OpenSSH is not vulnerable OSDC

17 Linux versions affected _ OpenSSL a thru f _ Debian Wheezy, Jessie, Sid _ Fixed for Wheezy & Sid _ Ubuntu 10.04, 12.04, 12.10, 13.10, _ Fixed packages exist _ RHEL 6 _ Patch exists _ And all others that ship OpenSSL _ Clients are also vulnerable! OSDC

18 Other affected stuff _ Cisco devices: We use Cisco SSL which is not OpenSSL. ; SSL VPN products potenvally affected _ Juniper has released fixes for their SSL VPN, none for J- Web etc. yet _ Big IP? Kemp? Fritz.Box? Your home NAS? _ More info (hopefully) here: hqp:// vuls/byvendor?searchview&query=field +Reference=720951&SearchOrder= OSDC

19 Mitigation & cleanup _ First, upgrade to fixed openssl _ apt- get install openssl libssl _ Next, restart all services that load old lib _ Use checkrestart or lsof n grep DEL grep ssl _ If you use stavc binaries, recompile everything _ If you use Google s mod_spdy on Apache2.2, don t _ It has its own stavcally linked mod_ssl which is shamefully out of date OSDC

20 What about certs? _ It is possible that privkeys have leaked _ If so, you need to revoke&reissue certs _ Some CAs offer free reissue _ If you don t have PFS, you have a problem _ AQackers who sniffed your traffic might be able to decode it OSDC

21 Thank you _.Do not despair, there is hope! hqp://xkcd.com/1353/ _...and now, back to our regular scheduled programme! OSDC

22 Software-defined Networking In an open-source cloud

23 Agenda _ High- Level overview: What is this about? _ The use case virtualized networks for IaaS _ Intro to OpenVSwitch _ How- to: Deploy OpenVSwitch _ Frontnet, Backnet, public net _ Firewalling _ Tying it all together OSDC

24 So what s the hype? _ Sovware- Defined Networking is the hype _ I m not good with hype _ Networking is decoupled from bare metal _ EssenVally you virtualize parts of your network _ Control and data plane are decoupled _ Many vendors jumped on the train _ HP, Cisco, VMWare, you name it OSDC

25 OpenFlow _ ImperaVve control _ Switches are dumb they only forward according to rules _ OpenFlow controllers make the rules _ First packet of each type is sent thru OpenFlow controller _ Subsequent ones go directly through switch OSDC

26 OpFlex _ Cisco s answer to OpenFlow _ Other vendors on board: Citrix, MSFT, RHAT, Canonical _ Not on board: J, HP, Huawei, vmware _ Balance intelligence between switch and controller _ DeclaraVve control ; just declare how you want it and the switch interprets that rule _ IETF proposed standard _ Drav- smith- opflex _ Open APIs _ AltruisVc goal: Eliminate SPOF (the controller) _ EgoisVc goal: Sell smarter (=$++) switches OSDC

27 The OSS Contender _ OpenVSwitch _ Openvswitch.org _ Open Source _ Apache 2.0 license, non- viral _ GPLv2 _ MulVlayer (2,3) virtual switch _ Supports lots of interesvng features _ VLANs, Ne{low, sflow, LACP, filtering, OSDC

28 OVS Overview Control Cluster Off-box ovsdb-server ovs-vswitchd User Kernel Management Protocol (6632/TCP) OpenFlow (6633/TCP) Netlink OVS Kernel Module _ Shamelessly lived from [1] OSDC

29 OSVDB _ Database holds configuravon items _ DefiniVons for bridges, tunnels, interfaces _ Controller addresses _ ConfiguraVon is reboot- safe _ Custom database system, not MySQLiteMongoDB _ Speaks custom protocol (OSVDB) _ Log based _ osvdb- tool show- log shows all changes _ Nivy for debug / change management! OSDC

30 How ovs works _ ImperaVve control _ All intelligence is in the controller _ Data path only carries out instrucvons _ Data Path _ Kernel module _ Licensed under GPLv2 _ Controller _ Lives in userland _ Licensed under Apache OSDC

31 Flow flow _ Everything is a flow _ CombinaVon of input port, VLAN, MAC, IP, TCP/UDP port OSDC

32 OVS management _ Command- line tools _ Ovs- vsctl for switch management _ Ovs- ofctl for flow management _ Ovsdb- tool for database management OSDC

33 What s our angle here? _ filoo is a hoster. _ We host VMs. _ VMs need networking. _ See where this goes? OSDC

34 What we wanted _ Internet- facing front- net interface _ Private LAN for VMs _ VM isolavon _ Firewalling _ Traffic shaping _ Fine- grained accounvng _ Live migravon OSDC

35 Overview - physical Front- end switch Back- end switch OSDC

36 Overview - virtual Firewall Firewall Firewall OSDC

37 Overview OVS stack OVS OVS OVS OSDC

38 Let s get started _ We usually compile ovs ourselves _ There are also packages in apt _ Those might work or not _ Download & compile OVS _ Latest stable: 2.1.0, latest LTS: _./boot.sh &&./configure && make && make install _ Kernel module from 3.3+ _ Enable in Kernel Networking - > OpVons - > Open Vswitch _ modprobe openvswitch OSDC

39 Let s get started 2 _ Set up ovs db _ Ovsdb- tool create conf.db vswitch.ovsschema _ Conf.db is in /usr/localetc/openvswitch _ /usr/src/openvswitch /vswitchd/vswitch.ovsschema _ Make sure ovs- vswitchd and ovsdb- server start before networking _ Add startup entries to rc.local _ Remove networking from rc.d _ start networking in rc.local OSDC

40 Initial bridges _ Front- net vlan: 199 _ Same procedure for back- net VLAN _ Add bridges _ ovs- vsctl add- br vmbr1 _ ovs- vsctl add- port vmbr1 vlan199 tag=199 _ ovs- vsctl set interface vlan199 type=internal _ Log in via IPMI _ ovs- vsctl add- port vmbr1 eth1 _ Machine is offline now _ Modify physical switching OSDC

41 VM networking _ We use KVM/QEMU _ Add the TAP interface _ /sbin/ip tuntap add dev tap1i0d0 mode tap user fcms _ qemu- system- x86_ device rtl8139,mac=00:f1:70:00:00:10,netdev=vlan0d0 - netdev type=tap,id=vlan0d0,ifname=tap1i0d0 _ Bring up the port _ /usr/local/bin/ovs- vsctl add- port vmbr0 tap1i0d0 199 other_config:stp- enable=false OSDC

42 From TAP to port to flow _ We have a tap interface tap1i0d0 _ Find the corresponding bridge port: _ ovs- ofctl show vmbr0 grep tap1i0d0 _ 1820(tap1i0d0): addr:fa:7a:67:e3:5d: _ Now we have a port number: 1820 _ We use this port for flow management OSDC

43 Multiple interfaces _ Add more TAP interfaces _ Assign one VLAN per customer _ Internal network across VMs on same node _ Make VLAN known on inter- node switches _ Via whatever switch automavon you have _ Cross- node internal networking _ VLAN limits apply hard cut at ~4090 _ Overlay networks to the rescue OSDC

44 Prevent MAC spoofing _ PORT=1820 We know this MAC _ ovs- ofctl add- flow vmbr0 "in_port="${port}" arp idle_vmeout=0 because priority=39500 we control acvon=resubmit("$ {PORT}",2) the hypervisor! _ ovs- ofctl add- flow vmbr0 "in_port="${port}" table=2 arp priority=200 idle_vmeout=0 arp_sha=00:f1:70:00:00:10 nw_src= acvon=normal" We know this _ ovs- ofctl add- flow vmbr0 "in_port="${port}" table=2 address too! priority=100 idle_vmeout=0 acvon=drop" OSDC

45 Caveats for MAC/ARP _ SomeVmes you want customers to spoof _ HA soluvons that switch cluster IP addresses _ You can cater for this in case you know the corresponding MACs _ Assign sequenval MACs and wildcard _ Or set specific rules _ OpVonal HA feature for VMs _ Never allow customers to wildcard here! OSDC

46 Firewalling with flows _ ovs- ofctl add- flow vmbr0 "in_port="${port}" table=1 tcp idle_vmeout=0 nw_dst= /32 nw_src= /32 tp_dst="80" priority=38000 acvon=drop _ From _ To _ Port 80 _ Drop OSDC

47 Port ranges _ ovs- ofctl add- flow vmbr0 "in_port="${port}" table=1 tcp idle_vmeout=0 nw_src= /32 nw_dst= /24 tp_src="0x05e8/0xfffc" priority=37960 acvon=drop _ Source _ DesVnaVon /24 _ Source port = 0x05E8/0xFFFC _ 0x05E8/0xFFFC = 1512/65532 _ Port _ OVS 1.11 supports Megaflows, i.e universal wildcarding OSDC

48 Default accept _ ovs- ofctl add- flow vmbr0 "in_port="${port}" table=1 priority=100 acvon=normal _ Fallthru rule _ Match everything else OSDC

49 Accounting _ We grab interface counters from the tap interfaces _ You can also use Ne{low/sFlow or ipfix _ We didn t go there yet, experiences welcome OSDC

50 Shaping _ Simple shaping: _ ovs- vsctl set Interface tap0 ingress_policing_rate= _ ovs- vsctl set Interface tap0 ingress_policing_burst=1000 _ QoS policies: _ ovs- vsctl set port eth1 qos=@newqos \ id=@newqos create qos type=linux- htb \ other- config:max- rate= queues=0=@q0,1=@q1 \ _ We don t do QoS policies, shaping works mostly as intended OSDC

51 Live migration _ We don t actually do OVS s own live migravon _ Start VM on target host in suspend- to- RAM mode _ Stop VM on losing host; down interface _ Resume VM on target host _ There are live migravon mechanisms in OVS _ L2 based _ Inter- OVS GRE tunnel _ Honestly, I have no clue OSDC

52 Thank you _ I hope you learned something _ If not, I hope you had a laugh at my expense _ If neither, I m really sorry. Beer? _ QuesVons? OSDC

53 Literature _ [1] hqp://openvswitch.org/slides/ OpenStack pdf OVS Deep Dive _ OVS IntroducVon: hqp://horms.net/projects/ openvswitch/ /openvswitch.en.pdf OSDC