Peer-to-peer Virtual Private Networks and Applications Renato Jansen Figueiredo Associate Professor Cloud and Autonomic Computing Center/ACIS Lab University of Florida Visiting Researcher at VU
Backdrop Virtual machines in cloud computing On-demand, pay-per-use, user-configurable Federated environments End-to-end Internet connectivity hindered by address space and presence of NATs, firewalls Network virtualization seamlessly connecting virtual machines across multiple providers 2
Rationale Virtualization techniques for decoupling, isolation, multiplexing also apply to networking E.g. VLANs, VPNs However, there are challenges in configuration, deployment, and management Peer-to-peer techniques provide a basis for scalable routing, and self-management Software routers, integration at network end-points enables deployment over existing infrastructure Architecture, design needs to account for connectivity constraints, and support TCP/IP efficiently; optimize for common cases 3
Application Examples Cloud-bursting Run additional worker VMs on a cloud provider Extending enterprise LAN to cloud VMs seamless scheduling, data transfers Federated Inter-cloud environments Multiple private clouds across various institutions Virtual machines can be deployed on different sites and form a distributed virtual private cluster Connecting devices of social network peers Media streaming, file sharing, gaming, 4
Background Talk - Outlook Architecting self-organizing virtual networks Topology, routing, tunneling, addressing, NAT traversal, performance Uses in Grid/cloud and end-user environments Virtual Private Clusters Social VPNs Applications FutureGrid high-throughput computing virtual appliances ConPaaS 5
Resource Virtualization Virtual machines (Xen, VMware, KVM) paved the way to Infrastructure-as-a-Service (IaaS) Computing environment decoupled from physical infrastructure Pay-as-you-go for computing cycles Virtual networks complement virtual machines for increased flexibility and isolation in IaaS VMs must communicate seamlessly regardless of where they are provisioned Traffic isolation; security, resource control 6
Virtual Machines and Networks Virtual Infrastructure V2 V3 V1 VMM + VN Physical Infrastructure Domain B Domain A WAN Domain C 7
Virtual Networks Single infrastructure, many virtual networks E.g. one per user, application, project, social network Each isolated and independently configured Addressing, protocols; authentication, encryption Multiplexing physical network resources Network interfaces, links, switches, routers 8
Network Virtualization Where? Software Virtualized endpoints Software Network Device Network Fabric Network Device (Virtual) machine (Virtual) machine Virtualized Fabric (e.g VLAN, OpenSwitch) 9
Landscape Peer-wise Internet connectivity constrained IPv4 address space limitations; NATs, firewalls Challenges - shared environment Lack of control of networking resources Cannot program routers, switches Public networks privacy is important Often, lack privileged access to underlying resources May be root within a VM, but lacking hypervisor privileges Dynamic creation, configuration and tear-down Complexity of management 10
Technologies and Techniques Amazon VPC: Virtual private network extending from enterprise to resources at a major IaaS commercial cloud OpenFlow: Open switching specification allowing programmable network devices through a forwarding instruction set OpenStack Quantum: Virtual private networking within a private cloud offered by a major open-source IaaS stack ViNe: Inter-cloud, high-performance user-level managed virtual network IP-over-P2P (IPOP) Peer-to-peer, inter-cloud, self-organizing virtual network 11
Example: OpenFlow Towards an open platform foundation supporting Software-Defined Networks (SDN) Interface standardized by Open Networking Foundation (ONF) Board members (as of 6/12): Google, Microsoft, Facebook, Yahoo!, Deutsche Telekom, NTT, Verizon Dozens of members: Citrix, Huawei, Orange, IBM, Dell, HP, Oracle, Goldman-Sachs, Current (as of 6/12): OpenFlow Switch 1.3.0 12
OpenFlow Switch and Controller Controller OpenFlow Protocol Secure Channel Group Table OpenFlow ingress port Add, update, delete Match flow Flow table entry Table Pipeline Flow Table Table miss OpenFlow output port 13
Peer-to-Peer Virtual Networks Overview User-level IP overlays deployable on Internet end resources (software routers, virtual NICs) Why virtual? Hide complexities associated with NAT traversal, IPv4 address space constraints from applications Support unmodified applications Why peer-to-peer? Self-organizing - reduce management complexity and cost Decentralized architecture for scalability and robustness 14
The IP-over-P2P (IPOP) Approach Isolation Virtual address space decoupled from Internet Packets picked, encapsulated, tunneled and delivered within the scope of virtual network Self-organization Overlay topology, routing tables Autonomously deals with joins, leaves, failures Decentralized P2P messaging architecture No global state, no central point of failure Tunnels (UDP, TCP, ), routing Decentralized NAT traversal No need for STUN server infrastructure [IPDPS 2006, Ganguly et al] 15
IPOP: Architecture Overview Unmodified applications Connect(10.10.1.2,80) Application Virtual Router Capture/tunnel, scalable, resilient, self-configuring Routing; object store 10.10.1.1 VNIC Isolated, private virtual address space Wide-area Overlay network Virtual Router VNIC 10.10.1.2 Application
P2P Overlay (Brunet) Bi-directional ring ordered by 160-bit IPOPid s Structured connections: Near : with neighbors Far : across the ring n2 n3 n4 IPOPid n5 n6 Multi-hop path between n1 and n7 n1 n7 Far n12 n8 Near n1 < n2 < n3 <. < n13 < n14 n11 n10 n9
Overlay: Edges and Routing Overlay edges Multiple transports: UDP, TCP, TLS NAT traversal (UDP hole-punching) Greedy routing Deliver to peer closest to destination IPOPid Constant # of edges per node (average k) O((1/k)log 2 (n)) overlay hops On-demand edges Created/trimmed down based on IP communication 18
Creating Overlay Edges CTM request A A s endpoint URIs: tcp://10.5.1.1:3000 (local) udp://128.227.56.9:4433 (NAT learned) Link request Overlay path A sends a Connect-to-me (CTM) request to B s IPOPid Contains all its URIs (UDP/TCP IP:port endpoints) Routed over P2P overlay to B B CTM reply B s endpoint URIs: tcp://172.16.2.1:5000 udp://129.15.56.9:6000 B sends CTM reply with its URIs overlay routed B initiates linking with A Attempts linking with parallel requests to A s URIs 19
NAT Traversal Direct edge between A and B A B Technique for cone UDP NATs: A s link request message to B creates ephemeral state in A s NAT allowing messages from B to pass through NAT (and vice-versa) Overlay: manage keep-alives so NAT mapping holes stay open; re-link if NAT mappings expire 20
Naming and Multiplexing One P2P overlay can multiplex multiple VNs E.g. multiple virtual clusters from different projects IP routing within the scope of a namespace User-provided string identifies IPOP namespace Each IPOP node is configured with a namespace IP-to-P2P address resolution: DHT-Get(namespace:IP) -> IPOPid 21
Managing Virtual IP Addresses Address assignment: static, or dynamic Supports DHCP Store configuration (including base address, mask) on DHT entry bound to namespace DHCP proxy runs on each IPOP node Pick DHCP request Lookup DHCP configuration for namespace Guess an IP address at random within range Attempt to store in DHT; wait for majority to acknowledge; retry upon failure 22
Optimization: On-demand edges At each node: Count IP-over-P2P packets to other nodes When number of packets within an interval exceeds threshold: Initiate connection setup; create edge Trimming on-demand edges no longer in use Overhead involved in connection maintenance 23
Optimization: Tunnel Edges Peers X, Y may not be able to communicate directly if they are behind symmetric NATs X, Y exchange list of neighbor URIs Each attempts to create edge to common intermediary Z to serve as proxy Routing abstracted as regular overlay edge X-Y connected by virtual edge Useful to maintain ring topology in the face of failures (routing outages, symmetric NATs) 24
Implementation IPOP open-source system C# user-level router Tap virtual network device Performance 1GbE physical LAN Latency (ms) Bwidth (Mb/s) Mem (KB) Host 0.27 941 n/a IPOP 0.52 284 38312 IPOP+sec 0.75 55 50976 25
Performance (WAN) Netperf stream native (Mbps) Netperf stream IPOP (Mbps) Netperf RR trans/s native Netperf RR trans/s IPOP EC2/UF EC2GoGrid UF/GoGrid 89.2 35.9 30.2 75.3 19.2 25.7 13.4 11.1 10.0 13.3 10.7 9.8
Access Control IPOP provides core primitives for packet capture/injection and overlay routing How to control which nodes connect to a particular IP namespace? Focus on two approaches: Each peer decides the peers they connect with SocialVPN Peers join groups and agree on a trusted third party as mediator GroupVPN 27
SocialVPN Users now commonly manage relationships to social peers through Online Social Networks Facebook, Google+ Communication hindered by OSN provider APIs, privacy concerns A generic IP network can enable existing and new social network applications But users don t have public IPs, don t want to necessarily open NATs/firewalls to all users Users don t want to configure and discover network services manually 28
Social VPNs Alice's Compute Node Alice's Friend's Compute Node Bob's Compute Node on EC2 OSN IP-over-P2P Tunnel XMPP Alice Bob Carl
Social VPNs From a user s perspective: it s simple My computer gets a virtual network card It connects me directly to my social peers All IP packets: authenticated, encrypted, end-to-end Leverage well-known PKI techniques No configuration besides establishing social links All I need to do to is log in to a web based social network Applications, middleware work as if the computers were on the same local-area network Including multicast-based resource discovery UPnP, mdns 30
Applications Social VPN is not the application It is not tied to an application either It enables applications that are of interest for collaboration Security needed beyond network layer Authenticated end-to-end private IP tunnels provide a foundation Traditional applications Media streaming, desktop sharing, file sharing, cycle sharing Platform for decentralized social network applications Fault-tolerant micro-blogging, private file sharing,.. 31
IPOP Social VPN Internals NAT traversal and routing core Private end-to-end tunnels Peer discovery and certificate exchange XMPP Jabber, Google Facebook APIs (was in first prototype; no longer in the code) Dynamic IP address assignment Facebook: more users than IPv4 24-bit private space Also must avoid conflicts with local private networks, and support mobility 32
Addressing and Mapping 160-bit P2P IDs used for overlay routing Each node generates random P2P ID Node issues a self-signed public key certificate with its P2P identifier; publishes through OSN APIs Certificates of friends nodes are discovered, retrieved, revoked through OSN APIs IPv4 addresses seen by applications Dynamically-generated non-conflicting private subnet Local node and friends nodes are mapped dynamically to addresses within range Naming possible through SocialDNS IP src/dest addresses translated (ports are not) [COPS 2008]
Address Translation Alice's Compute Node Alice's Friend's Compute Node Bob's Compute Node on EC2 Alice Send-to BobP2P Recv-from 0.1 2.2 AliceP2P 3.4 0.1 SVPN: 192.168.0.0/16 Alice: 192.168.0.1 Bob: 192.168.2.2 -> BobP2P SVPN: 172.16.0.0/16 Bob: 172.16.0.1 Alice: 172.16.3.4 -> AliceP2P
35 Group-oriented VPNs Well-suited for collaborative environments for cluster computing Nodes who join a group have peer-wise connectivity to all other nodes Based on public key cryptography Owner of a group is a certificate authority signing GroupVPN certificates Centralized Web-based interface hides lowlevel management from users Users create groups, determine who can join group Certificate signing automated; group membership Certificate revocation lists disseminated via P2P
36 Grid appliance clusters Virtual appliances Encapsulate software environment in image Virtual disk file(s) and virtual hardware configuration The Grid appliance Encapsulates cluster software environments Current examples: Condor, MPI, Hadoop Homogeneous images at each node IPOP/GroupVPN connecting nodes forms a cluster Deploy within or across domains
Grid appliance - virtual clusters Same image, per-group VPNs Condor + Virtual Network GroupVPN Credentials (from Web site) copy Group VPN A Condor worker instantiate Virtual machine Virtual IP - DHCP 10.10.1.1 Repeat Another Condor worker Virtual IP - DHCP 10.10.1.2
38 Grid appliance configuration At the end of GroupVPN initialization: Each node of a private virtual cluster gets a DHCP address on virtual tap interface A barebones cluster Additional configuration required depending on middleware Which node is the Condor negotiator? Hadoop front-end? Which nodes are in the MPI ring? Leverage P2P/IPOP primitives: Distributed hash table Advertise (put namespace,managerip); discover (get namespace) IP multicast discovery over GroupVPN
39 Applications in FutureGrid FutureGrid testbed DAS-like system distributed across US institutions Research, education, development, testing of Grid and cloud computing middleware, applications IaaS partitions: Nimbus, OpenStack, Eucalyptus Virtual networks: ViNe and IPOP Virtual appliances Lower barrier to entry pre-configured environments
40 IPOP + ConPaaS at VU ConPaaS framework/runtime to manage platform-asa-service environments Examples: Web service, task farming service Build upon IaaS primitives to create VMs Integration with IPOP Allow deployments to span across multiple providers (federation; bursting; fault-tolerance) Within VMs - no changes to IaaS stack Isolate data plane communications from public Internet Thilo Kielmann, Guillaume Pierre, Contrail/ConPaaS teams
IPOP + ConPaaS 172.16.10.10 172.16.10.20 ConPaaS applications, IPOP namespaces N2 172.16.10.20 N1 N3 IaaS Providers Private Cloud DAS site WAN EC2 Zone 41
Deployed Systems PlanetLab bootstrap overlays Grid appliance deployments: Archer - ~700-CPU cluster SocialVPN deployments: Thousands of downloads, hundreds of deployed nodes
On-going Work Integration of IPOP with IPsec for dynamicallyprovisioned cloud virtual networks Contrail, ConPaaS Overlay by-pass, integration with OpenFlow software-defined networks IPv6/IPv4 overlays, virtual clusters for highthroughput computing, education Archer (computer architecture) FutureGrid (virtual appliances for education) PRAGMA (Pacific Rim Grid) Unstructured P2P SocialVPN Discover, bootstrap, route through friends 43
Acknowledgments ACIS P2P group (IPOP) Over the years: P. O. Boykin, Heungsik Eom, Arijit Ganguly, Pierre St. Juste, Kyungyong Lee, Yonggang Liu, Girish Venkatasubramanian, David Wolinsky, Jiangyan Xu Vrije Universiteit, ConPaaS team FutureGrid, National Science Foundation Awards 0751112, 0758596, 0910812 44
Thank you For more information and downloads: http://ipop-project.org http://socialvpn.org http://futuregrid.org http://grid-appliance.org 45
46
Related Work There exist several VPN technologies: Enterprise VPNs (e.g. Cisco); Open-source (e.g OpenVPN); Consumer/gaming/SMB (e.g. Hamachi) Not easily applicable to federating cloud resources Proprietary code; difficulty in configuration/management Research work in the context of Grid/cloud computing VNET (Northwestern University), VIOLIN (Purdue University), Private Virtual Cluster (INRIA), ViNe (Tsugawa, Fortes @ UF) Smartsockets @ VU 47
Bootstrapping a New Node Received by left and right neighbors Forwarder MyIPOPid CTM (MyIPOPid) Forms a leaf connection with a public node - forwarder Selected at random from list of bootstrap nodes Sends CTM request addressed to its own IPOPid Received by nearest neighbors Creates structured connections; trims leaf connection 48
IPOP Namespaces D 1 Namespace N1: 10.128.0.0/255.192.0.0 C 1 D 2 C 2 N1 N2 A 1 : 10.129.6.83 IPOPid ARP cache x7 B 1 : 10.129.6.71 IPOP packet x8 x6 N2:10.129.6.71 IPOPid x2 A 2 DHTCreate(N2:A2,x2) x1 x2 DHTCreate(N2, 10.128.0.0/255.192.0.0) DHTLookup(N1:B1) x3 x1 x5 x4 N1:10.129.6.71 IPOPid x1 B 2 49
Motivation: Social DNS Users cannot define domain names used to access their services in VPN settings Dynamic private networks; difficult to keep track of services by IP addresses Objective: A decentralized, naming service that gives individuals the ability to select and share the domain names for their resources with SocialVPN peers Approach: Short names within social context Decentralized architecture where each node runs a local DNS server and communicates via SocialVPN Rank-based name conflict resolution 50
Security End-to-end authentication and encryption IPsec tunneling over IPOP Reuse existing software stack End-to-end security implemented in IPOP RSA priv/pub keys X.509 certificates Point-to-point authentication and encryption TLS edges have been implemented Difficulty to deal with NAT traversal Point-to-point security in IPOP: ongoing work Reuses framework and code base from end-to-end Avoid double-traversal of security stack for performance (e.g. shortcut connections based on IP traffic inspection) 51
Appliance firewall Security Block outgoing packets to physical net Except DHCP, DNS, IPOP s UDP port Confine traffic to within WOW and host-only IPsec or IPOP security With IPsec, kernel/user tools reused unmodified Network routing is P2P, however: Trust can be managed by central CA All intra-wow communication authenticated and end-toend encrypted using X.509-based PKI Private net/netmask 10 lines of IPsec configuration for entire WOW 52
Linking and NAT traversal R:A M:X Outgoing packet to N:Y (hole punched) N:Y S:B Outgoing packet to M:X (hole punched) Exchange each other s NAT assigned IP:port Dropped Src = N:Y Dst = M:X Src = S:B Dst = M:X R:A M:X N:Y Allow S:B Src = R:A Dst = N:Y Src = M:X Dst = N:Y Src = M:X Dst = S:B NAT M 128.227.56.83 NAT N 128.139.156.90 53
Avoiding LAN overheads LAN Router Application NIC Application VNIC Virtual Router Wide-area Overlay network Local Interface NIC Virtual Router Application VNIC
Supporting IPOP Routers Single IPOP router for a (V)LAN Virtual Router TAP Device Avoid need for IPOP software stack on end host VPN Software NIC1 NIC0 IP=10.1.1.2 Eth=A:B:C:D:E:0 Avoid IPOP overhead on LAN communication Internet IP=10.1.1.4 Eth=A:B:C:D:E:2 IP=10.1.1.3 Eth=A:B:C:D:E:1
DHCP Provides address allocation and DNS settings IPOP router keeps a history of allocations and ignores packets destined for them sent within the (V)LAN Virtual Router DHT TAP Device VPN Software NIC1 NIC0 DHCP request IP=10.1.1.2 Eth=A:B:C:D:E:0 Internet IP=10.1.1.4 Eth=A:B:C:D:E:2 IP=10.1.1.3 Eth=A:B:C:D:E:1
ARP Lookup Ethernet address from IP address IPOP router ignores ARP if IP in (V)LAN If destination is not on the LAN, check if such a peer exists in the overlay Virtual Router Reply IPOP router addr. not in router table? DHT TAP Device VPN Software NIC1 NIC0 ARP IP=10.1.1.2 Eth=A:B:C:D:E:0 Local reply Internet IP=10.1.1.4 Eth=A:B:C:D:E:2 IP=10.1.1.3 Eth=A:B:C:D:E:1
Social VPN Prototype Connectivity Each node is given an IP address and domain name Trust Use current social networking systems (XMPP) to bootstrap secure connections with friends Access Control The user locally decides to allow or block another user
Computation offloading Alice's Compute Node Alice's Friend's Compute Node Bob's Compute Node on EC2 Alice can leverage Her own resources to add more computational power to her device and save energy Alice Flexibility in selecting from a collection of trusted compute nodes Bob Carl Cloud provider is now just one more compute node Better QoS by leveraging proximity or social trust
Example: Resource discovery Service discovery time 100 UPnP servers over WAN U. Chicago, UC San Diego, and U. Texas UPnP client located at U. Florida Servers connected to PlanetLab SocialVPN overlay
Resource Discovery Service discovery time U. Texas U. Chicago UC San Diego Service discovery time (ms) min max min max min max 27.0 29.4 45.4 47.6 54.6 57.0 SocialVPN supports unmodified UPnP applications with service discovery time commensurable to WAN latency Wi-Fi setup has longer service discovery time than wired LAN (figure)
Offloading Offloading to PC and EC2 Energy consumption The benefits are compelling at large image sizes Higher power consumption of offloading to Amazon EC2 than offloading to local workstation due to network latency
Grid Appliance / Archer 1: Download appliance 2. Boot appliance: automatically joins Archer pool Free pre-packaged Archer Virtual appliances - run on free VMMs (VMware, VirtualBox, KVM) Archer Global Virtual Network 3. Run architecture simulation jobs on the Archer pool through Condor Portal and Wiki: Community-contributed content: applications, datasets, tutorials Simulators: Simics, SESC, Simplescalar Archer seed resources 300+ cores Fall 2008 System software: Condor scheduler NFS file systems