Large-Scale Distributed Computing VL 2010S. Thomas Perl, Stefan Kögl. Bandwidth Allocation in Clouds. Bandwidth Allocation in Clouds

Size: px

Start display at page:

Download "Large-Scale Distributed Computing 187.271 VL 2010S. Thomas Perl, Stefan Kögl. Bandwidth Allocation in Clouds. Bandwidth Allocation in Clouds"

Logan Walsh
8 years ago
Views:

1 Large-Scale Distributed Computing VL 2010S Thomas Perl, Stefan Kögl

2 Over view Workflow Motivation Implementation Test Setting Evaluation Findings Further work

3 Workflow 1. Negotiate parameters in SLAs 2. Allocate bandwidth to applications 3. Monitor per-application bandwidth usage 4. Adjust excessive bandwidth usage

4 Motivation Usual approach for monitoring: per network interface in/out per service (e.g. all traffic to port 80, ) per device (aggregated network interfaces) No per-application monitoring solutions Implement per-process monitoring

5 Implementation (bwmon) Bandwidth Monitor per Application Two Methods for Monitoring Group Processes to Applications SLA Parameters as Input Callbacks for Notifications

6 Components of bwmon

7 Implementation Comparison of methods Monitor Aggregates Data from /proc/net/ip_conntrack Accounting done by Kernel Linux-only Non-invasive Handles arbitrary Connections Pipe Accounts Data passed through it Accounting happens in User-Space Portable Invasive Handles one Connection per Instance

8 Implementation Agg r e gator Module Receives Data from Monitors / Pipes Groups Processes to Applications Receives SLA Parameters Issues Notifications

9 Implementation Example configur ation

10 Test Setting Intel Core 2 DUO 2.53 Ghz 4 GB RAM Linux Kernel Mbit fully-switched LAN 8 Mbit DSL Internet downlink (768k up) HTTP server: lighttpd bwmon running on Python 2.6.5

11 Evaluation Simulated Browser Session Pre-generated URL list of random URLs Scripted browser using WebKit's WebView Requesting URLs in list (+ assets) for 30 Minutes Same Traffic Pattern as on a Web-Server Measure Impact of Monitoring CPU: 303s User / 50s System Memory: ~25kB per monitored Process Impact independant of actual traffic

12 Evaluation HTTP File Transfer Create 100 MB file (from /dev/random) Randomness facilitates bad compression Multiple measurements (4 sessions/variant) Findings No change in total transfer time (wall time) Double User-Space CPU-Time Additional User- and Kernel-Space CPU Time for Pipe In Total: User * 4, Kernel * 2

13 Evaluation BitTorrent Session 700 MiB Ubuntu amd64 alternate ISO qbittorrent 2.2.5, 50 connections max (2Mbit downlink) Comparison of three measurement methods: /proc/net/dev (per-interface) ip_conntrack monitor (system-wide, filtering on qbittorrent ) In-application progress display (application-specific payload) Measurement error: only constant factor inbound: ~ 1.2 outbound: ~ 0.91 per-interface traffic compared to per-application traffic

14 Findings Monitor accur acy inbound BitTor r ent tr af fic bytes eth1 in monitor in torrent in time (minutes)

15 Findings Monitor accur acy outbound BitTor r ent tr af fic bytes eth1 out monitor out torrent out time (minutes)

16 Findings Monitor accur acy total BitTor r ent tr af fic bytes eth1 total monitor total time (minutes)

17 Findings Monitor accur acy total BitTor r ent bandwidth bandwidth (bytes/second) eth1 total bw monitor total bw time (minutes)

18 Findings Best Solution depends on circumstances ip_conntrack still has some rough edges Costs of measuring can be high (granularity vs. performance trade-off)

19 Fur ther work Controlled lab setup of BitTorrent swarm Pipe-based method as middleware in existing enterprise servers (Servlet containers, WSGI) Scaling of per-application bandwidth to fit the per-interface traffic: t intf = t app T intf T app Re-implementation of pipe monitor in C t. single measurement T total measurement (sum)

20 Links and downloads Implemented in Python License: GPL v3 or later Project Website: Git-Repository:

Packet Capture in 10-Gigabit Ethernet Environments Using Contemporary Commodity Hardware

Packet Capture in 10-Gigabit Ethernet Environments Using Contemporary Commodity Hardware Packet Capture in 1-Gigabit Ethernet Environments Using Contemporary Commodity Hardware Fabian Schneider Jörg Wallerich Anja Feldmann {fabian,joerg,anja}@net.t-labs.tu-berlin.de Technische Universtität