Managing Incompleteness, Complexity and Scale in Big Data

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Managing Incompleteness, Complexity and Scale in Big Data"

Transcription

1 Managing Incompleteness, Complexity and Scale in Big Data Nick Duffield Electrical and Computer Engineering Texas A&M University

2 Three Challenges for Big Data Complexity Problem: high-dimensional data with complex dependence between variables, difficult to model Solution: machine learning dominant relationships Incompleteness Problem: not all quantities can be directly measured Solution: statistically infer what we want from what we have Scale Problem: huge datasets: costly to store, slow to compute Solution: smart data reduction retains ability to answer most important queries

3 Big Data Complexity: Customer Experience Packet loss and delay; line quality; service parameters Objective Metrics of Network Performance? Noisy measures of customer experience Customer care calls; social media; keyword analysis Which objective metrics closely associated with customer dissatisfaction? If known, remediate and prevent future troubles Solution: (machine) learn metrics (and values), service settings, most associated with occurrence of customer calls. Set action thresholds. Monitor metrics, take action when thresholds exceed Operational savings Reduce call volume to customer care center, reduce churn Reverse problem Learn calling patterns and keywords most predictive of network problem

4 Incompleteness: Internet tomography What ISPs want Origin-Destination (OD) traffic rates between any two routers What ISPs have Measured traffic rates on each link Linear relation Link_Rates = A. OD_Rates A = routing matrix encodes which links that OD traffic traverses Solve? Under-constrained problem Different possible sets of OD_Rates yield the same set of measured Link_rates

5 Internet Tomography Gravity Model? OD_Rate(A à B) =const. Rate(Aà ALL) Rate(ALLà B) Can measure Rate(Aà ALL) at links emanating from A Problem with gravity! Gravity model is not a solution of Link_Rates = A. OD_Rates Solution: Tomogravity Use solution closest to gravity model! Penalized likelihood solution Quick to compute, good accuracy In daily use in ISPs, Routers M 2 gravity model solution Tomogravity = least square solution constraint subspace L = A.M M 1

6 Big Data Scale ISP operations generate 100s of Terabytes of usage measurement data daily Passive traffic measurements by (core) routers Session-level traffic summaries (flow records) Each flow record reports IP source and destination, #packets, bytes, timing,.. Core routers stream flow records to collectors for analysis Used widely in network management timescale from months (planning) to seconds (security) Still need tomo-gravity outside core!

7 Managing Data Scale through Sampling Turn Big Data into Smaller Data Savings in storage, bandwidth; speed up queries Reference sampling Reuse samples over multiple retrospective queries Know query class in advance, but not specific query Smart sampling matches data characteristics to analysis requirements E.g. uniform sampling is useless on heavy tails Streaming constraints Sample to be computable in small time per item Big data constraint often not met in classical methods

8 Statistically Optimal Stream Sampling Aim: Sample fraction of flow records Use to answer queries approximately Problem: heavy tails 10% of the flow records report 90% of bytes Uniform sampling misses most of the 10% Big hit on accuracy Solution: Statistically optimal non-uniform sampling algorithms (minimal estimation variance) Computationally feasible for stream sampling In use in ISPs

9 Taming the Heavy Tail Distribution of traffic estimates Uniform sampling Smart sampling

10 Next: Streaming ISP Graph Data ISP Communications Graph from Flow Records node = IP address; edge = flow from source to destination compromise control flooding Hard to detect against background Known attacks: Signature matching based on subgraphs, flow features, timing Unknown attacks: exploratory & retrospective analysis Smart sampling of subgraphs

11 Sampling + Knowledge Discovery Interplay between sampling and data mining is not well understood Need to understand how ML/DM algorithms are affected by sampling E.g. how big a sample is needed to build an accurate classifier? E.g. what sampling strategy optimizes cluster quality Expect results to be method specific I.e. smart samping + k-means

12 Sampling and Privacy Current focus on privacy-preserving data mining Opportunity for sampling to be part of the solution Naïve sampling provides privacy in expectation Your data remains private if you aren t included in the sample Intuition: uncertainty from sampling contributes to privacy This intuition can be formalized with different privacy models Sampling can be analyzed in the context of differential privacy Sampling alone does not provide differential privacy But applying a DP method to sampled data does guarantee privacy A tradeoff between sampling rate and privacy parameters Understand benefits as well as risks of information flows Network calculus of risk/reward trade-off from information sharing, joining

13 Outlook Big data challenges Incompleteness, complexity, scale Generic problems; transferable solutions Find causal relations in high dimensional data Use machine learning for discovery & prediction Big Data Tomography Solve ill-posed inverse problems with constraints from models and side data Smart Sampling Speed up computations and save on resources Tune sampling to mediate between data and queries Role of sampling in ML/DM, privacy,

Network Tomography and Internet Traffic Matrices

Network Tomography and Internet Traffic Matrices Network Tomography and Internet Traffic Matrices Matthew Roughan School of Mathematical Sciences 1 Credits David Donoho Stanford Nick Duffield AT&T Labs-Research Albert

More information

Detecting Network Anomalies. Anant Shah

Detecting Network Anomalies. Anant Shah Detecting Network Anomalies using Traffic Modeling Anant Shah Anomaly Detection Anomalies are deviations from established behavior In most cases anomalies are indications of problems The science of extracting

More information

Structural Analysis of Network Traffic Flows Eric Kolaczyk

Structural Analysis of Network Traffic Flows Eric Kolaczyk Structural Analysis of Network Traffic Flows Eric Kolaczyk Anukool Lakhina, Dina Papagiannaki, Mark Crovella, Christophe Diot, and Nina Taft Traditional Network Traffic Analysis Focus on Short stationary

More information

Reformulating the monitor placement problem: Optimal Network-wide wide Sampling

Reformulating the monitor placement problem: Optimal Network-wide wide Sampling Reformulating the monitor placement problem: Optimal Network-wide wide Sampling Gianluca Iannaccone Intel Research @ Cambridge Joint work with: G. Cantieni,, P. Thiran (EPFL) C. Barakat (INRIA), C. Diot

More information

anon.next: A Framework for Privacy in the Next Generation Internet

anon.next: A Framework for Privacy in the Next Generation Internet anon.next: A Framework for Privacy in the Next Generation Internet Matthew Wright Department of Computer Science and Engineering, The University of Texas at Arlington, Arlington, TX, USA, mwright@uta.edu,

More information

Dynamics of Prefix Usage at an Edge Router

Dynamics of Prefix Usage at an Edge Router Dynamics of Prefix Usage at an Edge Router Kaustubh Gadkari, Daniel Massey, and Christos Papadopoulos Computer Science Department, Colorado State University, USA {kaustubh, massey, christos@cs.colostate.edu}

More information

Final Project Report

Final Project Report CPSC545 by Introduction to Data Mining Prof. Martin Schultz & Prof. Mark Gerstein Student Name: Yu Kor Hugo Lam Student ID : 904907866 Due Date : May 7, 2007 Introduction Final Project Report Pseudogenes

More information

Network (Tree) Topology Inference Based on Prüfer Sequence

Network (Tree) Topology Inference Based on Prüfer Sequence Network (Tree) Topology Inference Based on Prüfer Sequence C. Vanniarajan and Kamala Krithivasan Department of Computer Science and Engineering Indian Institute of Technology Madras Chennai 600036 vanniarajanc@hcl.in,

More information

Planning Networks for VOIP. An Introduction

Planning Networks for VOIP. An Introduction Planning Networks for VOIP An Introduction Planning Networks for VOIP Page 2/10 Contents 1 Introduction...3 2 Voice Quality Requirements...3 3 Codecs...4 4 Network Layout...5 5 Planning Capacity...6 5.1

More information

An iterative tomogravity algorithm for the estimation of network traffic

An iterative tomogravity algorithm for the estimation of network traffic IMS Lecture Notes Monograph Series Complex Datasets and Inverse Problems: Tomography, Networks and Beyond Vol. 54 (2007) 12 23 c Institute of Mathematical Statistics, 2007 DOI: 10.1214/074921707000000030

More information

Internet Infrastructure Measurement: Challenges and Tools

Internet Infrastructure Measurement: Challenges and Tools Internet Infrastructure Measurement: Challenges and Tools Internet Infrastructure Measurement: Challenges and Tools Outline Motivation Challenges Tools Conclusion Why Measure? Why Measure? Internet, with

More information

MODULE 15 Clustering Large Datasets LESSON 34

MODULE 15 Clustering Large Datasets LESSON 34 MODULE 15 Clustering Large Datasets LESSON 34 Incremental Clustering Keywords: Single Database Scan, Leader, BIRCH, Tree 1 Clustering Large Datasets Pattern matrix It is convenient to view the input data

More information

Introduction to Data Mining

Introduction to Data Mining Introduction to Data Mining 1 Why Data Mining? Explosive Growth of Data Data collection and data availability Automated data collection tools, Internet, smartphones, Major sources of abundant data Business:

More information

Bisecting K-Means for Clustering Web Log data

Bisecting K-Means for Clustering Web Log data Bisecting K-Means for Clustering Web Log data Ruchika R. Patil Department of Computer Technology YCCE Nagpur, India Amreen Khan Department of Computer Technology YCCE Nagpur, India ABSTRACT Web usage mining

More information

KEITH LEHNERT AND ERIC FRIEDRICH

KEITH LEHNERT AND ERIC FRIEDRICH MACHINE LEARNING CLASSIFICATION OF MALICIOUS NETWORK TRAFFIC KEITH LEHNERT AND ERIC FRIEDRICH 1. Introduction 1.1. Intrusion Detection Systems. In our society, information systems are everywhere. They

More information

Network Tomography. Christos Gkantsidis

Network Tomography. Christos Gkantsidis Network Tomography Christos Gkantsidis Introduction Internet AS AS 2 What is the: Bandwidth Loss Rate Connectivity AS 3 of the links of the network? Using only end-to-end measurements. Network Tomography

More information

New York University Computer Science Department Courant Institute of Mathematical Sciences

New York University Computer Science Department Courant Institute of Mathematical Sciences ew York University Computer Science Department Courant Institute of Mathematical Sciences Course Title: Data Communication & etworks Course umber: g22.2662-00 Instructor: Jean-Claude Franchitti Session:

More information

Massive Cloud Auditing using Data Mining on Hadoop

Massive Cloud Auditing using Data Mining on Hadoop Massive Cloud Auditing using Data Mining on Hadoop Prof. Sachin Shetty CyberBAT Team, AFRL/RIGD AFRL VFRP Tennessee State University Outline Massive Cloud Auditing Traffic Characterization Distributed

More information

Information Management course

Information Management course Università degli Studi di Milano Master Degree in Computer Science Information Management course Teacher: Alberto Ceselli Lecture 01 : 06/10/2015 Practical informations: Teacher: Alberto Ceselli (alberto.ceselli@unimi.it)

More information

Use of Data Mining Techniques to Improve the Effectiveness of Sales and Marketing

Use of Data Mining Techniques to Improve the Effectiveness of Sales and Marketing Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology IJCSMC, Vol. 4, Issue. 4, April 2015,

More information

AN OVERVIEW OF QUALITY OF SERVICE COMPUTER NETWORK

AN OVERVIEW OF QUALITY OF SERVICE COMPUTER NETWORK Abstract AN OVERVIEW OF QUALITY OF SERVICE COMPUTER NETWORK Mrs. Amandeep Kaur, Assistant Professor, Department of Computer Application, Apeejay Institute of Management, Ramamandi, Jalandhar-144001, Punjab,

More information

Chapter 4. VoIP Metric based Traffic Engineering to Support the Service Quality over the Internet (Inter-domain IP network)

Chapter 4. VoIP Metric based Traffic Engineering to Support the Service Quality over the Internet (Inter-domain IP network) Chapter 4 VoIP Metric based Traffic Engineering to Support the Service Quality over the Internet (Inter-domain IP network) 4.1 Introduction Traffic Engineering can be defined as a task of mapping traffic

More information

Finding Fault Location: Combining network topology and end-to-end measurments to locate network problems?

Finding Fault Location: Combining network topology and end-to-end measurments to locate network problems? Finding Fault Location: Combining network topology and end-to-end measurments to locate network problems? Chris Kelly - chris.kelly@oit.gatech.edu Research Network Operations Center Georgia Tech Office

More information

Best Practices in Core Network Capacity Planning

Best Practices in Core Network Capacity Planning White Paper Best Practices in Core Network Capacity Planning Architectural Principles of the MATE Portfolio of Products What You Will Learn Core network capacity planning is the process of ensuring that

More information

D A T A M I N I N G C L A S S I F I C A T I O N

D A T A M I N I N G C L A S S I F I C A T I O N D A T A M I N I N G C L A S S I F I C A T I O N FABRICIO VOZNIKA LEO NARDO VIA NA INTRODUCTION Nowadays there is huge amount of data being collected and stored in databases everywhere across the globe.

More information

Intrusion Detection: Game Theory, Stochastic Processes and Data Mining

Intrusion Detection: Game Theory, Stochastic Processes and Data Mining Intrusion Detection: Game Theory, Stochastic Processes and Data Mining Joseph Spring 7COM1028 Secure Systems Programming 1 Discussion Points Introduction Firewalls Intrusion Detection Schemes Models Stochastic

More information

Web Usage Mining: Identification of Trends Followed by the user through Neural Network

Web Usage Mining: Identification of Trends Followed by the user through Neural Network International Journal of Information and Computation Technology. ISSN 0974-2239 Volume 3, Number 7 (2013), pp. 617-624 International Research Publications House http://www. irphouse.com /ijict.htm Web

More information

Planning an IP Network for Voice and Data

Planning an IP Network for Voice and Data A by NEC Unified Solutions, Inc. Planning an IP Network for Voice and Data Contents 1 Introduction...3 2 Voice Quality Requirements...3 3 Codecs...3 4 Network Layout...5 5 Planning Capacity...6 5.1 Step

More information

Conclusions and Future Directions

Conclusions and Future Directions Chapter 9 This chapter summarizes the thesis with discussion of (a) the findings and the contributions to the state-of-the-art in the disciplines covered by this work, and (b) future work, those directions

More information

On the Feasibility of Prefetching and Caching for Online TV Services: A Measurement Study on Hulu

On the Feasibility of Prefetching and Caching for Online TV Services: A Measurement Study on Hulu On the Feasibility of Prefetching and Caching for Online TV Services: A Measurement Study on Hulu Dilip Kumar Krishnappa, Samamon Khemmarat, Lixin Gao, Michael Zink University of Massachusetts Amherst,

More information

Comparison of Non-linear Dimensionality Reduction Techniques for Classification with Gene Expression Microarray Data

Comparison of Non-linear Dimensionality Reduction Techniques for Classification with Gene Expression Microarray Data CMPE 59H Comparison of Non-linear Dimensionality Reduction Techniques for Classification with Gene Expression Microarray Data Term Project Report Fatma Güney, Kübra Kalkan 1/15/2013 Keywords: Non-linear

More information

Information Security in Big Data: Privacy and Data Mining (IEEE, 2014) Dilara USTAÖMER 2065787

Information Security in Big Data: Privacy and Data Mining (IEEE, 2014) Dilara USTAÖMER 2065787 Information Security in Big Data: Privacy and Data Mining (IEEE, 2014) Dilara USTAÖMER 2065787 2015/5/13 OUTLINE Introduction User Role Based Methodology Data Provider Data Collector Data Miner Decision

More information

Experimentation driven traffic monitoring and engineering research

Experimentation driven traffic monitoring and engineering research Experimentation driven traffic monitoring and engineering research Amir KRIFA (Amir.Krifa@sophia.inria.fr) 11/20/09 ECODE FP7 Project 1 Outline i. Future directions of Internet traffic monitoring and engineering

More information

TE in action. Some problems that TE tries to solve. Concept of Traffic Engineering (TE)

TE in action. Some problems that TE tries to solve. Concept of Traffic Engineering (TE) 1/28 2/28 TE in action S-38.3192 Verkkopalvelujen tuotanto S-38.3192 Network Service Provisioning Networking laboratory 3/28 4/28 Concept of Traffic Engineering (TE) Traffic Engineering (TE) (Traffic Management)

More information

Balasubramanian Ramachandran

Balasubramanian Ramachandran Implementation and evaluation of OSPF Optimized Multipath Routing Balasubramanian Ramachandran M.S. Thesis defense Committee: Dr. Joseph B. Evans (chair) Dr. David W. Petr Dr. Susan Gauch Organization

More information

Restorable Logical Topology using Cross-Layer Optimization

Restorable Logical Topology using Cross-Layer Optimization פרויקטים בתקשורת מחשבים - 236340 - סמסטר אביב 2016 Restorable Logical Topology using Cross-Layer Optimization Abstract: Today s communication networks consist of routers and optical switches in a logical

More information

IPv6 Security : ICMPv6 Vulnerabilities. Navaneethan C. Arjuman National Advanced IPv6 Centre December 2012

IPv6 Security : ICMPv6 Vulnerabilities. Navaneethan C. Arjuman National Advanced IPv6 Centre December 2012 IPv6 Security : ICMPv6 Vulnerabilities Navaneethan C. Arjuman nava@nav6.usm.my National Advanced IPv6 Centre December 2012 1 Understanding ICMP What is ICMP? Internet Control Message Protocol (ICMP) ICMP

More information

An Anomaly-Based Method for DDoS Attacks Detection using RBF Neural Networks

An Anomaly-Based Method for DDoS Attacks Detection using RBF Neural Networks 2011 International Conference on Network and Electronics Engineering IPCSIT vol.11 (2011) (2011) IACSIT Press, Singapore An Anomaly-Based Method for DDoS Attacks Detection using RBF Neural Networks Reyhaneh

More information

Clustering Big Data. Anil K. Jain. (with Radha Chitta and Rong Jin) Department of Computer Science Michigan State University November 29, 2012

Clustering Big Data. Anil K. Jain. (with Radha Chitta and Rong Jin) Department of Computer Science Michigan State University November 29, 2012 Clustering Big Data Anil K. Jain (with Radha Chitta and Rong Jin) Department of Computer Science Michigan State University November 29, 2012 Outline Big Data How to extract information? Data clustering

More information

Today s Class. Kinds of Networks. What s a Network? Normal Distribution. Properties of Networks. What are networks Properties of networks

Today s Class. Kinds of Networks. What s a Network? Normal Distribution. Properties of Networks. What are networks Properties of networks Today s Class NBA 600 Networks and the Internet Class 3, Mon 10/22 Prof. Dan Huttenlocher What are networks Properties of networks Attributes of individuals in a network Very different from those of individuals

More information

3. Dataset size reduction. 4. BGP-4 patterns. Detection of inter-domain routing problems using BGP-4 protocol patterns P.A.

3. Dataset size reduction. 4. BGP-4 patterns. Detection of inter-domain routing problems using BGP-4 protocol patterns P.A. Newsletter Inter-domain QoS, Issue 8, March 2004 Online monthly journal of INTERMON consortia Dynamic information concerning research, standardisation and practical issues of inter-domain QoS --------------------------------------------------------------------

More information

Disjoint Path Algorithm for Load Balancing in MPLS network

Disjoint Path Algorithm for Load Balancing in MPLS network International Journal of Innovation and Scientific Research ISSN 2351-8014 Vol. 13 No. 1 Jan. 2015, pp. 193-199 2015 Innovative Space of Scientific Research Journals http://www.ijisr.issr-journals.org/

More information

Fast Prototyping Network Data Mining Applications. Gianluca Iannaccone Intel Research Berkeley

Fast Prototyping Network Data Mining Applications. Gianluca Iannaccone Intel Research Berkeley Fast Prototyping Network Data Mining Applications Gianluca Iannaccone Intel Research Berkeley Motivation Developing new network monitoring apps is unnecessarily time-consuming Familiar development steps

More information

An Efficient Way of Denial of Service Attack Detection Based on Triangle Map Generation

An Efficient Way of Denial of Service Attack Detection Based on Triangle Map Generation An Efficient Way of Denial of Service Attack Detection Based on Triangle Map Generation Shanofer. S Master of Engineering, Department of Computer Science and Engineering, Veerammal Engineering College,

More information

Statistical Methods for Network and Computer Security p.1/43

Statistical Methods for Network and Computer Security p.1/43 Statistical Methods for Network and Computer Security David J. Marchette marchettedj@nswc.navy.mil Naval Surface Warfare Center Code B10 Statistical Methods for Network and Computer Security p.1/43 A Few

More information

STANDPOINT FOR QUALITY-OF-SERVICE MEASUREMENT

STANDPOINT FOR QUALITY-OF-SERVICE MEASUREMENT STANDPOINT FOR QUALITY-OF-SERVICE MEASUREMENT 1. TIMING ACCURACY The accurate multi-point measurements require accurate synchronization of clocks of the measurement devices. If for example time stamps

More information

5 Performance Management for Web Services. Rolf Stadler School of Electrical Engineering KTH Royal Institute of Technology. stadler@ee.kth.

5 Performance Management for Web Services. Rolf Stadler School of Electrical Engineering KTH Royal Institute of Technology. stadler@ee.kth. 5 Performance Management for Web Services Rolf Stadler School of Electrical Engineering KTH Royal Institute of Technology stadler@ee.kth.se April 2008 Overview Service Management Performance Mgt QoS Mgt

More information

The Coremelt Attack. Ahren Studer and Adrian Perrig. We ve Come to Rely on the Internet

The Coremelt Attack. Ahren Studer and Adrian Perrig. We ve Come to Rely on the Internet The Coremelt Attack Ahren Studer and Adrian Perrig 1 We ve Come to Rely on the Internet Critical for businesses Up to date market information for trading Access to online stores One minute down time =

More information

technical brief Optimizing Performance in HP Web Jetadmin Web Jetadmin Overview Performance HP Web Jetadmin CPU Utilization utilization.

technical brief Optimizing Performance in HP Web Jetadmin Web Jetadmin Overview Performance HP Web Jetadmin CPU Utilization utilization. technical brief in HP Overview HP is a Web-based software application designed to install, configure, manage and troubleshoot network-connected devices. It includes a Web service, which allows multiple

More information

DDoS Protection Technology White Paper

DDoS Protection Technology White Paper DDoS Protection Technology White Paper Keywords: DDoS attack, DDoS protection, traffic learning, threshold adjustment, detection and protection Abstract: This white paper describes the classification of

More information

Limitations of Packet Measurement

Limitations of Packet Measurement Limitations of Packet Measurement Collect and process less information: Only collect packet headers, not payload Ignore single packets (aggregate) Ignore some packets (sampling) Make collection and processing

More information

Passive Queue Management

Passive Queue Management , 2013 Performance Evaluation of Computer Networks Objectives Explain the role of active queue management in performance optimization of TCP/IP networks Learn a range of active queue management algorithms

More information

Adaptive Tolerance Algorithm for Distributed Top-K Monitoring with Bandwidth Constraints

Adaptive Tolerance Algorithm for Distributed Top-K Monitoring with Bandwidth Constraints Adaptive Tolerance Algorithm for Distributed Top-K Monitoring with Bandwidth Constraints Michael Bauer, Srinivasan Ravichandran University of Wisconsin-Madison Department of Computer Sciences {bauer, srini}@cs.wisc.edu

More information

Network Discovery from Passive Measurements

Network Discovery from Passive Measurements Network Discovery from Passive Measurements Brian Eriksson UW-Madison bceriksson@wisc.edu Paul Barford UW-Madison pb@cs.wisc.edu Robert Nowak UW-Madison nowak@ece.wisc.edu ABSTRACT Understanding the Internet

More information

Efficient Doubletree: An Algorithm for Large-Scale Topology Discovery

Efficient Doubletree: An Algorithm for Large-Scale Topology Discovery Middle-East Journal of Scientific Research 15 (9): 1264-1271, 2013 ISSN 1990-9233 IDOSI Publications, 2013 DOI: 10.5829/idosi.mejsr.2013.15.9.11480 Efficient Doubletree: An Algorithm for Large-Scale Topology

More information

Enhanced Content Delivery Network to Improve the QoE

Enhanced Content Delivery Network to Improve the QoE Enhanced Content Delivery Network to Improve the QoE 1 Sachendra Singh Solanky, 2 Sandra Brigit Johnson, 3 Vakkalagadda Eswar Praphul 1 M.Tech Student SCSE, VIT University Chennai-600048, 2 M.Tech Student

More information

A Passive Method for Estimating End-to-End TCP Packet Loss

A Passive Method for Estimating End-to-End TCP Packet Loss A Passive Method for Estimating End-to-End TCP Packet Loss Peter Benko and Andras Veres Traffic Analysis and Network Performance Laboratory, Ericsson Research, Budapest, Hungary {Peter.Benko, Andras.Veres}@eth.ericsson.se

More information

Data and Privacy Policy Implications and Privacy Principles:

Data and Privacy Policy Implications and Privacy Principles: Document name: Data and Privacy Policy Implications and Privacy Principles Author(s): James Williams and Dale Finkleson Contributor(s): GNA Technical Group Date: 26 October 2015 Version: 0.9P Data and

More information

Probability-Model based Network Traffic Matrix Estimation

Probability-Model based Network Traffic Matrix Estimation Computer Science and Information Systems ():309 30 DOI: 0.98/CSIS3000T Probability-Model based Network Traffic Matrix Estimation Hui Tian, Yingpeng Sang, Hong Shen 3,4, and Chunyue Zhou School of Electronics

More information

Statistical Models in Data Mining

Statistical Models in Data Mining Statistical Models in Data Mining Sargur N. Srihari University at Buffalo The State University of New York Department of Computer Science and Engineering Department of Biostatistics 1 Srihari Flood of

More information

Distance Vector Routing Protocols. Routing Protocols and Concepts Ola Lundh

Distance Vector Routing Protocols. Routing Protocols and Concepts Ola Lundh Distance Vector Routing Protocols Routing Protocols and Concepts Ola Lundh Objectives The characteristics of distance vector routing protocols. The network discovery process of distance vector routing

More information

Active Graph Reachability Reduction for Network Security and Software Engineering

Active Graph Reachability Reduction for Network Security and Software Engineering Active Graph Reachability Reduction for Network Security and Software Engineering Alice X. Zheng Microsoft Research Redmond, WA alicez@microsoft.com John Dunagan Microsoft Redmond, WA jdunagan@microsoft.com

More information

Software Defined Networks

Software Defined Networks Software Defined Networks Damiano Carra Università degli Studi di Verona Dipartimento di Informatica Acknowledgements! Credits Part of the course material is based on slides provided by the following authors

More information

Extracting Performance Metrics from NetFlow in Enterprise Networks

Extracting Performance Metrics from NetFlow in Enterprise Networks Extracting Performance Metrics from NetFlow in Enterprise Networks 2nd EMANICS Workshop on NetFlow/IPFIX Usage Jochen Kögel jochen.koegel@ikr.uni-stuttgart.de 8. October 2009 Universität Stuttgart Institute

More information

Hybrid Intrusion Detection System Using K-Means Algorithm

Hybrid Intrusion Detection System Using K-Means Algorithm International Journal of Computer Sciences and Engineering Open Access Review Paper Volume-4, Issue-3 E-ISSN: 2347-2693 Hybrid Intrusion Detection System Using K-Means Algorithm Darshan K. Dagly 1*, Rohan

More information

A Review of Anomaly Detection Techniques in Network Intrusion Detection System

A Review of Anomaly Detection Techniques in Network Intrusion Detection System A Review of Anomaly Detection Techniques in Network Intrusion Detection System Dr.D.V.S.S.Subrahmanyam Professor, Dept. of CSE, Sreyas Institute of Engineering & Technology, Hyderabad, India ABSTRACT:In

More information

Sampling Biases in IP Topology Measurements

Sampling Biases in IP Topology Measurements Sampling Biases in IP Topology Measurements Anukool Lakhina with John Byers, Mark Crovella and Peng Xie Department of Boston University Discovering the Internet topology Goal: Discover the Internet Router

More information

The Feasibility of Supporting Large-Scale Live Streaming Applications with Dynamic Application End-Points

The Feasibility of Supporting Large-Scale Live Streaming Applications with Dynamic Application End-Points The Feasibility of Supporting Large-Scale Live Streaming Applications with Dynamic Application End-Points Kay Sripanidkulchai, Aditya Ganjam, Bruce Maggs, and Hui Zhang Instructor: Fabian Bustamante Presented

More information

Xiaoqiao Meng, Vasileios Pappas, Li Zhang IBM T.J. Watson Research Center Presented by: Payman Khani

Xiaoqiao Meng, Vasileios Pappas, Li Zhang IBM T.J. Watson Research Center Presented by: Payman Khani Improving the Scalability of Data Center Networks with Traffic-aware Virtual Machine Placement Xiaoqiao Meng, Vasileios Pappas, Li Zhang IBM T.J. Watson Research Center Presented by: Payman Khani Overview:

More information

Detecting Anomalies Using End-to-End Path Measurements

Detecting Anomalies Using End-to-End Path Measurements Detecting Anomalies Using End-to-End Path Measurements K. V. M. Naidu Debmalya Panigrahi Rajeev Rastogi Bell Labs Research India, Bangalore MIT Bell Labs Research India, Bangalore Abstract In this paper,

More information

Viete, čo robia Vaši užívatelia na sieti? Roman Tuchyňa, CSA

Viete, čo robia Vaši užívatelia na sieti? Roman Tuchyňa, CSA Viete, čo robia Vaši užívatelia na sieti? Roman Tuchyňa, CSA What is ReporterAnalyzer? ReporterAnalyzer gives network professionals insight into how application traffic is impacting network performance.

More information

Service Description DDoS Mitigation Service

Service Description DDoS Mitigation Service Service Description DDoS Mitigation Service Interoute, Walbrook Building, 195 Marsh Wall, London, E14 9SG, UK Tel: +800 4683 7681 Email: info@interoute.com Contents Contents 1 Introduction...3 2 An Overview...3

More information

Building MPLS VPNs with QoS Routing Capability i

Building MPLS VPNs with QoS Routing Capability i Building MPLS VPNs with QoS Routing Capability i Peng Zhang, Raimo Kantola Laboratory of Telecommunication Technology, Helsinki University of Technology Otakaari 5A, Espoo, FIN-02015, Finland Tel: +358

More information

Data Mining Analytics for Business Intelligence and Decision Support

Data Mining Analytics for Business Intelligence and Decision Support Data Mining Analytics for Business Intelligence and Decision Support Chid Apte, T.J. Watson Research Center, IBM Research Division Knowledge Discovery and Data Mining (KDD) techniques are used for analyzing

More information

Link-State Routing Can Achieve Optimal Traffic Engineering: From Entropy To IP

Link-State Routing Can Achieve Optimal Traffic Engineering: From Entropy To IP Link-State Routing Can Achieve Optimal Traffic Engineering: From Entropy To IP Dahai Xu, Ph.D. Florham Park AT&T Labs - Research Joint work with Mung Chiang and Jennifer Rexford (Princeton University)

More information

The Quality of Internet Service: AT&T s Global IP Network Performance Measurements

The Quality of Internet Service: AT&T s Global IP Network Performance Measurements The Quality of Internet Service: AT&T s Global IP Network Performance Measurements In today's economy, corporations need to make the most of opportunities made possible by the Internet, while managing

More information

Active Internet Traffic Filtering to Denial of Service Attacks from Flash Crowds

Active Internet Traffic Filtering to Denial of Service Attacks from Flash Crowds Active Internet Traffic Filtering to Denial of Service Attacks from Flash Crowds S.Saranya Devi 1, K.Kanimozhi 2 1 Assistant professor, Department of Computer Science and Engineering, Vivekanandha Institute

More information

CHAPTER 2. QoS ROUTING AND ITS ROLE IN QOS PARADIGM

CHAPTER 2. QoS ROUTING AND ITS ROLE IN QOS PARADIGM CHAPTER 2 QoS ROUTING AND ITS ROLE IN QOS PARADIGM 22 QoS ROUTING AND ITS ROLE IN QOS PARADIGM 2.1 INTRODUCTION As the main emphasis of the present research work is on achieving QoS in routing, hence this

More information

International Journal of Computer Science Trends and Technology (IJCST) Volume 3 Issue 3, May-June 2015

International Journal of Computer Science Trends and Technology (IJCST) Volume 3 Issue 3, May-June 2015 RESEARCH ARTICLE OPEN ACCESS Data Mining Technology for Efficient Network Security Management Ankit Naik [1], S.W. Ahmad [2] Student [1], Assistant Professor [2] Department of Computer Science and Engineering

More information

Mining Anomalies in Network-Wide Flow Data. Anukool Lakhina, Ph.D. with Mark Crovella and Christophe Diot

Mining Anomalies in Network-Wide Flow Data. Anukool Lakhina, Ph.D. with Mark Crovella and Christophe Diot Mining Anomalies in Network-Wide Flow Data Anukool Lakhina, Ph.D. with Mark Crovella and Christophe Diot SANOG-7, Mumbai, January, 00 Network Anomaly Diagnosis Am I being attacked? Is someone scanning

More information

Smart WWW Traffic Balancing

Smart WWW Traffic Balancing Smart WWW Traffic Balancing Erol Gelenbe Ricardo Lent Juan Arturo Nunez School of Electrical Engineering & Computer Science University of Central Florida Introduction The Internet is one of the biggest

More information

Employer Health Insurance Premium Prediction Elliott Lui

Employer Health Insurance Premium Prediction Elliott Lui Employer Health Insurance Premium Prediction Elliott Lui 1 Introduction The US spends 15.2% of its GDP on health care, more than any other country, and the cost of health insurance is rising faster than

More information

Detection of local affinity patterns in big data

Detection of local affinity patterns in big data Detection of local affinity patterns in big data Andrea Marinoni, Paolo Gamba Department of Electronics, University of Pavia, Italy Abstract Mining information in Big Data requires to design a new class

More information

Game-based Analysis of Denial-of- Service Prevention Protocols. Ajay Mahimkar Class Project: CS 395T

Game-based Analysis of Denial-of- Service Prevention Protocols. Ajay Mahimkar Class Project: CS 395T Game-based Analysis of Denial-of- Service Prevention Protocols Ajay Mahimkar Class Project: CS 395T Overview Introduction to DDoS Attacks Current DDoS Defense Strategies Client Puzzle Protocols for DoS

More information

Preprocessing Web Logs for Web Intrusion Detection

Preprocessing Web Logs for Web Intrusion Detection Preprocessing Web Logs for Web Intrusion Detection Priyanka V. Patil. M.E. Scholar Department of computer Engineering R.C.Patil Institute of Technology, Shirpur, India Dharmaraj Patil. Department of Computer

More information

Network Management, Performance Characteristics, and Commercial Terms Policy. (1) mispot's Terms of Service (TOS), viewable at mispot.net.

Network Management, Performance Characteristics, and Commercial Terms Policy. (1) mispot's Terms of Service (TOS), viewable at mispot.net. Network Management, Performance Characteristics, and Commercial Terms Policy Consistent with FCC regulations 1, Agri-Valley Broadband, Inc., d/b/a mispot (mispot) provides this information about our broadband

More information

BEHAVIOR BASED CREDIT CARD FRAUD DETECTION USING SUPPORT VECTOR MACHINES

BEHAVIOR BASED CREDIT CARD FRAUD DETECTION USING SUPPORT VECTOR MACHINES BEHAVIOR BASED CREDIT CARD FRAUD DETECTION USING SUPPORT VECTOR MACHINES 123 CHAPTER 7 BEHAVIOR BASED CREDIT CARD FRAUD DETECTION USING SUPPORT VECTOR MACHINES 7.1 Introduction Even though using SVM presents

More information

CROSS LAYER BASED MULTIPATH ROUTING FOR LOAD BALANCING

CROSS LAYER BASED MULTIPATH ROUTING FOR LOAD BALANCING CHAPTER 6 CROSS LAYER BASED MULTIPATH ROUTING FOR LOAD BALANCING 6.1 INTRODUCTION The technical challenges in WMNs are load balancing, optimal routing, fairness, network auto-configuration and mobility

More information

On the Impact of Route Monitor Selection

On the Impact of Route Monitor Selection On the Impact of Route Monitor Selection Ying Zhang Zheng Zhang Z. Morley Mao Y. Charlie Hu Bruce Maggs Univ. of Michigan Purdue Univ. Univ. of Michigan Purdue Univ. CMU Paper ID: E-578473438 Number of

More information

Real-Time Traffic Engineering Management With Route Analytics

Real-Time Traffic Engineering Management With Route Analytics Real-Time Traffic Engineering Management With Route Analytics Executive Summary Increasing numbers of service providers and mobile operators are using RSVP-TE based traffic engineering to provide bandwidth

More information

AN EFFICIENT SELECTIVE DATA MINING ALGORITHM FOR BIG DATA ANALYTICS THROUGH HADOOP

AN EFFICIENT SELECTIVE DATA MINING ALGORITHM FOR BIG DATA ANALYTICS THROUGH HADOOP AN EFFICIENT SELECTIVE DATA MINING ALGORITHM FOR BIG DATA ANALYTICS THROUGH HADOOP Asst.Prof Mr. M.I Peter Shiyam,M.E * Department of Computer Science and Engineering, DMI Engineering college, Aralvaimozhi.

More information

Street Smart Traffic Discovering and Disseminating Automobile Congestion Using VANETs

Street Smart Traffic Discovering and Disseminating Automobile Congestion Using VANETs Street Smart Traffic Discovering and Disseminating Automobile Congestion Using VANETs 1 Street Smart Traffic Discovering and Disseminating Automobile Congestion Using VANETs R. Krithika Raj 1 and M. Janani

More information

Project Report on Traffic Engineering and QoS with MPLS and its applications

Project Report on Traffic Engineering and QoS with MPLS and its applications Project Report on Traffic Engineering and QoS with MPLS and its applications Brief Overview Multiprotocol Label Switching (MPLS) is an Internet based technology that uses short, fixed-length labels to

More information

An Evaluation of Machine Learning Method for Intrusion Detection System Using LOF on Jubatus

An Evaluation of Machine Learning Method for Intrusion Detection System Using LOF on Jubatus An Evaluation of Machine Learning Method for Intrusion Detection System Using LOF on Jubatus Tadashi Ogino* Okinawa National College of Technology, Okinawa, Japan. * Corresponding author. Email: ogino@okinawa-ct.ac.jp

More information

Denial of Service Attacks, What They are and How to Combat Them

Denial of Service Attacks, What They are and How to Combat Them Denial of Service Attacks, What They are and How to Combat Them John P. Pironti, CISSP Genuity, Inc. Principal Enterprise Solutions Architect Principal Security Consultant Version 1.0 November 12, 2001

More information

Decoding DNS data. Using DNS traffic analysis to identify cyber security threats, server misconfigurations and software bugs

Decoding DNS data. Using DNS traffic analysis to identify cyber security threats, server misconfigurations and software bugs Decoding DNS data Using DNS traffic analysis to identify cyber security threats, server misconfigurations and software bugs The Domain Name System (DNS) is a core component of the Internet infrastructure,

More information

WEB SITE OPTIMIZATION THROUGH MINING USER NAVIGATIONAL PATTERNS

WEB SITE OPTIMIZATION THROUGH MINING USER NAVIGATIONAL PATTERNS WEB SITE OPTIMIZATION THROUGH MINING USER NAVIGATIONAL PATTERNS Biswajit Biswal Oracle Corporation biswajit.biswal@oracle.com ABSTRACT With the World Wide Web (www) s ubiquity increase and the rapid development

More information

Analysis of Communication Patterns in Network Flows to Discover Application Intent

Analysis of Communication Patterns in Network Flows to Discover Application Intent Analysis of Communication Patterns in Network Flows to Discover Application Intent Presented by: William H. Turkett, Jr. Department of Computer Science FloCon 2013 January 9, 2013 Port- and payload signature-based

More information

CLASSIFYING NETWORK TRAFFIC IN THE BIG DATA ERA

CLASSIFYING NETWORK TRAFFIC IN THE BIG DATA ERA CLASSIFYING NETWORK TRAFFIC IN THE BIG DATA ERA Professor Yang Xiang Network Security and Computing Laboratory (NSCLab) School of Information Technology Deakin University, Melbourne, Australia http://anss.org.au/nsclab

More information

Defending against Flooding-Based Distributed Denial-of-Service Attacks: A Tutorial

Defending against Flooding-Based Distributed Denial-of-Service Attacks: A Tutorial Defending against Flooding-Based Distributed Denial-of-Service Attacks: A Tutorial Rocky K. C. Chang The Hong Kong Polytechnic University Presented by Scott McLaren 1 Overview DDoS overview Types of attacks

More information