Network Anomaly Detection A Machine Learning Perspective Dhruba Kumar Bhattacharyya Jugal Kumar KaKta»C) CRC Press J Taylor & Francis Croup Boca Raton London New York CRC Press is an imprint of the Taylor & Francis Croup, an Informs business A CHAPMAN & HALL BOOK
Contents List of Figures xv List of Tables xvii Preface xix Acknowledgments xxi Abstract xxiii Authors xxv 1 Introduction 1 11 The Internet and Modern Networks 3 12 Network Vulnerabilities 4 13 Anomalies and Anomalies in Networks 5 14 Machine Learning 7 15 Prior Work on Network Anomaly Detection 9 16 Contributions of This Book 11 17 Organization 13 2 Networks and Anomalies 15 21 Networking Basics 15 211 Typical View of a Network 16 212 Communication Media 16 2121 Guided Media 17 2122 Unguided Media 18 213 Network Software 20 2131 Layered Architecture 21 2132 Connection-Oriented and Connection less Services 21 2133 Service Primitives 21 2134 Services and Protocols 22 vii
viii Contents 21A Reference Models 22 2141 The ISO OSI Reference Model 23 2142 TCP/IP Reference Model 24 215 Protocols 26 2151 Transport Control Protocol 27 2152 User Datagram Protocol 27 2153 Internet Protocol (IP) 28 2154 SMTP 29 2155 SNMP 29 2156 ICMP 29 2157 FTP 30 2158 Telnet 30 216 Types of Networks 31 2161 Local Area Networks (LAN) 31 2162 Wide Area Networks (WAN) 31 2163 Metropolitan Area Network (MAN) 32 43 2164 Wireless Networks 32 2165 Internetworks 33 2166 The Internet 33 217 Scales of Networks 34 218 Network Topologies 35 2181 Bus 35 2182 Ring 36 2183 Tree 36 2184 Star 37 219 Hardware Components 37 2191 Network Communication Devices 37 2192 Network Interface Card (NIC) 41 2193 Transceivers 42 2194 Media Converter 43 2110 Network Performance 43 21101 Network Performance Constraints 21102 Network Performance Parameter Tun ing 44 21103 Performance Oriented System Design 44 21104 Protocols for Gigabit Networks 45 21105 Faster Processing of TPDU 45 22 Anomalies in a Network 45 221 Network Vulnerabilities 46
Contents ix 46 47 48 2211 Network Configuration Vulnerabilities 2212 Network Hardware Vulnerabilities 2213 Network Perimeter Vulnerabilities 2214 Network Monitoring and Logging Vul nerabilities 48 2215 Communication Vulnerabilities 49 2216 Wireless Connection Vulnerabilities 49 222 Security-Related Network Anomalies 49 223 Who Attacks Networks 50 224 Precursors to an Attack 51 225 Network Attacks Taxonomy 52 2251 Denial of Service (DoS) 53 2252 User to Root Attacks (U2R) 54 2253 Remote to Local (R2L) 54 2254 Probe 55 226 Discussion 55 3 An Overview of Machine Learning Methods 57 31 Introduction 57 32 Types of Machine Learning Methods 59 33 Supervised Learning: Some Popular Methods 60 331 Decision and Regression Trees 61 3311 Classification and Regression Tree 62 332 Support Vector Machines 67 34 Unsupervised Learning 69 341 Cluster Analysis 70 3411 Various Types of Data 71 3412 Proximity Measures 72 3413 Clustering Methods 73 3414 Discussion 87 342 Outlier Mining 88 343 Association Rule Learning 96 3431 Basic Concepts 97 344 Frequent Itemset Mining Algorithms 99 345 Rule Generation Algorithms 103 346 Discussion 105 35 Probabilistic Learning 106 351 Learning Bayes Nets 106 352 Simple Probabilistic Learning: Naive Bayes 107 353 Hidden Markov Models 108
X Contents 354 Expectation Maximization Algorithm 110 36 Soft Computing 112 361 Artificial Neural Networks 113 362 Rough Sets 113 363 Fuzzy Logic 114 364 Evolutionary Computation 115 365 Ant Colony Optimization 115 37 Reinforcement Learning 116 38 Hybrid Learning Methods 117 39 Discussion 118 4 Detecting Anomalies in Network Data 121 41 Detection of Network Anomalies 121 411 Host-Based IDS (HIDS) 121 412 Network-Based IDS (NIDS) 122 413 Anomaly-Based Network Intrusion Detection 123 414 Supervised Anomaly Detection Approach 124 415 Issues 129 416 Unsupervised Anomaly Detection Approach 129 417 Issues 132 418 Hybrid Detection Approach 132 419 Issues 133 42 Aspects of Network Anomaly Detection 133 421 Proximity Measure and Types of Data 134 422 Relevant Feature Identification 135 423 Anomaly Score 135 43 Datasets 140 431 Public Datasets 141 4311 KDD Cup 1999 Dataset 141 4312 NSL-KDD Dataset 143 432 Private Datasets: Collection and Preparation 144 4321 TUIDS Intrusion Dataset 144 433 Network Simulation 151 44 Discussion 151 5 Feature Selection 157 51 Feature Selection vs Feature Extraction 158 52 Feature Relevance 158 53 Advantages 160 54 Applications of Feature Selection 160
Contents xi 541 Bioinformatics 160 542 Network Security 161 543 Text Categorization 162 544 Biometrics 162 545 Content-Based Image Retrieval 162 55 Prior Surveys on Feature Selection 163 551 A Comparison with Prior Surveys 163 56 Problem Formulation 166 57 Steps in Feature Selection 167 571 Subset Generation 168 5711 Random Subset Generation 168 5712 Heuristic Subset Generation 168 5713 Complete Subset Generation 169 572 Feature Subset Evaluation 169 5721 Dependent Criteria 169 5722 Independent Criteria 169 573 Goodness Criteria 169 574 Result Validation 170 5741 External Validation 170 5742 Internal Validation 170 58 Feature Selection Methods: A Taxonomy 171 59 Existing Methods of Feature Selection 173 591 Statistical Feature Selection 174 592 Information Theoretic Feature Selection 176 593 Soft Computing Methods 178 594 Clustering and Association Mining Approach 179 595 Ensemble Approach 180 510 Subset Evaluation Measures 181 5101 Inconsistency Rate 181 5102 Relevance 182 5103 Symmetric Uncertainty 182 5104 Dependency 183 5105 Fuzzy Entropy 183 5106 Hamming Loss 184 5107 Ranking Loss 184 511 Systems and Tools for Feature Selection 184 512 Discussion 189
xii Contents 6 Approaches to Network Anomaly Detection 191 61 Network Anomaly Detection Methods 191 611 Requirements 192 62 Types of Network Anomaly Detection Methods 192 63 Anomaly Detection Using Supervised Learning 193 631 Parametric Methods 194 632 Nonparametric Methods 195 64 Anomaly Detection Using Unsupervised Learning 641 Clustering-Based Anomaly Detection Methods 642 Anomaly Detection Using the Outlier Mining 643 Anomaly Detection Using Association Mining 65 Anomaly Detection Using Probabilistic Learning 199 199 202 203 207 651 Methods Using the Hidden Markov Model 207 652 Methods Using Bayesian Networks 209 653 Naive Bayes Methods 210 654 Gaussian Mixture Model 211 655 Methods Using the EM Algorithm 214 66 Anomaly Detection Using Soft Computing 216 661 Genetic Algorithm Approaches 216 662 Artificial Neural Network Approaches 217 663 Fuzzy Set Theoretic Approach 218 664 Rough Set Approaches 218 665 Ant Colony and AIS Approaches 219 67 Knowledge in Anomaly Detection 222 671 Expert System and Rule-Based Approaches 223 226 672 Ontology- and Logic-Based Approaches 225 68 Anomaly Detection Using Combination Learners 681 Ensemble Methods 226 682 Fusion Methods 227 683 Hybrid Methods 228 69 Discussion 229 7 Evaluation Methods 235 71 Accuracy 235 711 Sensitivity and Specificity 236 712 Misclassification Rate 237 713 Confusion Matrix 237 714 Precision, Recall and F-measure 238 715 Receiver Operating Characteristics Curves 240 72 Performance 241
Contents xiii 73 Completeness 242 74 Timeliness 242 75 Stability 243 76 Interoperability 243 77 Data Quality, Validity and Reliability 243 78 Alert Information 245 79 Unknown Attacks Detection 245 710 Updating References 245 711 Discussion 246 250 250 8 Tools and Systems 247 81 Introduction 247 811 Attacker's Motivation 247 812 Steps in Attack Launching 248 813 Launching and Detecting Attacks 248 8131 Attack Launching Tools and Systems 8132 Attack Detecting Tools and Systems 82 Attack Related Tools 251 821 Taxonomy 252 822 Information Gathering Tools 252 8221 Sniffing Tools 253 8222 Network Mapping or Scanning Tools 259 823 Attack Launching Tools 261 8231 Trojans 262 8232 Denial of Service Attacks 264 8233 Packet Forging Attack Tools 267 8234 Application Layer Attack Tools 270 8235 Fingerprinting Attack Tools 271 8236 User Attack Tools 273 8237 Other Attack Tools 275 824 Network Monitoring Tools 277 8241 Visualization Tools 277 83 Attack Detection Systems 280 84 Discussion 286 9 Open Issues, Challenges and Concluding Remarks 289 91 Runtime Limitations for Anomaly Detection Systems 290 92 Reducing the False Alarm Rate 290 93 Issues in Dimensionality Reduction 290 94 Computational Needs of Network Defense Mechanisms 291
xiv Contents 95 Designing Generic Anomaly Detection Systems 291 96 Handling Sophisticated Anomalies 291 97 Adaptability to Unknown Attacks 292 98 Detecting and Handling Large-Scale Attacks 292 99 Infrastructure Attacks 292 910 High Intensity Attacks 292 911 More Inventive Attacks 293 912 Concluding Remarks 293 References 295 Index 337