ABSENCE: Usage-based Failure Detection in Mobile Networks. Binh Nguyen, Zihui Ge, Jacobus Van der Merwe, He Yan, Jennifer Yates Mobicom 2015

Transcription

1 ABSENCE: Usage-based Failure Detection in Mobile Networks Binh Nguyen, Zihui Ge, Jacobus Van der Merwe, He Yan, Jennifer Yates Mobicom 215 1

2 Silent failures EPC core core RAN 2

3 Silent failures EPC core core RAN Silent failures: service disruptions/outages that are not detected by current monitoring systems. New features rolled out, bugs on devices, or combination of both. 2

4 Silent failures EPC core core RAN Silent failures: service disruptions/outages that are not detected by current monitoring systems. New features rolled out, bugs on devices, or combination of both. 2

5 Silent failures EPC core core RAN Silent failures: service disruptions/outages that are not detected by current monitoring systems. New features rolled out, bugs on devices, or combination of both. Detecting silent failures is challenging! 2

6 3 Detecting silent failures is difficult - passive network monitoring

7 Detecting silent failures is difficult - passive network monitoring Drops in traffic/usage on network elements do not imply service disruptions: Load balancing/maintenance activities. Dynamic routing/self-organizing Network (SON). Load Load balancing event expected load Time 3 actual load

8 Detecting silent failures is difficult - passive network monitoring Drops in traffic/usage on network elements do not imply service disruptions: Load balancing/maintenance activities. Dynamic routing/self-organizing Network (SON). Key Performance metric Indicators (KPI) may not reflect service issues: E.g., accessibility KPI looks good even when only a subset of users can access the network. Load Load balancing event expected load Time 3 actual load

9 Detecting silent failures is difficult - passive network monitoring Drops in traffic/usage on network elements do not imply service disruptions: Load balancing/maintenance activities. Dynamic routing/self-organizing Network (SON). A healthy network (from a monitoring perspective) does not Key Performance metric Indicators (KPI) may not reflect service issues: guarantee service experience of users! E.g., accessibility KPI looks good even when only a subset of users can access the network. Load Load balancing event expected load Time 3 actual load

10 4 Detecting silent failures is difficult - active service monitoring EPC core RAN

11 4 Detecting silent failures is difficult - active service monitoring Sending test traffic across the network on all service paths. EPC core RAN

12 Detecting silent failures is difficult - active service monitoring Sending test traffic across the network on all service paths. Many types of customer devices, applications, huge geographic environment to probe. EPC core RAN Active monitoring does not scale! 4

13 5 Relying on customer feedback It takes time for customers to give feedback. Relying on customer feedback is too slow: hours of delay. E.g., failure happens at 16:38 UTC but manifests in customer feedback at 21: UTC, 3.5 hours of delay. # of tickets Event starting time (16:38 UTC) Detected by Customer care (21: UTC) Too slow! 3.5 hours delay Time

14 ABSENCE: usage-based failure detection 6

15 ABSENCE: usage-based failure detection ABSENCE: Passive service monitoring approach - monitor usage of users in a passive manner. 6

16 ABSENCE: usage-based failure detection ABSENCE: Passive service monitoring approach - monitor usage of users in a passive manner. Absence of customer usage is a reliable indicator of service disruptions in a mobile network. 6

17 ABSENCE s key ideas A group of users Usage Mobile Network 7

18 ABSENCE s key ideas Usage Mobile Network A group of users If failure happens, users are not able to use the network as normal. 7

19 ABSENCE s key ideas A group of users Usage Mobile Network If failure happens, users are not able to use the network as normal. Large number of users cannot use the network leads to a drop in usage. Could detect both hard failures (outages) and performance degradations. 7

20 ABSENCE overview Use anonymized and aggregated Call Detail Record (CDR) collected in real time from an U.S. operator. 8

21 ABSENCE overview Use anonymized and aggregated Call Detail Record (CDR) collected in real time from an U.S. operator. Week Time

22 ABSENCE overview Use anonymized and aggregated Call Detail Record (CDR) collected in real time from an U.S. operator. Week Time

23 ABSENCE overview Use anonymized and aggregated Call Detail Record (CDR) collected in real time from an U.S. operator. Expected usage Week Time Absence of usage 8

24 ABSENCE overview Use anonymized and aggregated Call Detail Record (CDR) collected in real time from an U.S. operator. Expected usage Week 3 An Time Absence of usage 8 om aly?

25 Outline Motivation. ABSENCE overview. Is ABSENCE feasible? ABSENCE s challenges. ABSENCE s event detection. Synthetic workload evaluation. Operational validation. 9

26 Is ABSENCE feasible? Is usage predictable enough for anomaly detection? 1

27 Is usage predictable enough? 11

28 Is usage predictable enough? While individual user usage is not predictable, usage of a large group of users is predictable. 11

29 Is usage predictable enough? While individual user usage is not predictable, usage of a large group of users is predictable. For example: 3 weeks of usage overlapped, usage of a small group is less predictable than usage of a large group. Week1 Week2 Week3 Week1 Week2 Week3 # of calls # of calls Time Time 7 users 3 users 11

30 Is usage predictable enough? While individual user usage is not predictable, usage of a large group of users is predictable. For example: 3 weeks of usage overlapped, usage of a small group is less predictable than usage of a large group. Week1 Week2 Week3 Week1 Week2 Week3 # of calls # of calls Time Time Yes, usage of a large enough group of users is predictable! 7 users 3 users 11

32 Challenges Failures happens to different scopes: geo-area, device makes/models, service types. How to deal with users mobility? How to improve predictability of aggregate usage? How to make ABSENCE scalable, given a large amount of data in the network? 13

33 Challenges Failures happens to different scopes: geo-area, device makes/models, service types. How to deal with users mobility? How to improve predictability of aggregate usage? How to make ABSENCE scalable, given a large amount of data in the network? 13

34 How to detect failures with different scopes? 14

35 How to detect failures with different scopes? Group users based on their geographical information: ZIP code area, city, state. A user could belong to multiple geographical groups in the same time. Under each geographical group: further divided to device OS, make. 14

36 How to detect failures with different scopes? Group users based on their geographical information: ZIP code area, city, state. A user could belong to multiple geographical groups in the same time. Under each geographical group: further divided to device OS, make. 14

37 How to detect failures with different scopes? Group users based on their geographical information: ZIP code area, city, state. A user could belong to multiple geographical groups in the same time. Under each geographical group: further divided to device OS, make. Salt Lake City (841) + Android ios Samsung HTC iphone Gal. S5 Gal. S4 iphone 6 iphone 5 14

40 Outline Motivation. ABSENCE overview. Is ABSENCE feasible?. ABSENCE s challenges. ABSENCE s event detection. Synthetic workload evaluation. Operational validation. 15

41 Event detection algorithm 6 Time series Detected anomalies 5 4 metric Usage s time series 16

42 Event detection algorithm 3 Trend 25 metric 2 15 Trend Time series Detected anomalies 4 Seasonal Decompose time series: trend, 5 3 seasonal, noise metric 3 metric Trend: moving average. Seasonal: average of phasing values. 2-1 Noise = Time series - Trend - Seasonal Seasonal Noise component Lower 95% CI of Noise component Upper 95% CI of Noise component Usage s time series 1 5 metric Noise

43 Event detection algorithm Upper 95th C.I. 6 Time series Detected anomalies 2 15 Noise component Lower 95% CI of Noise component Upper 95% CI of Noise component metric 3 metric Lower 95th C.I Usage s time series Noise component 17

44 Event detection algorithm If noise is out of the 95th percent Confidence Interval (CI) of noise component => anomaly. Upper 95th C.I. 6 Time series Detected anomalies 2 15 Noise component Lower 95% CI of Noise component Upper 95% CI of Noise component metric 3 metric Lower 95th C.I Usage s time series Noise component anomaly 17

46 Synthetic workload evaluation 6 months of real CDR from an U.S operator. Synthetically introduce failures: Network failures: remove usage on base stations. Device failures: remove usage on devices. 19

47 Parameters and metrics Parameters Metrics 11, failures generated. Detection rate = detected events/introduced events. 1 ZIPs, 1 cities. Loss ratio = loss until detected/normal usage. Two popular device types. LTE/Voice. Example of failure scenarios: Duration: 1,2,3,6,12 hours. All Android devices in Los Angeles fail. Quiet and busy hours. All Iphone5 devices in Downtown Los Angeles fail. Impact degree: - 55%. 2

48 5-1 Overall detection rate 1 9 All failures 8 Detection rate (percent) Failure impact (percent) With the 11, introduced failures: ABSENCE detected >96% of failures that have more than 15% of impact. ABSENCE tends to miss events that are <1% of impact. 21

51 Loss ratio of detected failures CDF Loss ratio (percent) Busy hours All detected failures: ~97% of them are detected when <1% of usage is lost (during busy hours). 22

52 Outline Motivation. ABSENCE overview. Is ABSENCE feasible? ABSENCE s challenges. ABSENCE event detection. Synthetic workload evaluation. Operational validation. 23

53 Evaluate against known silent failures from the operator 24

54 Evaluate against known silent failures from the operator 19 silent failure events: not known by the network operator when they happened. Detected19/19, 1% true positive. 24

55 Alarm rate and true positive Use the 19 known events from operator. Alarm rate (m): average number of alarms per day that an operation team needs to handle. Cut-off threshold (n): filter out events that less impactful. Increase cut-off threshold could reduce alarm rate while maintaining true positive rate of ABSENCE True positive (%) True positive Alarm rate 14 7m 12 6m 5m 1 8 4m 6 3m 4 2m m 2 Alarm rate (alarms/day) n 2n 4n 3n 6n 4n 8n 5n 1n 6n 12n 7n Cut-off threshold 25

56 Alarm rate and true positive Use the 19 known events from operator. Alarm rate (m): average number of alarms per day that an operation team needs to handle. Cut-off threshold (n): filter out events that less impactful. Increase cut-off threshold could reduce alarm rate while maintaining true positive rate of ABSENCE True positive (%) True positive Alarm rate n 2n 4n 3n 6n 4n 8n 5n 1n 6n 12n 7n m 12 6m 5m 1 8 4m 6 3m 4 2m m 2 Alarm rate (alarms/day) 1% true positive, m alarms per day m is small enough Cut-off threshold 25

57 Alarm rate and true positive Use the 19 known events from operator. Alarm rate (m): average number of alarms per day that an operation team needs to handle. Cut-off threshold (n): filter out events that less impactful. Increase cut-off threshold could reduce alarm rate while maintaining true positive rate of ABSENCE ABSENCE s alarm rate is reasonable for practical! True positive (%) True positive Alarm rate n 2n 4n 3n 6n 4n 8n 5n 1n 6n 12n 7n Cut-off threshold m 12 6m 5m 1 8 4m 6 3m 4 2m m 2 Alarm rate (alarms/day) 1% true positive, m alarms per day m is small enough

58 Conclusions Absence of customer usage is a reliable indicator of service disruptions a mobile network. Appropriate grouping users results in predictable usage and high fidelity for anomaly detection. Synthetic evaluation and operational validation. Practical in an operational environment. 26

59 Thank you! ABSENCE: Usage-based Failure Detection in Mobile Networks Binh Nguyen, Zihui Ge, Jacobus Van der Merwe, He Yan, Jennifer Yates Mobicom