Collective Mind. Early Warnings of Systematic Failures of Equipment. Big Data Analytics for Proactive Fleet Management

Transcription

1 Collective Mind Early Warnings of Systematic Failures of Equipment Big Data Analytics for Proactive Fleet Management Dr. Artur Dubrawski Dr. Norman Sondheimer Auton Lab Carnegie Mellon University University of Massachusetts Amherst Big Data for Defense and Homeland Security Symposium, Alexandria, VA, January 29, 2013 Copyright Carnegie Mellon University 2013

2 Collective Mind Unique technology that integrates Multivariate Big Data mining Predictive Analytics Interactive visualization of trends and patterns Scales to large amounts of noisy data routinely collected in Operations and Maintenance Supports Proactive Approach to Fleet Health Management 2

3 Challenges of Proactive Approach to Fleet Health Management Unexpected systematic maintenance crises are common Out of spec parts Ill conceived maintenance procedure New mission/theater Inexperienced personnel, etc. Early detection is the key to Proactive Management Currently, various performance indicators are used to reactively survey the status of fleet Physics based models are used to identify propensity of individual aircraft to known problems Ideal complementary capability: Fast and reliable detection of unexpected issues No false alarms Never missing anything important Copyright Carnegie Mellon University

4 Proving Ground: Surveillance of Public Health Genesis: Post 9/11 concern of a possible bio terror attack, 2002 Winter Olympics in Utah, preexisting detection systems subject to substantial latencies Familiar requirements: Fast detection, high sensitivity, low false alert rates Homogeneous populations of subjects Solution: Monitor changes in demand for medical services and supplies Proven utility in issuance of reliable early warnings 4

5 Data driven Surveillance of Public Health: Two Key Analytic Challenges 1. How to decide which spike in data is due to a real issue as opposite to some (A) random or (B) known effect? Answer A: Aggressively use statistical significance analysis to quantify the risk of making Type I errors (false positives) Answer B: Look for plausible natural explanation(s) for every detected anomalous pattern Example: Spatio-temporal distribution of daily counts of cases of leptospirosis in Sri-Lanka and Tamil Nadu state of India 5

6 target baseline total current reference ,550 12,397 total ,645 12,515 Bi variate Temporal Scan: One of Possible Ways to Attack that Challenge 1. Establish time window of interest and period of reference 2. Compute sums of target and baseline counts 3. Put the results in a 2 by 2 table 4. Report the p value of the χ 2 or Fisher s exact test of independence E.g.: expected count = 8.2 p value = 7.39*10 8 target counts (SELECT SPECIFIC SUBSET OF VALUES OF DESCRIPTOR VARIABLES) Jan-01 Apr-01 Jul-01 Oct-01 baseline counts (SELECT ALL) Jan-02 time window of interest Apr-02 Jul-02 Oct-02 Jan-03 Apr-03 Jul-03 Oct-03 Jan-04 Apr-04 Jul-04 Oct-04 Jan-05 Apr-05 Jul-05 Oct-05 Jan-06 Apr-06 Jul-06 Oct-06 Temporal Scan alerts of unusual changes in frequency of events of interest (e.g. elderly patients reporting recently with fever) which cannot be explained by the changes in baselines (like e.g. increase of population size) Jan-01 Apr-01 Jul-01 Oct-01 Jan-02 Apr-02 Jul-02 Oct-02 Jan-03 Apr-03 Jul-03 Oct-03 Jan-04 Apr-04 Jul-04 Oct-04 Jan-05 Apr-05 Jul-05 Oct-05 Jan-06 Apr-06 Jul-06 Oct-06 6

7 Data driven Surveillance of Public Health: Two Key Analytic Challenges 1. How to decide which spike in data is due to a real issue as opposite to some (A) random or (B) known effect? Answer A: Aggressively use statistical significance analysis to quantify the risk of making Type I errors (false positives) Answer B: Look for plausible natural explanation(s) for every detected anomalous pattern 2. Where to look for these spikes? Ideal answer: Everywhere! However, typical data is highly dimensional multiplicity of possible projections hard to achieve desirable throughputs Example: Spatio-temporal distribution of daily counts of cases of leptospirosis in Sri-Lanka and Tamil Nadu state of India 7

8 One Way to Achieve Scalability: Replace Raw Data with Sufficient Statistics Pre compute key statistics about data ahead of its extensive analyses in order to amortize the bulk of the costs of future computations Example: Using Contingency Tables to represent categorical data Testing hypotheses about categorical data typically requires counting (co )occurrences Precomputing counts makes the future costs of analyses independent on the data size E[ P(Flyfisher Canadian) ] = NumberOf(Canadian Flyfishers)/NumberOf(Canadians) N=7 Records Raw data Canadian? Flyfisher? HockeyFan? Canadian=0 0 Flyfisher=0 1 1 Flyfisher=1 2 2 Contingency table M=3 Attributes Canadian=

9 One Way to Achieve Scalability: Replace Raw Data with Sufficient Statistics Pre compute key statistics about data ahead of its extensive analyses in order to amortize the bulk of the costs of future computations Example: Using Contingency Tables to represent categorical data Complaint: Testing hypotheses about categorical data typically requires counting (co )occurrences Contingency Precomputing Tables counts can makes reach the unmanageable future costs of analyses sizes (numbers independent of cells) on the if data the size underlying E[ P(Flyfisher Canadian) data highly ] = NumberOf(Canadian dimensional and Flyfishers)/NumberOf(Canadians) if the involved variables can assume many different values N=7 Records Raw data Canadian? Flyfisher? HockeyFan? Canadian=0 0 Flyfisher=0 1 1 Flyfisher=1 2 2 Contingency table M=3 Attributes Canadian=

10 All Dimensional Tree: Example of a More Pragmatic Representation 1. Replace the contingency table with a tree It represents the same counts of co occurrences [Moore & Lee, 1998] A1=* A2=* C=8 Vary nodes contain queries in which specific attributes are instantiated Vary A1 Vary A2 Data A1 A A1=1 A2=1 C=0 A1=1 A2=* C=1 Vary A2 A1=1 A2=2 C=1 A1=2 A2=1 C=2 A1=2 A2=* C=3 Vary A2 A1=2 A2=2 C=1 A1=3 A2=1 C=1 A1=3 A2=* C=4 Vary A2 A1=3 A2=2 C=3 A1=* A2=1 C=3 A1=* A2=2 C=5 Count nodes store queries and the corresponding counts of the records of data matching them 10

11 2. Take advantage of sparseness and redundancies in data Do not store anything with zero counts and do not store sub trees of Most Common Values Data A1 A All Dimensional Tree: Example of a More Pragmatic Representation NULL A1=1 A2=* C=1 Vary A2 mcv->2 NULL (mcv) NULL (mcv) Vary A1 mcv->3 A1=2 A2=* C=3 Vary A2 mcv->1 A1=2 A2=2 C=1 NULL (mcv) A1=* A2=* C=8 A1=* A2=1 C=3 Vary A2 mcv->2 This tree may consume much less memory than the equivalent Contingency Table NULL (mcv) Yet, we can still cheaply re compute all of the removed counts 11

12 Collective Mind Enables Fast Processing of Large and Complex Data It is fast and scalable Advanced statistics; Smart data structures; Fast algorithms Response times to complex count queries can be reduced by 1 3 orders of magnitude when compared to alternatives Efficiency enables massive scale multivariate analyses of Big Data Visualizations of data at interactive speeds Automated and highly responsive ad hoc analyses Key practical benefits: Comprehensive searches for unusual patterns made possible We don t know what we don t know dilemma can be substantially mitigated Interactive visualizations and queries boost awareness of issues Side notes on performance 1. Query response times do not depend on the number of records in data Largest set loaded so far: 125M records, 15 dimensions Minutes to screen for patterns over 4 dimensions, evaluating 4.5B hypotheses 2. Memory footprint varies with complexity of data Most complex data loaded so far: 7.7M records, 19 dimensions, 5.2*10 25 unique data cube cells, 478B of them with non zero counts Requires 10GB of memory 3. Speed can be traded for memory 12

13 How Valuable are Early Warnings? normal operations Current Processes crisis undetected investigation solution being implemented normal operations parts exchange rate time 13

14 How Valuable are Early Warnings? normal operations Collective Mind Gains Time crisis undetected investigation solution being implemented normal operations parts exchange rate time gain time gain time 14

15 Early Warnings Can Reduce Part Exchanges normal operations Caused by Systematic Problems crisis undetected solution being implemented normal operations parts exchange rate exchanges avoided time 15

16 Example: Collective Mind in Support of the F 16 Health Of Fleet Reports Evidence obtained with help from the F 16 Weapons System Supply Chain Management Systematic failures identified earlier F 16 Onboard Oxygen Generating System Concentrator Collective Mind Massive Screening produced an early warning that the component was experiencing significant supply issues; base was forced to cannibalize to meet mission requirements Early warning saves multiple repairs for substantial cost avoidance Issues previously undetected F 16 Digital Electronic Engine Control Unit Pre existing process would NOT have driven F 16 WS SCM to this item Problems continued intermittently for a year 16

17 Those Examples Tracked Remove And Replace Maintenance Actions The Data Offers Many More Views Typical maintenance data spans multiple streams and multiple dimensions Streams: maintenance records, built in test, vibration, configuration, flight data, supply, etc. Maintenance data dimensions: actions taken, when discovered, aircraft configuration, mission type, squadron, etc. This leads to billions of potentially interesting projections of data Traditionally, data surveillance is selective and of limited sensitivity Comprehensive approach is often deemed computationally infeasible Analytic resources come in short supply Risk of missing critical clues is substantial Emerging issues identified later than they could CM capabilities address those challenges: 1. Massive Screening of highly multivariate data for abnormal patterns (fleet level) 2. Systems Performance Monitor (detecting Bad Actors ) 3. Exploration of identified patterns across streams 4. Explanation and prediction of patterns Using routinely collected data (doing more without more) 17

18 Millions $250 $200 $150 $100 $50 Quantified Task: Avoided Cost of Part Exchanges Evidence obtained with help from the F 16 Strategic Analysis Support Section and the US Air Force Cost Analysis Directorate $0 Currrent F-16 Cost Avoidance Potential F-16 Cost Avoidance USAF Aircraft Potential All Military Aircraft Potential Fleet Status Reports Poor Performing Aircraft Bad Actor Components Total Those savings only reflect avoided exchanges More potential, not yet estimated firmly, includes a few relatively straightforward benefits: Improved equipment availability Improved mission capability Reduction of analytic efforts Costs of transition to regular use is not included in the estimates F 16 : Return from subset of items analyzed from early detection and mobilization: $6.5M p.a.; Expected return once fully deployed: $18.0M p.a. More recent estimate: >$100M p.a. from just one Collective Mind capability Observed cost of integrating new platform data: Once established with Data Universe, adding new aircraft type took 8 man hours to setup Transfer of CM Massive Screening, System Performance Monitor and Basic Exploration and Explanation from USAF F 16 to USN V 22 environment took 1 man month Larger effort to integrate into new analytic processes Modest additional investment can enable scaling Collective Mind throughout DoD aircraft fleets and throughout all DoD equipment fleets wherever logistics data warehouses exist 18

19 Additional Capabilities of Collective Mind: Fusing Evidence from Multiple Sources of Data Maintenance Records Vibration Measurements Some vibration exceedences correlate with parameters of flight Holistic View Built In Diagnostics Flight Parameters Certain types of vibration alerts can be forecast to enable preventative maintenance Some of them occur in particular flight regimes 19

20 Example of a CM Massive Screening for Changes in Cross Stream Correlations Find pairs of Built In Test reported faults (BIT) followed by specific part replacements (MAF) that show substantial changes in cooccurrences This graph shows the result of using a 6 month analysis window slid daily, computing BIT MAF succession probability lift over time (blue) Lift = P(MAF BIT) / P(MAF) The particular part type is being replaced following the specific BIT message much more often in the second half of 2008, then the peak repeats in 2009 Probability( MAF after BIT ) Date MAF R&R count BIT fault count 20

21 Example of CM Cross Stream Explanatory Analysis: Explaining V 22 Vibration Exceedences Example: One vibration channel notoriously triggering exceedences in flight Ground tests: Re Test OK look like false alerts Hypothesis: Can they be explained as e.g. environmental artifacts and not signs of failure? Perhaps they correlate with the parameters of flight? Approach: Use flight parameters (~60 dimensions) as input data for Collective Mind predictive model that will learn how to discriminate: Flight parameter snapshots coinciding with the exceedences The rest of flight data If the trained model can accurately tell when the exceedences occur, these exceedences could be explained with the instantaneous parameters of flight 21

22 Example: Explaining Vibration Exceedences One example aircraft and one example exceedence type: actual exceedences Predicted probability of exceedence CM flight data alert signal magnitude Flying hours Collective Mind utility potential: Upfront dismissal of fake failures reduces maintenance workload 22

23 Example: Explaining Vibration Exceedences One example aircraft and one example exceedence type: actual exceedences Predicted probability of exceedence Periods of specific flight conditions CM flight data alert signal magnitude Flying hours Collective Mind capability: Identify key factors that allow accurate prediction of exceedences 23

24 How many detections are true Predicting Vibration Exceedences: Example Results Predicting specific type of vibration exceedences to occur between 10 and 40 flying hours from now: About 50% of all these exceedences could be reliably predicted by CM with lead time of flying hours and ~40% of those flagged by CM would be true positives How many actual events predicted The precision/recall trade off can be optimized against cost benefits The precision/recall trade off can be varied by optempo Potential utility: Anticipate future maintenance need and execute it preemptively i.e. before the exceedence actually occurs 24

25 Summary of Key Capabilities and Areas of Impact Capabilities: Early Warnings of emerging crises Fleet wide: Comprehensive monitoring of data for new drivers Item level: Detection of Bad Actors Data Fusion Explanation of patterns and trends Prediction of future events Dimensions of Impact: Cost avoidance E.g. avoided exchanges of parts Improved readiness and equipment availability Better visibility of issues Doing more without more Very little cleaning of data required Transferable to new platforms Example results: Clickable list of findings sorted by statistical significance Each of them identifies a view of data that pertains to one detection Temporal visualization highlights unexpected activity Interactive drill downs, slicing anddicing, and pivoting, help interpreting the results 25

26 Collective Mind: Summary 1. Collective Mind leverages new computational capabilities to support proactive approach to Fleet Health Management 2. It uses statistical Big Data mining and predictive trending to monitor routinely collected data for early indications of reliability issues 3. Capabilities developed so far have demonstrated utility for USAF and NAVAIR aircraft maintenance, built in test, vibration, and flight data 4. Applicability is not limited to USAF and NAVAIR Benefits should be realizable across multiple DoD equipment management organizations, and beyond 26

27 Examples of Additional Opportunities Throughout DoD, DHS and IC Organizations DoD maintenance, logistics, and supply Expand to 14,000 aircraft, 36,000 vehicles, 300 ships, 800 missiles plus a myriad of smaller equipment, 50% of which is supported organically by 650,000 personnel Active operations E.g. identify emerging trends and patterns in enemy activity through sigacts data (IARPA and DARPA efforts) Personnel health and safety E.g. identify emerging trends and patterns in medical care data, track effectiveness of preventative care E.g. monitor safety of food supply through predictive analysis of inspection, microbial testing, and disease data (current deployment through USDA, ongoing research efforts through USDA and CDC) Threat detection and characterization E.g. nuclear threat detection at ports of entry and within perimeter (current research an transition efforts through DHS (DNDO, CBP), DoE (NA 22) and DoD (DTRA)) Contracting, acquisitions, and engineering E.g. tracking actual performance vs. contract for better risk awareness, budget over run mitigation, online audits for performance based contracting, etc., with the goal to reduce cost, acquisition time, and risk E.g. monitoring impact of design changes, interactive trace backs to attribute adverse events, etc. Enterprise knowledge and personnel training E.g. identification of the best and the worst practices based on performance, retrieval and retention of working solutions to mitigate effects of personnel turnover, etc. 27

28 Example: Nuclear Threat Assessment (Joint work with Lawrence Livermore NL, support from DNDO, CBP, NA 22 and DTRA) Problems addressed: 8M cargo containers enter US each month, 0.4% of them trigger radiation alerts Nuclear threats that are already in the US need to be detected and assessed Requirements of high throughput and high accuracy are hard to meet simultaneously Improved threat determination rates at fewer false alerts True Negative Rate Approach: Use Machine Learning, available historical data, and nuclear domain expertise to: Boost sensitivity of the existing radiation detection systems without increasing false alert rates Utilize various types of information: intensity, spectral, radiographic, contextual Utilize multiple and/or mobile sensors mobile sensor data Supervised Detection Supervised Classification Anomaly Detection New Class Detection new labeled classes explained detections unexplained alerts new phenomena OFFLINE PROCESS U S E R S follow-up investigation Log (False Negative Rate) Stationary sensing: Learn to classify threat using radiation measurements and contextual information (cargo manifest, etc) Mobile sensing: Use multiple observations to probabilistically map threats of known types as well as undetermined yet potential threats 28

29 Example: Fast Predictive Queries in Large Graphs Commute times demonstrated to yield the most accurate predictions of node labels wikipedia pages topics/ words papers authors Our GRANCH algorithm allows queries on 20M node graphs to process in minutes on a single processor Demonstrated through the DARPA Guard Dog program A "free ontology" yields accuracy improvements simply by linking in wikipedia pages Same tool used for social, financial, event, knowledge, or heterogeneous networks Sample Intelligence Queries on Graphs with People, Places, Events Query an event to find people most likely to be involved Query a set of people to find most likely additional collaborators Query a set of people to find most likely locations for them to meet Query a set of people to find most likely event for them to participate in 29

30 Example: Active Target Selection in Graphs Problem description: Fixed budget in the number of entities to investigate Maximize the number of positives found Approach: Utility bounds allow pruning for a tractable look ahead search Impact criterion approximates further look ahead Results: Up to 10x more hits than uniform sampling 20% more than competing active target selection methods Applications Selecting potentially fraudulent transactions for follow up Identifying suspicious individuals in criminal investigations Product recommendations Scientific discovery Related Capability: Active Survey Poll nodes to estimate the fraction of positives in the network May be used to estimate public opinion/support for products/ideas/people May be used to estimate effectiveness of marketing campaigns 30

31 Thank you! Collective Mind User Quotes: Collective Mind provides unique capability to quickly identify what is driving maintenance. It is the only tool capable of analyzing the volume of data generated by our weapon system Mr. Joseph Smith, F 16 System Program Analyst "As the Project Managers for the CAMEO Team we are excited about the potential Collective Mind enables for a holistic approach for the health and well being of the V 22 Osprey." Mr. Joseph Schmidley, V 22 ALE IPT Lead, and Mr. Sam McNeely, V 22 ALE We expect in excess of $100M annual cost avoidance due to implementation of Collective Mind Early Warning Bad Actor capability in the F 16 program alone, however Asset Management Tracking capability must be stood up and the information becomes discoverable." Mr. Robert Riegert, Project Lead, Systems Lifecycle Integrity Management (SLIM), A4ID, Department of the Air Force Collective Mind could automate analyses that lead to tremendous DoDwide savings. Dr. Robert Neches, Director, Advanced Engineering Initiatives, OASD(R&E) 31

32 Contact Information 32