Techniques for Early Warning of Systematic Failures of Aerospace Components

Transcription

1 Techniques for Early Warning of Systematic Failures of Aerospace Components Artur Dubrawski Auton Lab, Carnegie Mellon University 5000 Forbes Avenue, NSH 3121 Pittsburgh, PA Norman Sondheimer Electronic Enterprise Institute University of Massachusetts Amherst Amherst, MA Abstract Fleets of aerospace equipment are managed through carefully controlled supply chain processes. When any of the planning assumptions are no longer valid, an unexpected demand on maintenance and supply can develop, reducing the availability of equipment. Earlier detection of unexpected trends in demand can mitigate their impact. Systematic failures of components in man-made fleets bear an analogy to disease outbreaks in the human community. This paper presents evidence that mathematics originally developed for public health surveillance can be effectively used in support of aerospace fleet management. The central results include the formulation of new datadriven analytics to allow maintenance and supply managers to be notified about emergence of a possible problem substantially earlier than it was possible before, the ability to routinely screen incoming data for indications of problems of a broad variety of types even if their number is very large, and the ability to pragmatically prioritize investigative efforts according to the statistical significance of the detections. 12 TABLE OF CONTENTS 1. INTRODUCTION CURRENT AND PROPOSED APPROACHES MULTIVARIATE APPROACHES TO HEALTH SURVEILLANCE UTILITY OF COMPREHENSIVE SEARCHES FOR UNEXPECTED PATTERNS OF MAINTENANCE ACTIVITY THE COLLECTIVE MIND APPROACH EXAMPLES INTEGRATION INTO FLEET OPERATIONS CONCLUSION ACKNOWLEDGEMENTS... 8 REFERENCES... 8 BIOGRAPHIES INTRODUCTION Fleets of aerospace equipment are managed through carefully controlled supply chain processes. When any of the planning assumptions are no longer valid, for example, due to a change in the op-tempo, or a delivery of an order of out-of-specifications parts, or fielding of an ill-conceived /11/$ IEEE 2 IEEEAC paper#1719, Version 3, Updated 2011:01:12 maintenance procedure, an unexpected demand on maintenance and supply can develop and the availability of equipment may decrease. Detection of early warning signals in maintenance and supply activity is critical to dealing with such problems proactively. The complexity of aerospace fleets makes it particularly difficult for human managers to promptly recognize leading indicators of emerging patterns of systematic failures. Only when supply nears the safety stock level, or when the availability is significantly affected, it is likely that attention will be paid. The possibilities for systematic failures stem from a variety of only partially observable processes, making it difficult to completely prevent their occurrence. Common organizational processes that either support or constitute operations can characterize their main categories: Design. Equipment enters service operations with estimates of expected life for the unit as a whole and for its components. Inventory plans and maintenance plans are based on these models. This category captures the cases where these assumptions are wrong. These failures are often seen when long service life equipment is subjected to a design change in mid-life. Such changes are not subjected to the same extensive and expensive system tests as the new equipment. Supply. The manufacture and delivery of replacement parts, whether new or refurbished, can fail in various ways. The quality of the components can fail to deliver expected performance. Scheduled deliverables can be missed. Maintenance. As in equipment design, maintenance processes come with estimates of effectiveness. These estimates can be wrong. As with quality of parts, the actual maintenance effectiveness can deviate from plan. Operations. All equipment is designed to handle the expected patterns of usage, for example, some automobiles are designed for on-road use only and some for both on-road and off-road use. Inevitably, equipment usage sometimes differs in practice from its intended scope. It is quite common to find equipment being used well past its expected longevity, and its structural components that were never expected to fail wearing out. Experienced fleet managers have seen many examples of all of the above adverse scenarios. 1

2 When maintaining fleets of complex equipment, it is extremely difficult to stay on schedule and on budget while meeting the pre-set availability minima. The unexpected can be expected. One of the greatest challenges is managing through rushes of unexpected failures of a single type of component. Repair shops are then required to work overtime. Broken parts must be express shipped. Field inventories are becoming depleted. Repair shop inventories of their constituent parts are run down. Suppliers cannot respond to the increased demand. Prices rise on any public spares markets. Managers spend their time expediting their way through the crisis, while they are searching for the root cause and for the appropriate response. Too often, availability of the fleet is significantly affected. Systematic failures of components in man-made fleets bear a structural analogy to disease outbreaks affecting human communities. When an out-of-spec order of parts enters the fleet supply, in some ways it is like a new virus entering the community in that it eventually leads to increased demand on equipment maintenance system or, respectively, to an increased demand for health care services. Often, healthcare providers are unaware of the system-wide nature of the problem until relatively late in its progression. Fortunately, data mining and machine learning technology based on time series analysis has been demonstrated to enable early warnings of the advent of human epidemics by observing the operation of the health care system. Statistically significant anomalies, e.g., in the number of patients reporting with certain symptoms, or in the volume of sales of certain types of non-prescription medicines, can draw public health officials attention, allowing them to investigate issues and mitigate their consequences earlier than otherwise possible. This paper presents evidence that similar mathematics that alerts to unusual patterns in health care has value in alerting to unusual patterns in aerospace fleet maintenance. The central results include (1) Formulation of a new data-driven analytics to allow maintenance and supply managers to be notified about emergence of a possible problem substantially earlier than it was possible before, (2) Providing the ability to routinely screen incoming data for indications of problems of a broad variety of types even if their number is very large, and (3) The ability to pragmatically prioritize investigative efforts according to the statistical significance of the detections. We illustrate the concept using one of a variety of feasible anomalous pattern detection algorithms, a bi-variate temporal scan. It monitors the time series of daily counts of events of interest in reference to the baseline time series of choice, computing a p-value of a test of significance of any observed elevation of activity of interest. The algorithm has been integrated into a time series visualization and analysis prototype tool. The tool has been successfully field-tested in the operation of military aircraft fleets of various sizes. Results show a 10 to 20% improvement in the content of monthly watch lists produced to monitor potential component problems, when compared to the preexisting processes. 2. CURRENT AND PROPOSED APPROACHES Fleet managers are aware of the need to identify emerging systematic issues as early as possible. For example, the United States Air Force (USAF) requires its fleet managers to maintain Reliability and Maintainability (R&M) programs that promote the ability to identify and correct system deficiencies before they affect combat capability. Maintainers collect and report all maintenance actions and product quality deficiencies. The fleet manager is required to develop a proactive R&M program to analyze the collected data and to identify, track, assess, and correct R&M deficiencies impacting their assigned air and space equipment. Experienced equipment managers are always watching out for the tell-tale patterns that may evidence the emergence of a new issue. To help them, the USAF supports a set of maintenance metrics. As the USAF Metrics Handbook for Maintenance Leaders describes it, many metrics can be considered leading indicators [1]. Such indicators directly affect maintenance system s capability to provide resources to execute the mission. Table 1 shows a set of metrics associated with fleet availability, taken from the Handbook. The USAF equipment managers routinely monitor those factors. They look for unexpected behavior, in particular for failures to stay within limits of expected performance, and for trends in the values, especially whether the observed metric reveals a dramatic change in a wrong direction. Ground abort rate Air abort rate MAF total air abort rate (home station air aborts + J diverts) Code 3 break rate 8-/12-hour fix rate Repeat rate Recur rate Logistics departure reliability Average deferred/delayed discrepancies per aircraft Discrepancies awaiting maintenance (AWM) or awaiting parts (AWP) MSE rate Functional check flight (FCF) release rate CANN rate Issue effectiveness rate Stockage effectiveness rate Bench-stockage effectiveness rate Mission capability (MICAP) aircraft part rate Average repair cycle days Phase flow a phase time distribution interval (TDI) Table 1. Leading indicators of fleet availability In practice, fleet analysts appear to pay the most attention to mission capability rates which are often indicative of when the need for maintenance of a part, or for resupply of a part is grounding aircraft. In addition, metrics related to maintenance work hours and part cannibalization rates (CANN, when working parts are moved between aircraft). The analysts look out for both the parts with the highest rates and those showing the fastest increasing trends. In addition, engineering staff monitors the achieved life of 2

3 parts against their expected failure rates. Engineering life estimation techniques, including the use of Weibull statistic and other reliability analyses, can be used as inputs for actuarial calculations. These approaches work best when sufficiently large numbers of failures are identified. Another way of approaching risk management is through development of mathematics to support condition-based maintenance, equipment fault diagnosis, and failure prognosis [2]. These approaches work on a fault-by-fault and failure-mode-by-failure-mode basis, developing sensors and physics driven models for each type of fault or failure. This is potentially a very useful framework, however it often leads to adding costs and additional failure modes through the additional sensors. We observe that tangible utility can be achieved by much simpler means of mining routinely collected maintenance activity data for unexpected emerging patterns. We have shown that an unusual activity may be an indication of a potential systematic issue [3]. To detect potential emerging systematic flaws, it should be possible to look at the fleetwide records of component failures, maintenance actions, error codes and the like. Using efficient algorithms it should be possible to exhaustively search through a large number of crisis scenarios in multivariate data (e.g. monitoring for patterns across individual work unit codes, bases, aircraft blocks, or any combination of such criteria) and report the most unusual findings sorted by their statistical significance, to human analysts for further evaluation. As soon as the emerging systematic issue is sufficiently reflected in the monitored data, it should be possible to alert the users and, effectively, to achieve predictive trending, giving the fleet manager an opportunity to be proactive in mitigating the unexpected issues. Our experiments so far have been focused on alerting based on unusual patterns of unscheduled maintenance and inherent failures. However, the same principles would apply if it were used with any of the metrics listed in Table MULTIVARIATE APPROACHES TO HEALTH SURVEILLANCE We have been inspired in our work by the success of application of event detection algorithms to monitor data related to human health for rapid identification of emerging epidemics [4]. Epidemiology is a familiar domain of temporal and spatio-temporal analyses of data which may carry leading indicators of problems in public health. Detecting such indicators in incoming field data has been demonstrated to save lives. Traditionally, monitoring for diseases has been done using univariate methodologies borrowed from Statistical Process Control (SPC, familiar to fleet health managers), in which current observations are compared against the expectations derived from historical reference data, while accounting for predictable fluctuations due to e.g. seasonality of predictable normal disease activity. SPC is also useful in monitoring stability of multivariate temporal processes [5]. The simplest (and the most common) multivariate version of a SPC chart that can take advantage of correlations between multiple involved variables is the Hotelling method [6]. It uses historical data to model a joint distribution of a set of variables under null hypothesis. For example, when concurrently monitoring sales of cough syrup and nasal spray in local pharmacies, we could model their statistical distribution as a two-dimensional Gaussian with a mean daily cough sale count, mean nasal spray count and a covariance matrix derived from data observed over the previous year. If the volume of sales observed today falls outside of e.g. the 98% confidence ellipse of the covariance matrix, an alert can be issued. This could happen for three reasons: (i) cough sales are surprisingly low or high; (ii) nasal sales are surprisingly low or high; or (iii) neither set of sales are abnormal by themselves, but their ratio is abnormal. This approach has been successfully applied in the DoD ESSENCE framework (Electronic Surveillance System for the Early Notification of Community-based Epidemics), [7, 8]. Its appeal is primarily derived from its simplicity, however Hotelling s method has been found to perform poorly at distinguishing changes in mean of the joint distribution from changes of the structure of correlations between variables. Some of those deficiencies can be addressed by using other control charts, such as multivariate Cumulative Sum (CUSUM, [9, 10, 11]), or Bayesian Networks to efficiently represent multivariate joints [12, 13, 14]. However, as it often happens in practice, highly multivariate scenarios in which data comes in a relatively short supply cannot be reliably addressed using full joint distribution models. Practical-minded alternatives employ searches through the feature space for subsets of dimensions which contain statistically significant departures from what is expected [15]. We found these approaches particularly useful in the context of our application. 4. UTILITY OF COMPREHENSIVE SEARCHES FOR UNEXPECTED PATTERNS OF MAINTENANCE ACTIVITY One of desirable features of a data-driven approach to equipment (or human) health surveillance is the ability to comprehensively screen records of maintenance (or, respectively, patient visits) for signs of emergence of trends of unexpected activity. Any such trend that cannot be readily explained using information derivable from the historical reference data and/or from some current baselines should be presented to human analysts for evaluation. Typically, fleet managers are aware of indicators of certain kinds of problems that they had encountered in the past, but they are also often concerned that they may be overlooking indicators of other, perhaps subtler, less obvious, or less common issues. We argue that by enabling exhaustive surveillance of perhaps all possible reasonably complex projections of data, the resulting comprehensive screening process would be able to flag all statistically significant and potentially relevant anomalous patterns in maintenance activity. It would help addressing the we don t know what 3

4 we don t know problem. There are a few issues making the comprehensiveness of such monitoring systems a challenging goal. The key is the computational costs of exhaustively searching through highly multivariate data. Maintenance records are typically stored in transactional databases, and each entry in such data corresponds to a maintenance event. It is dated and/or timestamped, and contains a number (typically in order of ) of categorical descriptor fields whose values characterize the event. Commonly encountered descriptors include coded explanations of the original problem, when and how it occurred, what has been done to investigate it and to remedy it, followed by a number of fields describing demographics of the aircraft (such data scheme is strikingly similar to what can be seen in a record of outpatient department visits, where each entry contains some demographic information about the patient, reported symptoms and signs, initial diagnoses, initial tests and results, prescribed treatment, etc.). In general, the number of possible multivariate projections of such data (equivalently, the number of unique select database queries) can be huge, and the fleet health analysts are often forced to focus their attention on just a subset of predefined queries selected for routine monitoring. This exposes them to the risk of missing critical clues if they are not included in the list of what in the world of epidemiology is often referred to as reportable diseases. 5. THE COLLECTIVE MIND APPROACH One of the specific goals of our work is to develop algorithms for early detection of emerging patterns of crises that manifest in maintenance records. Often, adverse effects of subtle changes to maintenance or operating procedures may go unnoticed for long periods of time. They become apparent only when a substantial percentage of the fleet becomes affected. We have shown that by monitoring routinely collected maintenance data for unusual increases in observed levels of specific activities, it is possible to identify emerging problems substantially earlier than in the preexisting framework. Our approach relies on computationally scalable searches for projections of data which reveal the most unusual temporal patterns. Each considered projection is analyzed using one of the aforementioned SPC algorithms for statistically significant abnormalities. Results of analyzing very many projections, after taking into account the risks of multiple hypotheses testing, are ranked according to statistical significance of patterns detected in them and presented to the users in a descending order. The analysts can then review them one by one using interactive data navigation and visualization tool developed for this purpose, in order to determine practical utility and relevance of the retrieved information. Findings with a potentially substantial impact on fleet performance can promptly trigger mitigative or corrective actions. This functionality provides a framework for comprehensive monitoring of the maintenance records at the level of fleet (or its subsets) for patterns of abnormal levels of activity. It takes advantage of the fact that the fleet of aircraft is composed of many devices of either identical or very similar pedigree (the Collective Mind paradigm). Therefore we may expect that their individual performance (in terms of e.g. reliability) can be predicted based on the observed performance of their peers which share similar usage histories. The statistical models built for making such predictions benefit from evidence aggregated across many such peers (aircraft or components). To enable exhaustive searches through highly multivariate data we employ a cached sufficient statistics approach to represent data. The particular data structure, called T-Cube [16, 17], is an equivalent to the data cube concept known in On-Line Analytical Processing (OLAP) and Business Intelligence applications. It differs from a regular data cube in that it can be used to very efficiently answer all conceivable (not just the most common) queries against databases of multidimensional time series, including disjunctive queries. Rapid access to complex data not only makes advanced data-intensive analytics feasible, but it also enables user-level data navigation (drill-downs, roll-ups, visualization) at the interactive speeds. T-Cube also allows for rapid execution of massive screening through data for statistically significant patterns at different levels of aggregation. The exhaustive search strategy guarantees that no event of interest will ever be missed. Successful applications in the domains of public health and food safety indicate that the combined benefits lead to improved situational awareness of the analysts working with information systems powered with T-Cube. It was also found very useful in the fleet health management application. Once a particular projection of data has been selected in the subsequent step of the comprehensive screening procedure, it is processed by the anomaly detection algorithm. We use for this purpose the bi-variate temporal scan, however, many other temporal anomaly detection algorithms could be used in its stead. Typically, it monitors the time series of daily counts of events of interest (such as unscheduled maintenance of the navigation system on all fighter jets subjected to the recent weapons control software upgrade) in reference to the baseline time series of choice (such as the daily counts of all kinds of unscheduled maintenance, except for navigation system, performed on the fighters). The algorithm retrieves from the maintenance records the target and baseline counts occurring within the current time window of interest (such as the current week) and during the period of reference (such as the prior 12 months) and performs a statistical test of independence (either Fisher s exact or Chi-square test) of entries in the resulting 2-by-2 contingency table of counts (see Table 2). A low p-value from that test indicates an unusual departure of the monitored navigation system related activity during the current week which cannot be explained by a coinciding 4

5 change in the general level of the fighter jet fleet maintenance activity. Such finding may become a source of concern and a call for further investigation by the fleet managers and analysts. Current Reference Target α β α+β Baseline γ δ γ+δ α+γ β+δ α+β+γ+δ Table 2: Structure of a contingency table of counts used by the temporal scan anomaly detection algorithm. Note that only the four highlighted cells form the table. The marginal sums are shown for convenience. The bi-variatedness of the temporal scan algorithm brings about a slight advantage over the more common univariate alternatives in that it can dismiss alerts when the target time series shows an increase which cannot be explained by its own historical distribution, but which is correlated with a corresponding increase in the baseline time series. This way, an unusual ramp of maintenance activity corresponding to the navigation system would not be reported as suspicious if it coincided with an increase of the overall maintenance activity across the fighter fleet, which might have been easily explainable by simply a temporary increase in tempo of operations. However, if the number of navigation system repairs is increasing while the total maintenance activity is not, the alternative explanation may include some unexpected, distinct and potentially adverse process. Temporal scan can be run in a prospective mode (using the most recent data to compute the current counts in Table 1), or in a retrospective mode, where the current window of interest slides along the temporal axis, and anomalousness is evaluated independently for each such period. More specifically, for a given combination of data attribute values (e.g., a specific subset of work unit codes, action taken codes, and geographic location of occurrence), a given time window width (e.g., 28 days), a given baseline of maintenance activity (e.g., all maintenance actions ), and a given p-value threshold for statistical significance, the temporal scan algorithm slides the time window through the time period of the dataset, one day (or flying hour, or sortie) at a time. If the count of maintenance events inside the time window having the specified attribute values is statistically significantly high relative to the count of events outside, given the baseline activity, then the volume of data selected by that combination of attributes for that temporal window cannot be explained by simple means such as an increase in baseline activity (in this example the overall fleet maintenance activity), and therefore the corresponding period of time is flagged as anomalous. Collective Mind massive screening executes temporal scan for all combinations of data attribute values of up to certain number of attributes at one time (typically three higher numbers often produce overly specific findings with relatively small support of data). For each such combination, the procedure reports the attribute values and the most statistically significant time window (assuming at least one time window was significant), in ascending order by computed p-values (that is, from the most anomalous to the least-but-still-significantly anomalous). These can then be examined and further drilled-down by an analyst, engineer, or item manager to determine whether they indicate true problems or not. 6. EXAMPLES Figure 1 depicts two screenshots of the T-Cube Web Interface (TCWI) configured to support Collective Mind predictive trending capability (for more information about TCWI please visit The left pane in Figure 1 shows time series of daily counts of maintenance activity accumulated across the whole fleet of one type of aircraft, including all kinds of maintenance actions related to all components and subsystems. The right pane presents the result of comprehensive search for components revealing statistically significant increases in unscheduled maintenance that cannot be explained with simple means such as natural variance or changes in overall fleet activity. The top of the window contains the list of such findings sorted according their statistical significance. For instance, the sixth most unusual component, marked with work Unit Code of 286, during the 21 days prior to and including November 5 th, 2010, was involved in 209 unscheduled maintenance events, counted across all aircraft in the fleet. Based on historical performance of this component, and on the current fleet activity, we expected this count to be closer to 87 instead. Such discrepancy cannot be explained very well by random chance: the p- value of the statistical test of significance is extremely low (in the range of ). By clicking on this entry of the list of findings, the analyst can bring up a plot of time series of maintenance activity shown in the lower part of the right pane of Figure 1. The red line traces the p-value of the Chi-square test computed for all days within the displayed window of time. We do see a relatively fast increase of this alert signal beginning in the first half of October. That is when the earliest signs of a potential emergence of a problem could have been statistically identified. From there, the analyst can use additional interactive visualization and data analysis functionality provided by TCWI to investigate the finding in more detail and in the broader context, as needed to identify its actual impact and to determine the optimal corrective actions. One of the typical scenarios of beneficial analysis enabled by Collective Mind approach involves issues that are easy to overlook. Some of them could not be isolated with the current process because of insignificant observable departures from the expected values of traditional metrics listed in Table 1. In one such instance identified in historical data, an increased maintenance activity shown in a specific 5

6 Figure 1 Example detection of an emerging pattern of unexpected maintenance activity performed using T-Cube Web Interface predictive trending tool. subset of data was seen more clearly than a barely noticeable increase in the total non-mission capable hours. This problem originally impacted a few tens of aircraft over a period of one year. Majority of these incidents could have been avoided using the Collective Mind tools. In another documented case of retrospective testing, Collective Mind analytics would have produced the first alert three months earlier than the current process, potentially mitigating the spread of the systematic issue which affected more than one hundred aircraft. Comprehensive searches for anomalous patterns in multivariate maintenance data enable an effective early warning mechanism. It allows for earlier adjustments to the processes, earlier resolution of causes of the detected problems, and reduction of the number of equipment and part exchanges. 7. INTEGRATION INTO FLEET OPERATIONS The arguments and results above should all make intuitive sense: the earlier in the know the better, when it comes to managing aerospace fleets. It is generally assumed that the earlier the fleet manager can become aware of emerging systematic problems, the more effective the management of the equipment fleet could be, and the less expense will be incurred, the more equipment will be available, and the safer operations will be. Yet our experience in field-testing of these ideas has run into doubters. They have argued that even when you find some classes of problems there is nothing that can be done to impact the adverse process; that 6 the less reliable components must remain in the fleet; that they need to be informed much earlier than feasible to react effectively; or that latencies of the budgeting process preclude effective responses. Of course, not all early warnings can be used productively, for example, when the staff is already overextended beyond their ability to refocus. That can be compensated to some extent by utilizing warnings on issues that could be instantaneously corrected such as identification of straightforward misunderstandings in maintenance procedures or in supply logistics. Either way, effective integration of Collective Mind approach into fleet operations remains a challenge and it is the topic of our ongoing work. We have been fortunate to observe the metrics described here being used in managing an operation of a fleet of several hundred aircraft for a period of one year. The management team uses a monthly Health of the Fleet review to manage availability and expense. In preparing for this review, a watch list is created from components showing the largest deterioration with respect to key metrics. Typically, these use traditional leading indicators such as maintenance hours, failure rates, or increases in these rates. Approximately two weeks of labor is allocated to preparing the list. That effort allows approximately 20 items to be considered in detail for the final validated list of up to 10 items. This creates a natural insertion point for the Collective Mind early warning methodology. The preparation of the watch list proceeded as before. In addition, a report was made available to the management team showing approximately 100 components with the most

7 unusual patterns identified by the time series metrics described here. This was out of a complete set of over 3,500 components. In approximately two hours the analyst in charge of preparing the watch list was able to scan the items on the automatically generated Collective Mind list, and eliminate items already under consideration and those for which a ready explanation was available. Of the remainder, the organization visually inspects the data used to produce the rankings. Only a limited number of the items from the original alert list make it to the final watch list. The organization estimates 10 to 20% of the contents of their watch list - approximately one to two items per month, perhaps 18 per year - have been changed by the use of the new metrics. Estimates of cost saving are in excess of $1,000,000 using a very conservative estimate that the mathematics gives just one month worth of the effective early warning for just a half of the alerts actually added to the list. One month is a relatively low estimate. Issues going unnoticed for months are well known. The calculation of value here is based on the identification of systematic issues as compared to the utility of the current methods. Because the proposed approach is computationally efficient, the watch list could be re-checked whenever the data are updated. This can be done on an ad-hoc basis or regularly, say, on a daily basis as opposed to monthly as in the current process. Addition of new items can be handled quickly. Those that indicate a correctable problem can be addressed well before the next monthly review cycle. Although this may be a local artifact, the management team only uses a three-year history of maintenance data. The analytic approaches described here currently cover ten years with no noticeable processing delay. The dramatic increase in the number of inspected components results in an increase of attention to maintenance and failure patterns. Even if not all Collective Mind detections are necessarily indicative of actual systematic problems, they may provide feedback bringing about other benefits, e.g., identification of the need for the organization to adjust schedules. User feedback received so far indicates that Collective Mind analytics can be beneficially extended to, at least, engineering and, possibly, supply chain management users. Further, our experience so far has involved a fairly large and mature fleet of equipment. It is conceivable that the maturity of the processes understates the value of the early warning capability. Perhaps in the case of newer or smaller fleets, the attainable value added would be greater. 8. CONCLUSION Fleets of aircraft are managed using carefully controlled supply chain processes. When any of the planning assumptions are invalidated, an unexpected demand on maintenance and supply can develop and adversely affect availability of the equipment. Earlier detection of unexpected trends in demand can reduce their impact. The analytic framework presented in this paper leverages routinely collected records of equipment maintenance to increase awareness of situation of the fleet managers. It stems from the analogy to detecting outbreaks of disease in data pertaining to public health. It enables issuance of early warnings of possible emerging crises of equipment availability, supporting proactive management of the fleet health. Our approach relies on the use of statistical analysis of multivariate time series, as well as data mining and machine learning techniques originally developed for biosurveillance applications. It enables comprehensive searches for patterns of interest. Typical equipment maintenance data spans multiple dimensions (item identification and characteristics, problem description, when and how occurred, actions taken, etc.,) each with a few to very many possible discrete values, indexed with timestamps. In practice, there can be millions of possible projections of such data and each such projection may contain important patterns. This complexity necessarily makes the currently used monitoring processes selective and of limited sensitivity, and a more comprehensive alternative is often deemed either computationally intractable or infeasible due to the constraints to the availability of analytic resources. As a result, the risk of missing critical clues is substantial, and emerging issues tend to be identified later than they could. Collective Mind approach capitalizes on the opportunities for improvement. It offers computational efficiency enabling completion of massive scale searches for anomalous patterns in minutes. As a result, comprehensive surveillance becomes possible and critical clues become much harder to miss. Careful statistical analysis produces ranked lists of findings, so that investigative resources can focus on the most significant and alarming findings. This offers potential for more effective resource allocation. Routine analyses can be automated and run frequently; any significant findings can be automatically communicated to designated recipients. Our analytic approach has been integrated into a time series visualization and analysis prototype tool. The tool has been successfully field-tested in the operation of military aircraft fleets of various sizes. Results show a 10 to 20% improvement in the content of monthly watch lists produced to monitor potential component problems, when compared to the preexisting processes, leading to sizable opportunities for cost avoidance. The prototype tool has been promptly accepted and adapted for regular use by the equipment health analysts. The key practical benefits of our approach include comprehensiveness of screening for patterns and attainable high timeliness of detection of emergence of such patterns. These features allow the fleet health analysts to be much more aware of any developing threat scenarios and more 7

8 agile in their responses to threat before it impacts a large fraction of fleet, than they were before. We believe that the potential advantageous applicability of the methodology and techniques presented here reaches well beyond monitoring records of equipment maintenance. 9. ACKNOWLEDGEMENTS This material is based upon work supported by funds from the National Science Foundation under Grant and the United States Air Force under Contract FA C Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of the National Science Foundation or the United States Air Force. REFERENCES [1] Metrics Handbook for Maintenance Leaders, Air Force Logistics Management Agency, [2] G. Vachtsevanos, F.L. Lewis, M. Roemer, A. Hess and B. Wu, Intelligent Fault Diagnosis and Prognosis for Engineering Systems, Hoboken: Wiley, [3] A. Dubrawski, M. Baysek, S. Mikus, C. McDaniel, B. Mowry, L. Moyer, J. Ostlund, N. Sondheimer and T. Stewart, Applying Outbreak Detection Algorithms to Prognostics, Papers from the 2007 AAAI Fall Symposium on Artificial Intelligence for Prognostics, November 9-11, [4] A. Dubrawski, Detection of Events in Multiple Streams of Surveillance Data: Multivariate, multi-stream and multi-dimensional approaches, Infectious Disease Informatics and Biosurveillance, Integrated Series in Information Systems, Vol. 27, Springer, [5] D.C. Montgomery, Introduction to Statistical Quality Control, 4th ed., Wiley, [6] H. Hotelling, Multivariate Quality Control, C. Eisenhart, M.W. Hastay and W.A. Wallis, editors, Techniques of Statistical Analysis, McGraw-Hill, [9] R.B. Crosier, Multivariate generalizations of cumulative sum quality-control schemes, Technometrics 30(3), , [10] J.J. Pignatiello and G.C. Runger, Comparisons of multivariate CUSUM charts, Journal of Quality Technology 22, , [11] R.D. Fricker, Directionally Sensitive Multivariate Statistical Process Control Procedures with Application to Syndromic Surveillance, Advances in Disease Surveillance 3(1), 1-17, [12] G.F. Cooper, D.H. Dash, J.D. Levander, W.K. Wong, W.R. Hogan and M.M. Wagner, Bayesian biosurveillance of disease outbreaks, Proceedings of the Conference on Uncertainty in Artificial Intelligence, , [13] W. Wong, A. Moore, G. Cooper and M. Wagner, What's Strange About Recent Events (WSARE): An Algorithm for the Early Detection of Disease Outbreaks, Journal Machine Learning Research 6, , [14] A. Dubrawski, K. Elenberg, A. Moore and M. Sabhnani, Monitoring Food Safety by Detecting Patterns in Consumer Complaints, Industrial Applications of Artificial Intelligence IAAI 06, July [15] M. Sabhnani, A. Dubrawski and J. Schneider, Multivariate Time Series Analyses Using Primitive Univariate Algorithms, Advances in Disease Surveillance 4, 112, [16] M. Sabhnani, A.W. Moore and A. Dubrawski, T-Cube: A data structure for fast extraction of time series from large datasets, Technical Report CMU-ML , Pittsburgh: Carnegie Mellon University, [17] A. Dubrawski, M. Sabhnani, S. Ray, J. Roure and M. Baysek, T-Cube as an Enabling Technology in Surveillance Applications, Advances in Disease Surveillance 4, 6, [7] H.S. Burkom, Y. Elbert, A. Feldman and J. Lin, The role of data aggregation in biosurveillance detection strategies with applications from ESSENCE, Morbidity and Mortality Weekly Report 53(Supplement), 67-73, [8] H.S. Burkom, S.P. Murphy, J.S. Coberly and K.J. Hurt- Mullen, Public health monitoring tools for multiple data streams, Morbidity and Mortality Weekly Report 54(Supplement), 55 62,

9 BIOGRAPHIES Artur Dubrawski researches intelligent systems that work, are useful, and make economic sense; and ways to effectively build and deploy them. For over 20 years his work has been driven by realworld applications, currently in the areas of public health, food safety, nuclear safety, health of equipment, and predictive maintenance. It involves investigating machine learning algorithms and data structures to facilitate probabilistic modeling, predictive analysis, interactive exploration, and understanding of data. Dr. Dubrawski directs the Auton Lab and teaches graduate courses on data mining and business intelligence at Carnegie Mellon University. He leads fundamental and applied research projects funded by government agencies and industrial partners. He spent a substantial part of his professional career serving in managerial and technical roles in hightech industry. Norman Sondheimer is co-director of the Electronic Enterprise Institute (EEI) at the University of Massachusetts Amherst and a Senior Research Scientist in the University s Computer Science Department. Dr. Sondheimer has championed efforts in equipment maintenance for many years. He has supervised the fielding of diagnostic tools for medical imaging, locomotive, jet engine and building systems equipment. He helped lead the effort to create remote monitoring centers that are now in service. Most recently, Sondheimer chaired a Defense Advanced Research Projects Agency Information Processing Technology Office (DARPA/IPTO) study team on Self-Aware Platforms and the Collective Mind that 9