Predictive Model Evaluation for PHM

Transcription

1 Preditive Model Evaluation for PHM Chunsheng Yang 1, Yanni Zou 2, Jie Liu 3, Kyle R Mulligan 4 1 National Researh Counil Canada, Ottawa, Ontario K1A 0R6, Canada Chunsheng.Yang@nr.g.a 2 Jiujiang University, Jiangxi, China zouyanni@163.om 3 Dept. of Mehanial and Aerospae Eng., Carleton University, Ottawa, ON, K1S 5B6, Canada Jie.Liu@arleton.a 4 GAUS, Dept. of Mehanial Engineering, Universit e de Sherbrooke, Sherbrooke, QC, J1K 2R1, Canada Kyle.Mulligan@USherbrooke.a ABSTRACT In the past deades, mahine learning tehniques or algorithms, partiularly, lassifiers have been widely applied to various real-world appliations suh as PHM. In developing high-performane lassifiers, or mahine learning-based models, i.e. preditive model for PHM, the preditive model evaluation remains a hallenge. Generi methods suh as auray may not fully meet the needs of models evaluation for prognosti appliations. This paper addresses this issue from the point of view of PHM systems. Generi methods are first reviewed while outlining their limitations or defiienies with respet to PHM. Then, two approahes developed for evaluating preditive models are presented with emphasis on speifiities and requirements of PHM. A ase of real prognosti appliation is studies to demonstrate the usefulness of two proposed methods for preditive model evaluation. We argue that preditive models for PHM must be evaluated not only using generi methods, but also domain-oriented approahes in order to deploy the models in real-world appliations. Keyword: Mahine Learning Algorithms, Preditive Models Evaluation, Generi Methods, Binary Classifier, Prognostis and Health Management (PHM), Prognostis, Time to Failure 1. INTRODUCTION One of the objetives of Prognostis and Health Management (PHM) systems is to help redue the number of unexpeted failures by ontinuously monitoring the omponents of interest and prediting their failures Chunsheng Yang, et.al. This is an open-aess artile distributed under the terms of the Creative Commons Attribution 3.0 United States Liense, whih permits unrestrited use, distribution, and reprodution in any medium, provided the original author and soure are redited. suffiiently in advane to allow for proper planning of maintenane. The ore of PHM is prognostis, whih is defined as an emerging tehnique being able to predit failure before it happened and to preisely estimate remaining useful life (RUL) or time to failure (TTF) (Jardine 2006, Shwabaher 2007). For the failure predition, one effetive way is to develop preditive models to predit failures. Reently, a data-driven preditive modeling attrats muh attention from researhers in mahine learning ommunity. This modeling tehnique is to develop lassifiers using lassifiation algorithms, suh a lassifier is able to lassify the system operation statues into normal (negative) or abnormal (positive). Usually, the positive status is used as an alert for failure preditions. For example, Yang (2005) developed preditive models (binary lassifiers) from the historial operation and maintenane data to predit train wheel failure. The ultimate goal of prognostis is to predit the probability of a failure and preisely estimate its TTF for a monitored omponent/subsystem in omplex systems. This an be done by developing numeri prognosti models, either datadrive models or knowledge-based models. Due to speifiities of prognostis, evaluation of prognosti methods/models is muh more hallenge. Reently, some researhers started to look into the performane evaluation of prognosti methods. Saxena et al (2014, 2009) investigated the evaluation metris for omparing the performane of prognosti methods by fousing on the RUL estimation-based prognostis models. In their work, they proposed to evaluate prognosti methods/algorithms in online and offline running environments, and suggested some metris suh as respond time, predition horizon and auray. Suh metris are useful for omputing regressionbased RUL estimation models. However, it is still diffiult to ompute the performane of preditive models beause those metris are designed for numeri models, not for disrete lassifiation-based models. Our paper addresses this issue by fousing on performane evaluation of preditive models in offline running environment. In International Journal of Prognostis and Health Management, ISSN ,

2 general, the offline environment an provide the ground truth for evaluation. We believe that the evaluation of preditive models an be treated as lassifier evaluation from the point of view of mahine learning (we will use term model and lassifier alternatively later in this paper). It is desirable that suh a model evaluation not only helps pratitioners ompute the performane of a model, but also assists them assess business value of a preditive model. The ideal way to evaluate preditive models is to perform field trials in real-world environments. This is, however, unrealisti and ostly. Existing methods are to ompute the performane of models using generi methods available from the mahine learning ommunity. However, these generi evaluation approahes sometimes fail to take speifiities of PHM appliations into onsideration, and thus may not satisfy the needs of preditive model evaluation for PHM. To evaluate effetively preditive models, the metris have to take the TTF and failure overage into onsideration (detailed in Setion 4). Unfortunately, to our best knowledge, none of the existing generi methods do so. In order to improve generi methods for evaluating prognosti models, domainoriented approahes, whih inorporate the speifi requirements of domain problems into evaluation metris by tailoring existing generi methods, are proposed. Prior to this, generi methods are reviewed, and its defiienies are disussed when applying them to lassifier evaluation for fielded appliations. Then new approahes developed for prognosti appliations are presented. These approahes help developers not only evaluating the model performane, but also identifying the usefulness and business value, whih ould be ahieved one a model is deployed in appliation. The objetive is to emphasize that a preditive model should be arefully evaluated by inorporating PHM speifi requirements into evaluation metris. The paper is strutured to inlude the following setions: Setion 2 reviews generi methods; Setion 3 disusses the limitations of generi methods from the point of view of PHM appliations; Setion 4 presents the approahes developed for evaluating preditive models; Setion 5 presents a ase study for omparing the evaluation for different methods; and the final Setion onludes the paper. 2. OVERVIEW OF GENERIC METHODS 2.1. Preliminaries One of the most important tasks in lassifiation researh is to build lassifiers from a given dataset, n { xk, yk} k 1( yk {1,2,3... m}), where m is the number of lasses and n is a total number of samples (or instanes). The developed lassifier f () lassifies a new data x k as yˆ k f ( x ) ( yˆ {1,2,3 m}). The ultimate goal is to make yˆ k k y k k by inorporating all kinds of notions of osts, errors, and losses into modeling. To evaluate generalization performane of a lassifier f (), a testing dataset, whih is unseen or new data for the lassifier, is used. As a result, a onfusion matrix an be obtained by running the lassifier on the testing dataset. We note this onfusion matrix as C, expressed as Equation 1 C In Equation 1, ij is the number of mislassified instanes, i.e., the number of the instanes lassified as the i th lass whih are really in the j th lass. Predited lasses are represented by the rows of the matrix whereas true lasses are represented by the olumns. An ideal lassifier should ultimately generate a onfusion matrix in whih 0, i j by inorporating noise and ij errors in lassifiation. Using this onfusion matrix, the performane of a lassifier an be omputed. Over the past deades, generi methods have been developed from the onfusion matrix. Generally speaking, generi methods an be grouped into soring methods and graphial methods. In the following subsetions, these generi methods are reviewed Soring methods The soring metris inludes Auray (or Error Rate), True Positive Rate(TPR), False Positive Rate(FNR), True Negative Rate (TNR), False Negative Rate (FNR), Sensitivity, Speifiity, Reall, Preision, and F-Sore. These metris (Japkowiz and Shah 20) provide a simple and effetive way to measure the performane of a lassifier. To simplify the definitions and notations, these metris are illustrated using binary lassifiation, i.e., m = 2, and y {1,1 }. For binary lassifiation, the onfusion k i1 m1 m2 ii 1m 2m matrix in Equation 1 is simplified as a 2x2 matrix shown in Equation 2. C im mm (2) (1) 2

3 where: is the number of true positives; is the number of false positives; is the number of false negatives; and is the number of true negatives. Table 1. The soring metris Metris Computing method From this onfusion matrix, soring metris are omputed as shown in Table 1. These soring metris are widely used in omparing or ranking the performane of lassifiers beause they are simple, easy-to-ompute, and easy-tounderstand. Suh an evaluation typially examines unbiased estimates of preditive auray of different lassifiers (Srinivasan 1999). The assumption is that the estimates of performane are subjet to sampling errors only and that the lassifier with the highest auray would be the best hoie of the lassifiers for a given problem. These metris overlook two important pratial onerns: lass distribution and the ost of mislassifiation. For example, the auray metri assumes that the lass distribution is known in the training dataset and testing dataset and that the mislassifiation osts for false positive and false negative error are equal (Provost 1998). In real-world appliation, the ost of mislassifiation for different errors may not be the same. It is sometimes desirable to minimize the mislassifiation ost rather than the error-rate in lassifiation task. Auray Error Rate TPR TNR FPR FNR Sensitivity 2.3. Graphial methods To overome the limitations of soring metris and inorporate the onsideration of prior lass distributions and the mislassifiation osts, several graphial methods had been developed. These graphi methods are ROC (Reeiver Operating Charateristis) Spae, ROCCH (ROC Convex Hull), Isometris, AUC (Area under ROC urve), Cost Curve, DEA (Data Envelopment Analysis), and Lift urve. They are also useful for visualizing performane of a lassifier. Here is an overview for eah method ROC Spae The ROC analysis was initially developed to express the tradeoffs between hit rate and false alert rate in signal detetion theory (Egan 1975). It is now also used for evaluating lassifier performane (Bradley 1997, Provost and Fawett 2001, 1997). In partiular, ROC is a powerful way for performane evaluation of binary lassifiers, and it has beome a popular method due to its simple graphial representation of overall performane. Speifiity Reall Preision F-sore 2 reall preision reall preision Using TPR and FPR from Table 1 as the Y-axis and X-axis respetively, a ROC spae an be plotted. In this ROC spae, eah point (FPR, TPR) represents a lassifier (Fawett 2003, Flah 2004, 2003). Figure 1 shows an example of a basi ROC graph for five disrete lassifiers. Based on the position of a lassifier in ROC spae, we an evaluate or rank the performane of lassifiers. In ROC spae, the point (0, 1) represents a perfet lassifier whih has 100% auray and zero error rates. The upper left points indiated that a lassifier has a higher TPR and a lower FPR. For instane, C4.5 has a better TPR and a lower FPR than nb. The lassifier on the upper right-hand side of an ROC spae makes positive lassifiation with relative weak evidene and has a higher TPR and a higher FPR. Therefore, the performane of a lassifier is determined by a trade-off between TPR and FPR. However, it is hard to deide whih lassifier is best from ROC spae. For 3

4 Expeted Cost TPR INTERNATIONAL JOURNAL OF PROGNOSTICS AND HEALTH MANAGEMENT example, it is diffiult to tell whih lassifier is better for CN2 and nb in Figure 1 (The unit of FPR/TPR axils is %) ROCCH As mentioned, ROC spae is useful for visualizing the performane of lassifiers regardless of lass distribution and mislassifiation osts. In order to further use ROC spae to selet the optimal lassifiers, a useful way is to generate ROCCH (Provost et al. 2003, Bradley 1997). This ROC onvex hull represents the best performane that an be ahieved by a set of lassifiers. The lassifiers on ROCCH usually ahieve the best auray Figure 1. A ROC graph for 5 lassifiers (Flah 2004) Isometris C4.5 nb Ripper CN2 SVM FPR To further use ROCCH to rank the performane of lassifiers, we an add isometris to ROCCH (Flah 2004, 2003). Isometris are olletions of points with the same value for a given metri (Flah 2003). In ROC spae, isometris are lines. To draw the isometris for a given metri suh as auray, the auray isometris an be added to ROCCH using the approah proposed by Flah (2003). In the same way, other metris isometris suh as preision ould also be added to ROCCH in order to deide the optimal lassifiers based on preision metri and a given lass distribution. denoted as AUC (Bradley 1997, Huang and Ling 2005). AUC an be used to rank or ompare the performane of lassifiers. It is more appropriate whenever the lass distribution and error osts are unknown beause it orresponds to the shape of the entire ROC urve rather than any single point in ROC spae (Swets et al. 2003). AUC has been proven more powerful than Auray in experimental omparisons of several popular learning algorithms (Huang and Ling 2005). To ompute the AUC for binary lassifiers, Hand and Till (2001) proposed a simple formula: AUC where n 0 and n 1 are the number of positive and negative examples respetively, and r i is the rank of the i th positive example in the ranked list for all examples in the test dataset Cost urve n0 i 1 ( r n n 0 i 1 i) The ost urve is a graphial representation for visualizing binary lassifier performane over a full range of possible 0.5 False Positive Rate (3) Figure 2. A ROC urve with 10 points AUC In ROC spae, a ROC urve is plotted for a given lassifier by sampling different lass proportions of positive and negative samples in the training dataset. A ROC urve is a segment line onneting points in ROC spae from left-hand side to right-hand side. Suh a ROC urve illustrates the error tradeoffs. It desribes the preditive behavior of a lassifier independent of lass distribution and error osts, so it deouples generalization performane of a lassifier from these fators. The area under the ROC urve is Probability of Positive Figure 3. Cost lines and their lower envelope 4

5 lass distributions and mislassifiation osts. It transforms a ROC urve or a ROCCH into another two dimensional spae using normalized expeted osts and probabilities instead of the TPR and FPR. In the ost urve, the y-axis is the expeted ost, whih is the normalized value of the total mislassifiation ost, and the x-axis is a probability funtion, whih ombines with the ost of a false positive and a false negative. A point (TPR, FPR) in the ROC spae an be projeted into the ost urve spae as a ost line. By projeting a set of points onto the ROC urve into the ost urve spae, a set of ost lines an be obtained. The lower envelope of these ost lines forms a ost urve. For example, a ROC urve ontaining 10 points for a given lassifier is shown in Figure 2. The orresponding set of ost lines to 10 points in ROC spae is shown in Figure 3 where the lower envelope forms a ost urve for a given lassifier. Further details on ost urve onstrution are desribed in (Drummond and Holte 2004, 2000A) DEA The DEA method is a more effetive way for multi-lass lassifier evaluation. It is widely used in deision-making support and management siene. It addresses the key issues in determining the effiienies of various produers or DMUs (Deision Making Units) by onverting a set of inputs into a set of outputs (Banker et al. 1984, Zheng et al. 2004). DEA is a linear programming based approah, whih onstruts an effiient frontier (envelopment) over the data and omputes eah data point s effiieny related to this frontier. Eah data point orresponds to a DMU or a produer in appliations. Zheng, et al.(2004) has attempted to apply DEA to lassifier evaluation. They have proven that DEA and ROCCH have the same onvex hull for binary lassifiations. In other words, DEA is equivalent to ROCCH for binary lassifiation tasks Lift urve The Lift is often used in marketing analysis. It is also useful for lassifier evaluation. Lift measures how muh better a lassifier is at prediting positive than a baseline lassifier that randomly predits positives (Caruana et al. 2006, 2004, Giudii 2003, Prehelt 1996). It is defined as follows: Left PT % / APT %. Here PT % is % of true positives above the given threshold and APT % is % of samples in dataset above the threshold. For example, an agent intends to send advertising to the potential ustomers but only afford to send ads to 10% of the population and wish these ustomers will response. How the agent hooses these samples from the populations is a lassifiation problem. A lassifier should be trained for prediting positive response ustomers. A lassifier with the maximum Lift will help the agent to get the maximum numbers of the ustomers who will respond to the advertisement in this dataset. 3. ISSUES WITH GENERIC METHODS Generi approahes, either soring methods or graphi methods, have played an important role in lassifier evaluation. However, there are some limitations. As Salzberg (1997) pointed out, good lassifier evaluations are not done nearly enough using datasets from real world problems. For example, in 200 surveyed papers on neural network learning algorithms (Flexer 1996), 29% of these algorithms are not evaluated with any real problem data, and only 8% of these algorithms an be ompared with real problem data. Flexer did another survey on 43 neural network papers from leading journals and found that only 3 out of 43 papers used real problem data to test their algorithms or tune the parameters for the algorithms. It is desirable that a lassifier should be evaluated arefully not only using right metri but also the right data from real world problems, or so-alled fielded appliations. However, existing generi methods may not able to validate the performane of lassifiers for some real-world problems due to some ingrained defiienies. The defiienies with generi evaluation methods an be summarized as follows. First, generi methods request i.i.d. sampling in evaluation. In pratie many data from real-world problems may not meet this requirement of instane independeny. For example, instanes in time-series from prognosti appliations are not independent. They are time dependent. The i.i.d. based random sampling may separate dependent data into different groups suh as training and testing dataset. In fat, the instanes assoiated with a time series should stay in the same group. Seondly, generi metris have some defiienies themselves. The soring metris do not aount the ost of mislassifiation or error rates for evaluation. This is a serious problem beause some errors may ost more than others in different problems. For example, in the medial diagnosti appliation, the false negative for a aner ould ost muh more than a false positive. Reward thresholds for positive preditions over time from replaement Reward t 3 -t 2 Figure 5. A reward funtion for positive preditions -t 1 t 0 TTF 5

6 Thirdly, interpretation of evaluation results from generi metris may be diffiult to tell pratitioners meaningful information. In other words, interpretation of evaluating results is hard to understand, even misleading. For example, AUC, a normalized value between 0 and 1, is supposed to address the defiienies of ROC urve. Theoretially, the higher the AUC value, the better the performane of a lassifier. Therefore, a lassifier with 0.8 of AUC value should be better than one with 0.75 of AUC value. However, suh interpretation may be meaningless or useless for an end user. From the point of view of business value, the interpretation may be totally different. The lassifier with 0.8 of AUC value may save $50,000, and the lassifier with 0.75 of AUC value may save $60,000 after they are deployed in a real-world appliation. Finally, generi methods do not take the speifiities or settings of real-world appliations into onsideration, in partiularly, for prognosti appliations. As an emerging appliation of lassifiation to real-world problems, prognostis (Shwabaher and Goebel 2007, Yang and Létourneau 2005)] is to develop the preditive models (lassifiers) from large-sized operation and maintenane data by using tehniques from mahine learning. Suh a prognosti model is able to predit the likelihood of a failure with a relatively aurate TTF estimation. Existing generi methods failed to inorporate these speifi fators into lassifier evaluation. 4. MODEL EVALUATION FOR PROGNOSTICS As emphasized above, preditive models evaluation needs to take domain speifiities into aount. Suh speifiities over two aspets: apability of failure predition and TTF estimation. From the point of view of TTF, it is desirable that a preditive model an generate alerts in a targeted time window prior to a failure. A model that predits a failure too early leads to non-optimal omponent use. On Alert Failure When p < t T there is a potential for replaing a omponent before the end of its useful life and therefore losing some omponent usage. From the point of view of failure preditions, it is expeted that a preditive model should be able to predit all potential failures with reasonable number of alerts (positive preditions) instead of few failures within many alerts Sore-based approah To address the speifiities of prognosti appliation and evaluate the performane of preditive models effetively, a sore-based approah is proposed by whih the problem overage and TTF is inorporated into lassifier evaluation. The sore-based method is briefly reviewed in the following. In the sore-based method, a reward funtion is first defined for prediting the orret instane outome. The reward for prediting a positive instane is based on time to failure between alert generation and the atual failure, i.e. the determined target window. Figure 5 shows a graph of this funtion. The time target window is formed as [-t 2, -t 1 ]. The parameters t 1 and t 2 are determined based on the requirement of prognosti appliation. The maximum gain is obtained when the model predits the failure in the target window prior to a omponent failure. Outside this target window, prediting a failure an lead to a negative reward threshold. As suh a predition orresponds to misleading advie. Aordingly, false positive preditions (preditions of a failure when there is no failure) are penalized by a reward of -1.5 in omparison to a 1.0 reward for true positive preditions (preditions of failure when there is a failure). The reward funtion aounts for TTF predition for eah alert; to evaluate model overage, alert distribution over the different failure ases must be taken into aount. This is ahieved by the following formula or overall performane evaluation presented in Equation 4. -t T Target Alert TTF NrDeteted Sore NrOfCase sign p i1 s i (4) Figure 4. Time relation between alert time and failure time the other hand, if the failure predition is too lose to the atual failure then it beomes diffiult to shedule an effetive maintenane. Figure 4 shows the requirement for prognosti models in the relationship between alert time and TTF. We use t T and t 0 to denote the beginning and then end of this target window. The time at whih the preditive model predits a failure (and alert is raised) is noted p. where: p is the number of positive preditions in the testing dataset; NrDeteted is the number of failures, whih ontain at least one alert in the target interval; NrofCase is the total number of failures in a given testing dataset; 6

7 p Sign is the sign of s i. When Sign <0 and i1 brdeteted=0, sore is set to zero; and s i is alulated with the reward funtion above for eah alert. From Equation 4, we an find that the total sore for all positive predition is ustomized from auray and the first part (Nrdeteted/NrOfase) is relevant to the reall metri. In terms of proess, the threshold of the reward funtion is determined based on the requirements of the prognosti appliation at hand (rewards, target period for preditions). Then, all models are run using test dataset(s) and their respetive sores are alulated using Equation 4. The model with the highest sore is onsidered the best model for the appliation. This sore-based method has been used to suessfully evaluate the performane of lassifiers for train wheel prognostis (Yang and Létourneau 2005) and airraft omponent prognostis (Létourneau et al 1999) Cost-based approah Although the sore-based approah takes TTF predition and problem detetion overage into aount in evaluating preditive lassifiers, the sores omputed do not inform end users on the expeted ost savings of preditive models. In pratie, the best way is to estimate ost savings that will be ahieved if a preditive lassifier is deployed. To this end, a ost-saving method (Drummond and Yang 2008, Yang and Létourneau 2007) has been developed for preditive lassifier evaluation. Estimating ost saving is a hallenging task. It fully depends on the real ost information from appliations. In partiular the ost may be hanged with hanges of time and deployment environments. Therefore, two different metris for estimating ost savings are proposed: one for aurate ost information, and one for unertain or missed ost information. When aurate ost information is available, a ost-saving metri an be used (Yang and Létourneau 2007) to estimate the business value for end users. By using this metri, four kinds of ost information are requested: the ost of a false alert (an inspetion without omponent replaement), a pro rated ost for early replaement, the ost for fixing a faulty omponent, and the ost of an undeteted failure (i.e., a funtional failure during operation without any prior predition from the prognosti model). The first three osts are generally easy to obtain while the last one is diffiult to approximate aurately. This is beause failures during operations may inur various on other osts that are themselves diffiult to estimate. The following are the details of the ost saving estimation. To estimate the ost saving (CS) for a model, the differene between the ost of operation without the model ( C nm ) and the ost with the prognosti model ( C ) are omputed using Equations 5 and 6: C nm C pm pm ( N M) d ( M N) (5) a T b F N d ( M N) (6) e where: a is a pro rata ost for early replaement. For example, $10 for eah lost day of usage; b is the ost for a false alert; is the ost for an undeteted failure (diret ost for a failure during operation); d is the ost for replaing the omponent (either after a failure or following an alert); N is the number of undeteted failures; M is the number of deteted failures; T e is the sum of p t T for all predited M failures, i.e., Te pi tt (see Figure 3) i1 where p i is the time of the i th predition; and F is the number of false alerts The ost parameters, a, b, and d are provided by the end user, T e, F, M, and N are omputed after applying the given model to a testing dataset. 5. CASE STUDY The WILDMiner projet targets the development of datamining-based models for train wheel failures preditions (Yang and Létourneau 2005). The objetive is to redue train wheel failures during operation whih disrupt operation and ould lead to atastrophes suh as train derailments. The data used to build the preditive models ome from the WILD (Wheel Impat Load Detetors) data aquisition system. This system measures the dynami impat of eah wheel at strategi loations on the rail network. When the measured impat exeeds a predetermined threshold, the wheels on the orresponding axle are onsidered faulty. A train with faulty wheels needs to immediately redue speed and then stop at the nearest siding so that the ar with faulty wheels an be deoupled and repaired. A suessful preditive model would be able to predit high impats ahead of time so that problemati wheels are replaed before they disrupt operation. For this study, we used WILD data olleted over a period of 17 months from a fleet of 804 large ars with axles eah. After data pre-proessing, we ended up with a dataset ontaining 2,409,696 instanes grouped in 9906 time-series 7

8 (one time-series for eah axle used in operation during the study). We used 6400 time-series for training (roughly the equivalent of the first months) and kept the remaining 3506 time-series for testing (roughly the equivalent of the last 6 months). Sine there are only 9 ourrenes of wheel failures in the training dataset, we seleted the orresponding 9 time-series out of the initial 6400 timeseries in the training dataset. We reated a relevant dataset for modeling whih ontains 4364 instanes from the seleted 9 time-series. Using the obtained dataset and the WEKA pakage, we built the preditive models. The modeling proess onsists of three steps. First, we labeled all instanes in the dataset by onfiguring the automated labeling step. Seond, we augmented the representation with new features suh as the moving average for a key attribute. Finally, we used WEKA s implementation of deision trees (J48) and naïve Bayes (SimpleNaiveBayes) to build four preditive models referred to as A, B, C and D. Model A and B were obtained with default parameters from J48 and SimpleNaiveBayes, respetively. In a seond experiment, we modified the mislassifiation osts to ompensate for the imbalaned between positive and negative instanes and then re-run J48 and SimpleNaiveBayes to generate model C and D. For evaluation purpose, we ran these four models on the test dataset. The test dataset has 1,609,5 instanes. Out of the 3506 time-series ontained in the test dataset, 81 omprise a validated wheel failure. We applied the proposed evaluation method to estimate the ost saving from these four models. Sine we assumed that the operators will at as soon as they reeive an alert, we only kept the first alert (predition of failure) from eah time-series. We then extrated the performane parameters N, M, and F by ounting the number of time-series for whih we did not get any preditions, the number of series for whih we did get a predition followed by an atual failure (true positive), and the number of time-series for whih we got a false alert, respetively. To ompute, we added the differenes between the time of the predition and the beginning of the target window (i.e., p t T ) for eah of the M time-series for whih the model orretly predited a failure. In this appliation, the time unit for the differene p t T is day and t T is set as 20 days prior to failure. For example, when the time-to-failure predition (p) of an alert is 40 days to an atual failure, its differene, p t, is 20 days ( T p t = -40 (- 20) = 20). T The ost information (noted a, b,, and d in Equations (5) and (6) was provided by an independent expert in the railway industry. These are as follows: a= $2/per day for loss of usage, b=$500/per false alert, =$5000/per undeteted failure and d=$00/per omponent replaement. All osts are in US dollars. Using these values and the results for the performane parameters, we obtained the results shown in Table 2. Table 2. The results of 4 preditive models on test data T e Model Name M+N N F Cost Saving (US$) Model A ,750.0 Model B ,376.0 Model C ,206.0 Model D ,3.0 To ompare the evaluation results from different metris, we also ompute auray and false alert rate by using generi methods and the sores by using sore-based approah by running models on the same test dataset as we omputed ost-saving for Table 2. The results are shown in Table 3. Table 3. The evaluation results from different methods Cost Saving Model Name Auray AUC Sore (US$) Model A 75% ,750.0 Model B 89% ,376.0 Model C 93% ,206.0 Model D 86% ,3.0 The ase study illustrates the simpliity and usefulness of the ost-based evaluation approah. With this approah, the end user gets a quik understanding of the potential ost savings to be expeted from eah of the model. Key fators suh as timeliness of the alerts and overage of failures are taken into aount, whih is not the ase with other methods typially used in data mining researh. As shown in Table 2, it is obvious that Model C outperformed other models in terms of the ost saving. From the evaluation results shown in Table 3, it is interesting to note that, for the ase study onsidered, the results obtained from the ast-based method is onsistent with the sore-based approah. In terms of the AUC and Auray results, Model B should be better than Model D. However, the ost-based approah and sore-based method 8

9 show different results. These results support our argument that preditive models deserve speifi evaluation methods for its performane evaluation. 6. CONCLUSIONS In this paper generi methods whih are widely used in lassifier evaluation are first surveyed. The limitations for existing lassifier evaluation, partiularly for preditive models evaluation have also been disussed. To address the issues two useful and effetively methods for prognosti model evaluation have been proposed whih aount for prognosti appliation speifiities. These methods have been used to evaluate the preditive models for several PHM appliations suh as train wheel prognostis and airraft omponent prognostis. In terms of the results obtained from the ast study, it is obvious that the proposed domain-oriented evaluation methods for preditive models are useful and able to provide a diret business value for pratitioners. We believed that more and more feasible and effetive domain-oriented approahes will be developed to meet the needs of preditive model evaluation for real-world appliations. This is a right way to promote mahine learning tehniques in solving real-world problems. Domain-oriented approahes may inorporate speifiities of domain problems into the performane assessments; therefore, they are helpful and useful in evaluating lassifier for appliations. At the same time, they are also welomed by end users beause of the feasibility to identify the business values suh as ost savings. We strongly argue that a preditive model has to be arefully evaluated not only using generi methods but also domain-oriented approahes before deploying it in PHM appliations. Generi methods ould help developers in investigating overall performane of a model from the statistial viewpoint at the initial stage of model development. Domain-oriented approahes should be further used to evaluate the usefulness and business value. ACKNOWLEDGEMENT This work is supported in part by the National Natural Siene Foundation of China (Grant Nos ). REFERENCES Banker R, Chanes A, Cooper W, et al. (1984). Some Models for Estimating Tehnial and Sale Ineffiienies in Data Envelopment Analysis, Management Siene, Vol. 30 No. 9, Bradley A (1997). The Use of the Area under the ROC Curve in the Evaluation of Mahine Learning Algorithms, Pattern Reognition, Vol. 30, Caruana R and Niulesu-Mizil A (2006). An Empirial Comparison of Supervised Learning Algorithms. Proeedings of the 23rd International Conferene on Mahine Learning (ICML2006) Caruana R and Niulesu-Mizil A (2004). Data Mining in Metri Spae: An empirial Analysis of Supervised Learning Performane Criteria, Proeedings of the 10th ACM SIGKDD International Conferene on Knowledge Disovery and Data Mining (KDD2004), Seattle, Washington, USA, Drummond C and Yang C (2008). Reverse-Engineering Costs: How muh will a Prognosti Algorithm save?, Proeedings of the 1st International Conferene on Prognostis and Health Management. Denver, USA Drummond C and Holte R (2004). What ROC Curve Can t Do (and Cost Curve Can), ECAI Workshop on ROC Analysis in Artifiial Intelligene Drummond C and Holte R (2000). Expliitly Representing Expeted Cost: An Alternative to ROC Representation, Proeedings of the 6th ACM SIGKDD International Conferene on Knowledge Disovery and Data Mining (KDD2000), New York, USA, Egan J (1975). Signal Detetion Theory and ROC Analysis, New York Aademi Press Fawett T (2003). ROC Graphs: Notes and Pratial Considerations for Data Mining Researhers, Tehnial report, Intelligent Enterprise Tehnologies Laboratory, HP Flah P, (2004). The Many Faes of ROC Analysis in Mahine Learning, ICML 04 tutorial, Flah P, (2003). The Geometry of ROC Spae: Understanding Mahine Learning Metris through ROC Isometris. Proeedings of the 20th International Conferene on Mahine Learning (ICML03), Washington, DC, USA Flexer A (1996). Statistial Evaluation of Neural Network Experiments: Minimum Requirement and Current Pratie. Proeedings of the 13rd European Meeting on Cybernetis and Systems Researh. Austrian Soiety for Cyberneti Studies, Giudii P (2003). Applied Data Mining, John Wiley and Sons, New York Hand D and Till R (2001). A Simple Generalization of the Area Under the ROC Curve for Multiple Class Classifiation Problems, Mahine Learning, Vol.45, Huang J and Ling C (2005). Using AUC and Auray in Evaluating Learning Algorithms, IEEE Transations on Knowledge and Data Engineering, Vol. 17, No. 3, Japkowiz N. and Shah Mohak (20). Evaluation of Learning Algorithms, A Classifiation Perspetive, Cambridge University Press Jardine A., Lin D., and Banjevi D. (2006). A review on mahinery diagnostis and prognostis implementing ondition-based maintenane. Mehanial Systems and Signal Proessing Vol.20,

10 Létourneau S, Famili F, et al. (1999). Data Mining for Predition of Airraft Component Replaement, IEEE Intelligent Systems Journal, Speial Issue on Data Mining Prehelt L (1996). A Quantative Study of Experimental Evaluations of Neural Network Algorithms: Current Researh Pratie, Neural Networks, Vol. 9 Provost F, Fawett T and Kohavi R (1998). The ase Against Auray Estimation for Comparing Indution Algorithms, Proeedings of the 15th International Conferene on Mahine Learning, Provost F and Fawett T (2001). Robust Classifiation for Impreise Environment, Mahine Learning, Vol.42, No.3, Provost F and Fawett T (1997). Analysis and Visualization of Classifier Performane: Combination under Impreise Class and Cost Distributions, Proeedings of International Conferene on Knowledge Disovery and Data Mining (KDD1997) Provost F, Fawett T, and Kohavi R (1998). The ase Against Auray Estimation for Comparing Indution Algorithms, Proeedings of the 15th International Conferene on Mahine Learning, Salzberg S (1997). On Comparing Classifiers: Pitfalls to Avoid and a Reommended Approah. Data Mining & Knowledge Disovery, Vol. 1, Saxena A, Sankararaman S, and Goebel K,, (2014). Performane Evaluation for Fleet-based and Unit-based Prognosti Methods, European Conferene of the Prognostis and Health Management Soiety Saxena A, Celaya J., Saha B, Saha S, and Goebel K. (2009). On Applying the Prognosti Performane Metris, Proeedings of International Conferene on Prognostis and Health Management Shwabaher M and Goebel K (2007). A Survey of Artifiial Intelligene for Prognostis, The 2007 AAAI Fall Symposium, Arlington, Virginal, USA Srinivasan A (1999). Note on the loation of optimal lassifiers in n-dimensional ROC spae. Tehnial Report PRG-TR-2-99, Oxford University Computing Laboratory Swets J, Dwaes R, et al. (2000). Psyhologial Siene Can Improve Diagnosti deisions, Psyhologial Siene in the Publi Interest, Vol. 1, 1-26 Yang C and Létourneau S (2007). Model Evaluation for Prognostis: Estimating Cost Saving for the End Users, The Proeedings of the 6th International Conferene on Mahine Learning and Appliations (ICMLA 2007), Cininnati, OH, USA Yang C and Létourneau S (2005). Learning to Predit Train Wheel Failures, Proeedings of the th ACM SIGKDD International Conferene on Knowledge Disovery and Data Mining (KDD2005), Zheng Z, Padmanabhan B, et al. (2004). A DEA Approah for Model Combination, Proeedings of the 10th ACM SIGKDD International Conferene on Knowledge Disovery and Data Mining (KDD2004), Seattle, Washington, USA, BIOGRAPHIES Chunsheng Yang is a Senior Researh Offier at the National Researh Counil Canada. He is interested in data mining, mahine learning, prognosti health management(phm), reasoning tehnologies suh as asebased reasoning, rule-based reasoning and hybrid reasoning, multi-agent systems, and distributed omputing. He reeived an Hons. B.S. in Eletroni Engineering from Harbin Engineering University, China, an M.S. in omputer siene from Shanghai Jiao Tong University, China, and a Ph.D. from National Hiroshima University, Japan. He worked with Fujitsu In., Japan, as a Senior Engineer and engaged on the development of ATM Network Management Systems. He was an Assistant Professor at Shanghai Jiao Tong University from 1986 to 1990 working on Hyperube Distributed Computer Systems. He was a Program Co-Chair for the 17th International Conferene on Industry and Engineering Appliations of Artifiial Intelligene and Expert Systems. Dr. Yang is a guest editor for the International Journal of Applied Intelligene. He has served Program Committees for many onferenes and institutions, and has been a reviewer for many onferenes, journals, and organizations, inluding Applied Intelligene, NSERC, IEEE Trans., ACM KDD, PAKDD, AAMAS, IEA/AIE. Dr. Yang is a Senior IEEE member. Yanni Zou is an Assistant Professor with Jiujiang University, Jiangxi, China. She is also a Ph.D. Candidate at Shool of Information Tehnology, Nanhang University. She is interested in image proessing, proess ontrol, robotis, intelligent systems, and mahine learning. She reeived an Hons. B.S. in Eletroni Engineering from Nanhang University China, an M.S. in System Engineering, Anhui University of Siene and Tehnology, Anhui, China. Jie Liu is urrently an Assistant Professor in the Department of Mehanial & Aerospae Engineering at Carleton University, Ottawa, Canada. He reeived his B.Eng. in Eletronis and Preision Engineering from Tianjin University, China, his M.S. in Control Engineering from Lakehead University, Canada, and his Ph.D. in Mehanial Engineering from the University of Waterloo, Canada. He is leading researh efforts in the areas of prognostis and health management, intelligent mehatroni systems, power generation and storage. He has published over 25 papers in peer-reviewed journals. He is also a registered professional engineer in Ontario, Canada. Kyle R Mulligan obtained his Bahelor s degree in Systems and Computer Engineering at Carleton University in Ottawa in He then ontinued to obtain his Master s degree in Biomedial Engineering in 2009 at Carleton 10

11 University. In 2014 he ompleted a Dotorate degree in Mehanial Engineering at the Université de Sherbrooke with a speialization in data-driven prognostis for assessing strutural failures. He is an ative volunteer at loal high shools to promote the awareness and importane of Engineering in today s soiety. His researh interests inlude: medial imaging tehniques, biomehanis, nanorobotis, and strutural/prognosti health management.