Internatonal Journal of Producton Research, Volume 46, Issue 18, 008, pp 5075-5095 A robust kernel-dstance multvarate control chart usng support vector prncples F. Camc, R. B. Chnnam *, and R. D. Ells Impact Technologes, LLC, 00 Canal Vew Boulevard, Rochester, NY 1463 Department of Industral & Manufacturng Engneerng, Wayne State Unversty, Detrot, MI 4801, USA * Correspondng Author: Tel: +313-577-4846; Fax: +313-577-8833; E-mal: r_chnnam@wayne.edu
Abstract It s mportant to montor manufacturng processes n order to mprove product qualty and reduce producton cost. Statstcal Process Control (SPC) s the most commonly used method for process montorng, n partcular, makng dstnctons between varatons attrbuted to normal process varablty to those caused by specal causes. Most SPC and multvarate SPC (MSPC) methods are parametrc n that they make assumptons about the dstrbutonal propertes and auto-correlaton structure of n-control process parameters, and f satsfed, are effectve n managng false alarms/postves and false negatves. However, when processes do not satsfy these assumptons, the effectveness of SPC methods s compromsed. Several non-parametrc control charts based on sequental ranks of data depth measures have been proposed n the lterature, but ther development and mplementaton have been rather slow n ndustral process control. Several non-parametrc control charts based on machne learnng prncples have also been proposed n the lterature to overcome some of these lmtatons. However, unlke conventonal SPC methods, these non-parametrc methods requre event data from each out-ofcontrol process state for effectve model buldng. Ths paper presents a new non-parametrc multvarate control chart based on kernel-dstance that overcomes these lmtatons by employng the noton of one-class classfcaton based on support vector prncples. The chart s non-parametrc n that t makes no assumptons regardng the data probablty densty and only requres normal or n-control data for effectve representaton of an n-control process. It does however make an explct provson to ncorporate any avalable data from out-of-control process states. Expermental evaluaton on a varety of benchmarkng datasets suggests that the proposed chart s effectve for process montorng. Keywords: Control chart; Support vector machnes; Kernel-dstance
I. Introducton In order to mprove product qualty and reduce producton cost, t s necessary to detect equpment malfunctons, falures, or other specal events as early as possble. For example, accordng to the survey conducted by (Nmmo 1995; Chen, Kruger et al. 004), the US-based petrochemcal ndustry could save up to $10 bllon annually f abnormal process behavor could be detected, dagnosed and approprately dealt wth. By montorng the performance of a process over tme, statstcal process control (SPC) attempts to dstngush process varaton attrbuted to common causes from varaton attrbuted to specal causes, and hence, forms a bass for process montorng and equpment malfuncton detecton (Martn, Morrs et al. 1996). It s also the most commonly used tool to analyze and montor processes (Eckelmann and Anant 003). Most SPC methods are parametrc n that they make assumptons about the dstrbutonal propertes and auto-correlaton structure of n-control process parameters, and f satsfed, are very effectve n managng false alarms/postves and false negatves. Among others, they offer the followng statstcal and practcal advantages: 1) only data from an n-control state s necessary to ntalze the control chart, ) data from a relatvely lmted number of sub-groups (say 0 to 30) s adequate for accurate ntalzaton, and 3) tradeoffs between false alarms/postves (Type-I error) and false negatves (Type-II errors) can be managed (by changng the wdth of the control lmts and the sub-group sample sze, respectvely). Whle conventonal SPC charts (such as the Shewhart type control charts (Shewhart 196), the cumulatve sum (CUSUM) control charts, and the exponentally weghted famly control charts) are developed for unvarate processes (Manabu, Shnj et al. 004), multvarate statstcal process control (MSPC) s employed to montor processes that have correlated mult-varables (Montgomery 001). One type of MSPC s multvarate charts extended from unvarate SPC methods, ncludng Hotellng s T chart (Hotellng 1947), multvarate EWMA (Lowry, Woodall et al. 199); (Runger and Prabhu 1996), (Testk and Borror 004) and multvarate CUSUM charts (Nga and 3
Zhang 001); (Runger and Testk 004). Another type of MSPC s based on latent varable projecton, such as Prncpal Component Analyss (PCA) and Partal Least Squares (PLS) (MacGregor and Kourt 1995) (Rach and Cnar 1996) (Yoon and MacGregor 004). An alternatve approach to account for the dynamc aspects of the data n MSPC s to use MRA (mult-resoluton analyss) (Baksh 1998; Teppola and Mnkknen 000). For a good revew of MSPC charts, see (Lowry and Montgomery 1995). Among these, Hotellng s T chart s wdely used n practce (Sun and Tsung 003). The standard assumpton behnd majorty of SPC and MSPC methods mentoned above s that the process varables follow a Gaussan dstrbuton (Rose 1991), a questonable assumpton n several ndustral processes (Polansky 001), and n partcular, hghly automated processes (Chnnam and Kolark 199). (Schllng and Nelson 1976) and many other researchers have nvestgated the effects of non-normalty on the control lmts and chartng performance. To allevate such effects, some dstrbuton-free or non-parametrc control charts have been proposed based on sequental ranks of data depth measures (Lu and Sngh 1993; Aradhye, Baksh et al. 001; Stoumbos and Reynolds 001; Chakrabort, Van der Laan et al. 003; Messaoud, Wehs et al. 004), but ther development and mplementaton have been rather slow n ndustral process control (Chakrabort, Van der Laan et al. 001). Several non-parametrc control charts based on machne learnng and pattern recognton prncples have also been proposed n the lterature. For example, (Cook and Chu 1998) proposed radal bass functon (RBF) networks to recognze shfts n correlated manufacturng processes, (Chnnam 00) proposed support vector machnes (SVMs) for recognzng shfts n correlated and other manufacturng processes, and (Smth 1994) and (Pugh 1991) consdered mult-layer perceptron (MLP) networks for mplementng Shewhart type control charts, all relaxng the Gaussan assumpton. Whle these methods have shown success n relaxng the Gaussan assumpton, the fundamental lmtaton wth these and most other machne learnng methods proposed n the lterature for process control s that they cast the problem as that of 4
classfcaton or pattern recognton, and hence, strctly requre example data from all out-ofcontrol states of nterest. Ths s a crtcal lmtaton for obtanng example cases from all such states mght be dffcult, expensve, or even mpossble. The second lmtaton s that they make no explct provson to make tradeoffs between Type-I errors (false alarms) and Type-II errors (nablty to detect shfts n process condton). Most of these machne learnng methods also necesstate modelng and tranng for each specfc falure type. A model that s developed for a specfc type abnormal event (out-of-control state) cannot necessarly gve good classfcaton accuracy for another type of abnormal event. Ths paper presents a new non-parametrc kernel-dstance control chart that employs the noton of one-class classfcaton or novelty detecton to overcome these lmtatons and adopts support vector machne (SVM) prncples for dong so. There are several reasons for basng the proposed control chart on SVM prncples: 1) They are a system for effcently tranng lnear learnng machnes n the kernel-nduced feature space, ) they successfully control the flexblty of kernel-nduced feature space through generalzaton theory, and 3) they explot exstng optmzaton theory n dong so. An mportant feature of SVM systems s that, whle enforcng the learnng bases suggested by generalzaton theory, they also produce sparse dual representatons of the hypothess, resultng n extremely effcent algorthms (Crstann and Taylor 000). Ths s due to the Karush-Kuhn-Tucker condtons (Kuhn and Tucker 1951) (Mangasaran 1994), whch hold for the soluton and play a crucal role n the practcal mplementaton and analyss of these machnes. Another mportant feature of the support vector approach s that due to Mercer s condtons on the kernels (Mercer 1909), the correspondng optmzaton problems are convex and hence have no local mnma. Ths fact, and the reduced number of non-zero parameters, marks a clear dstncton between these systems and other machne learnng algorthms, such as neural networks (Crstann and Taylor 000). The end result s that the proposed kernel-dstance control chart s non-parametrc, only requres data from an n-control process state, makes provson to utlze any avalable data from out-of-control 5
states, and allows some tradeoff between Type-I and Type-II errors. The proposed kernel-dstance control chart supports both unvarate and multvarate processes and can montor both process locaton and dsperson aspects through a sngle control chart. A notable excepton n the lterature that also offers several smlar features s another kerneldstance control chart (called k-chart) ndependently proposed by (Sun and Tsung 003), also based on a one-class classfer, Support Vector Data Descrpton (SVDD), developed by Davd Tax (Tax 001; Tax and Dun 004). However, a sgnfcant lmtaton wth SVDD, and hence the k-chart by Sun and Tsung, s that t lacks any ablty to make a good dstncton between outlers and normal data wthn the tranng set. In addton, they make no provson to utlze any avalable data from out-of-control states, and lastly, they offer no structured method for makng tradeoffs between Type-I and Type-II errors. Ths can also result n poor representaton of the ncontrol process state f the data avalable for ntalzng the control chart s not pre-processed for elmnaton of outlers. Our proposed kernel-dstance control chart overcomes these lmtatons by ntegratng the SVDD method wth prncples borrowed from Support Vector Representaton and Dscrmnaton Machne (SVRDM) by Yuan and Casasent (003). Many of these postve attrbutes of our proposed kernel dstance control chart are llustrated n Fgure 1. Here, we partcularly work wth the same multvarate process control dataset employed by (Sun and Tsung 003). Fgure 1(a) llustrates a plot of normal process data projected onto the space of the two most domnant prncpal components (dentcal to Fgures 11-1 of (Sun and Tsung 003)), along wth a known outler pont (from Fgure 13 of (Sun and Tsung 003)). If one were to use the data from Fgure 1(a) to ntalze the k-chart, there s no guarantee that the outler pont wll be recognzed to be an outler (see for example Fgure 9 of (Sun and Tsung 003)). On the contrary, the kernel dstance control chart proposed here gves us several optons. Supposng that the outler data pont s not so labeled and presented for chart ntalzaton, the proposed method recognzes the pont to be an outler (as shown n Fgure 1(a)). If the outler pont s labeled pror to chart ntalzaton as an outler, the proposed method recognzes ths label and puts t outsde 6
the normal boundary (as shown n Fgure 1(b)). If for some reason, one chooses to treat ths outler pont as a normal data pont, the proposed method wll accept ths constrant and treats the pont as normal and defnes normal process boundary wth the pont as a boundary pont (as s evdent from Fgure 1(c)). Thus, the proposed kernel dstance control chart s very robust and can take advantage of any avalable data/knowledge regardng process faults. In addton, our proposed method also offers an effectve heurstc for optmzng the kernel parameters, somethng mssng from SVDD and the k-chart from Sun and Tsung. For all these reasons, we label the proposed method the robust kernel-dstance control chart or rk-chart n short. (a) (b) (c) Fgure 1. Flexblty of rk-chart. In Panel a) the pont well outsde the normal data s unlabeled but s recognzed and declared to be an outler by rk-chart. In panel (b) the pont s labeled as an outler, and the rk-chart agan declares t an outler makng use of the labelng. The results are dentcal to those wthout labelng. Fnally, n panel c) the pont s labeled as normal, and rk-chart accepts ths constrant and treats the pont as normal and defnes normal process boundary wth the pont as a boundary pont. In all these cases, the rk-chart was tuned wth typcal parameters (specfed later). 7
The rest of ths paper s organzed as follows: Secton II provdes background nformaton on support vector machnes, the theory behnd rk-chart s presented n Secton III, expermental results n Secton IV, and concludng remarks n Secton V. II. Support Vector Machnes Ths secton provdes a bref background on support vector machnes (SVMs) rooted n Statstcal Learnng Theory, a noton frst ntroduced by (Vapnk 1998). We frst explan the bascs of SVMs for bnary classfcaton and then dscuss how the technque can be extended to deal wth the problem of one-class classfcaton for developng rk-chart. II.1. Bnary classfcaton Lnear case SVMs belong to the class of maxmum margn classfers. They perform pattern recognton between two classes by fndng a decson surface that has maxmum dstance to the closest ponts n the tranng set, whch are termed support vectors. We start wth a tranng set of ponts d x, 1,..., N where each pont x belongs to one of two classes dentfed by the label y 1, 1 and d s the dmensonalty of the ponts. Let the tranng set be denoted by {( x, )} N y 1. Assumng lnearly separable data, the goal of maxmum margn classfcaton s to separate the two classes by a hyperplane such that the dstance to the support vectors s maxmzed. Ths s acheved by mnmzng w subject to the constrant y ( x w b) 1 0, where w s normal to the hyper-plane. Fgure llustrates these concepts for the separable case. In order to provde for non-separable cases, the formulaton s modfed as follows: Mnmze w C (1) Subject to: y b 1 0 x w () Ths quadratc optmzaton problem can be solved effcently usng the followng Lagrangan dual formulaton: 8
Maxmze 1 y y x x (3) j j j, j where Subject to: 0 C and. y 0 (4) denote the Lagrange multplers. The Lagrangan formulaton of the problem offers the advantage of havng Lagrange multplers n the constrants and tranng data n the form of nner products between data vectors (Müller, Mka et al. 001). In the soluton, non-zero values represent support vectors that are on the separatng hyper-plane (satsfyng the equaton w x b 0 ). w Orgn -b w II.. Bnary classfcaton Nonlnear case Support Vectors Seperatng Hyperplane Margn / w Fgure : Separaton of classes by hyper-plane. In many cases, classes are not lnearly separable. In order to learn non-lnear relatons wth a lnear machne, we need to select a set of non-lnear features and to rewrte the data n the new representaton. Ths s equvalent to applyng a fxed non-lnear mappng of the data to a feature space, n whch the lnear machne can be used. Thus, the non-lnear separable case could be handled n two steps: frst a fxed non-lnear mappng transforms the data nto a feature space F, and then a lnear machne s used to classfy them n the feature space. One mportant property of lnear learnng machne s that t can be expressed n a dual representaton. That s, the hypothess can be expressed as a lnear combnaton of the tranng ponts so that the decson rule can be evaluated usng just nner products between the test pont and the tranng ponts. If one could 9
compute the nner product ( x ) ( x) n feature space drectly as a functon of the orgnal nput ponts, t becomes possble to merge the two steps needed to buld a non-lnear learnng machne. We call such a drect computaton method a kernel functon. A kernel s a functon K, such that for all x, z X K( x, z) ( x) ( z ) (5) where s a mappng from X to an nner product feature space F. By replacng the nner product wth an approprately chosen kernel functon, one can mplctly perform a non-lnear mappng to a hgh dmensonal feature space wthout ncreasng the number of tunable parameters, provded the kernel computes the nner product of the feature vectors correspondng to the two nputs. Thus, the use of kernels makes t possble to map the data mplctly nto a feature space and to tran a lnear machne n such a space, potentally sdesteppng the computatonal problems nherent n evaluatng the feature map. In practce, the approach taken s to defne a kernel functon drectly, hence mplctly defnng the feature space. In ths way, we avod the feature space not only n the computaton of nner products, but also n the desgn of the learnng machne tself. Mercer s theorem (Mercer 1909) provdes a characterzaton of when a functon K( x, z) s a kernel (. e. what propertes are necessary to ensure that t s a kernel for some feature space). An mportant famly of kernel functons s the polynomal kernel: K( x, z) (1 x z ) d (6) where d s the degree of the polynomal. In ths case the components of the mappng are all the possble monomals of nput components up to the degree d. An even more popular kernel n the lterature s the Gaussan (Tax and Dun 1999; Yuan and Casasent 003): xz K( x, z ) e (7) 10
For a good dscusson on makng kernels, see (Crstann and Taylor 000). For more detaled nformaton on the broader topc of support vector machnes see (Müller, Mka et al. 001). III. Robust Kernel-Dstance Control Chart (rk-chart) III.1. rk-chart as a one-class classfer As stated earler, the proposed robust kernel-dstance control chart (rk-chart) employs the noton of one-class classfcaton for process montorng, and n dong so, models the boundary of process data from an n-control state and declares the process to be n control or out of control dependng on where the new observaton les wth respect to the boundary that exsts n the feature space. In the statstcal process control lterature, the two most mportant measures that are of partcular nterest n the context of process montorng are measures of process central tendency and process dsperson. As orgnally proposed by W. Shewhart (Shewhart 196), the sample arthmetc mean s the most employed measure for central tendency (among other measures such as mode and medan). Whle the smplest statstcal parameter that descrbes varablty n observed data s the sample range, the sample standard devaton s a better estmate of varablty for t consders every observaton. Not unlke many multvarate statstcal process control (MSPC) methods, the proposed rk-chart jontly models these measures of central tendency as well as dsperson n a mult-dmensonal space. Whle we recommend the sample mean and standard devaton as statstcal measures for process montorng (resultng n a - dmensonal space), the proposed rk-chart method s a general method and can ncorporate any other type of locaton and dsperson measures, promsng applcaton specfc measures, as well as other measures such as hgher-order statstcal moments (e. g., skewness and kurtoss). The d only requrement for rk-chart s that the vector of sample statstcal measures s real, x, where d denotes the dmensonalty of the vector. However, note that the number of support 11
vectors (as well as the sze of the tranng set) necessary for adequate representaton of normal or n-control process state wll ncrease as a functon of d. In the case of multvarate processes, the feature space could smply be the process varable space or a transformaton thereof (such as the space of domnant prncpal components). Whle t s also possble to ntroduce features from the co-varance matrx of the subgroup sample to montor process dsperson, care should be exercsed to manage d. Gven that rk-chart models the process data n two or hgher dmensonal spaces, n the case of unvarate processes, t s necessary that the subgroup sample sze be greater than one to facltate extracton of at least two features (such as mean and standard devaton). Whle rk-chart can n theory deal wth both unvarate and multvarate processes, and jontly montors both locaton and dsperson measures, t does not necessarly have the ablty to dentfy the type of process fault (such as a process locaton shft or process varance shft). Thus, n developng the proposed control chart, we devate from classcal SVM desgned for bnary classfcaton to representaton of boundary from a sngle class (. e. d x ). rk-chart s partcularly nspred from Support Vector Data Descrpton (SVDD) (Tax and Dun 1999) and Support Vector Representaton Machne (SVRM) (Yuan and Casasent 003) and gves the mnmum volume closed sphercal boundary around the n-control process data, represented by center c and radus r. Mnmzaton of the volume s acheved by mnmzng r, whch represents structural error (Müller, Mka et al. 001): Mn r (8) Subject to: x c r, x : th data pont (9) The formulaton above does not allow any data to fall outsde of the sphere. In order to make provson wthn the model for potental outlers wthn the tranng set, a penalty cost functon s ntroduced as follows (for data that le outsde of the sphere): Mn r C (10) 1
Subject to: x c r, 0 (11) where C s the coeffcent of penalty for each outler (also referred to as the regularzaton parameter) and s the dstance between the th data pont and the hyper-sphere. Once agan, ths s a quadratc optmzaton problem and can be solved effcently by ntroducng Lagrange multplers for constrants (Vapnk 1998): L( r, c, ξ, α, γ) r C r ( x x c x cc) (1) where and are Lagrange multplers, 0, 0, and x x s nner product of Note that for each tranng data pont x, a correspondng and x and x. are defned. L s mnmzed wth respect to r, c, and ξ, and maxmzed wth respect to α and γ. Takng the dervatves of (1) wth respect to r, c, ξ, and equatng them to zero, we obtan the followng constrants: c x (13) Gven that 0, 0, constrant (14) can be rewrtten as: C 0 (14) 1 (15) 0 C (16) The followng quadratc programmng equatons can be obtaned by substtutng (13), (14), (15), and (16) n (1). Max ( x x ) ( x x ) (17) j j, j Subject to: 0 C, 1 (18) Standard algorthms exst for solvng ths problem (Tax 001). The above Lagrange formulaton also allows further nterpretaton of the values of α. If necessary, the Lagrange 13
multplers (, ) wll take a value of zero n order to make the correspondng constrant term zero n (1). Thus, the rk-chart formulaton satsfes the Karush-Kuhn-Tucker (KKT) condtons for achevng a global optmal soluton. Notng that C, f one of the multplers becomes zero, the other takes on a value of C. When a data pont x s nsde the sphere, the correspondng wll be equal to zero. If t s outsde of the sphere,.e. 0, wll be zero resultng n to be C. When the data pont s at the boundary, and wll be between zero and C to satsfy (15). The quadratc programmng soluton often yelds a 'few' data ponts wth a non-zero value, or support vectors. What s of partcular nterest s that support vectors can SV effectvely represent the data whle remanng sparse. Let S { x : 0} denote the set of support vectors. In general, t s hghly unlkely that a hyper-sphere can offer a good representaton for the boundary of n-control process data n the orgnal nput space. Hence, data ought to be transformed to a hgher dmensonal feature space where t can be effectvely represented usng a hyper-sphere. Not unlke SVMs, rk-chart also employs kernels to acheve ths transformaton wthout compromsng computatonal complexty. Thus, the dot product n (17) s replaced by a Kernel functon, leadng us once agan to the followng quadratc programmng problem: Max K( x, x ) K( x, x ) (19) j j, j III.. Gaussan kernel optmzaton wthn rk-chart Subject to 0 C, 1 (0) The proposed robust kernel-dstance control chart employs the one-class classfcaton formulaton from above along wth a Gaussan kernel. The Gaussan kernel has been shown to partcularly offer better performance over other kernels for one-class classfcaton problems (see (Tax 001) for more dscusson on ths) and hence the motvaton for usng t. The ssue s 14
optmzaton of the scale parameter of (7). Whle could be specfed by the user, rk-chart employs the procedure outlned n Table 1 for choosng. Step 1: Table 1: Heurstc procedure for choosng for the Gaussan kernel. Calculate the average 'nearest neghbor dstance', denoted n d, between all the data ponts n the nnd nnd dataset (. e. n E ( x ) where x x x x x k, j E denotes d j k average operator over the tranng set ). Step : For each data pont x :.1: Construct a local robust kernel-dstance boundary, denotedrk-b L, utlzng the set S { x j : x j x ( nd ) } (. e. data wthn a sphere of radus n d ). In buldng L rk B, set E x E ( x ) for the Gaussan kernel (. e. the average dstance of - L S S * data wthn S to the mean of S ). Let r denote the optmal radus of rk-b L,. e. the quadratc programmng soluton. L *.: If x rk- B ( r ),. e. the data pont s rejected by a scaled or nner - L * rk B of radus r where 0 1 (a parameter pre-specfed by the user, typcally around 0.95), t s added nto the boundary lst, denoted BL S. It s also necessary that for any data pont to be part of BL S, t cannot be accepted by any other nner rk-b L j. In addton, f S x, x excluded from determnaton of S BL BL S. for t mght be an outler. Fg. 3 llustrates ths procedure for Step 3: The optmal global rk-b G (. e. the rk-chart boundary that represents the complete tranng set ) s then constructed usng the followng Gaussan kernel parameter: * BL SV arg max { S S ( )}. nnd nnd mn( x ) max( x ) Tax and Dun (Tax and Dun 1999) show that ths settng of the local sphere radus to n d results n good performance. Boundary Pont s Local Sphere of radus xn d Local rk-boundary Inner Local rk-boundary Interor Pont Fgure 3: Determnaton of class boundary lst: Data ponts on the boundary wll be rejected by nner local rk-boundares. 15
In the context of a global rk-chart, smaller values yeld more representng ponts (. e. support vectors) and a tghter hyper-sphere, whereas larger values gve fewer support vectors and result n a bgger hyper-sphere. The goal s to dentfy a value for that results n good agreement between the 'support vector lst' SV BL S of the global rk-chart boundary and the 'boundary lst' S resultng from local rk-boundares (see Table 1 for precse defntons of these terms, and Fgure 3 for a depcton of the class boundary lst): * BL SV arg max { S S ( )} (1) nnd nnd mn( x ) max( x ) In general, smaller values result n a global support vector lst that s a superset of the boundary lst, wth some ponts that are not part of the boundary lst. On the contrary, larger values result n a global support vector lst that s a subset of the boundary lst. In assessng ths agreement, rk-chart computes the ftness of a value by employng a two-part strategy: effectve representaton and compactness. Effectve representaton s acheved by ensurng that the global support vector lst 'best matches' the boundary lst. Compactness on the contrary emphaszes a smaller support vector lst, whch mproves generalzaton. Compactness s managed through a user-defned parameter 0 1. The hgher the value of the more compact the support vector lst and the hgher the Type II error (. e. nablty to detect novel condtons), resultng n a larger hyper-sphere. There s typcally a value, denoted by c, that results n near perfect agreement between the support vector lst and the boundary lst. As exceeds c, the support vector lst gets smaller. The actual value of employed n constructng the proposed global rk-chart s: ( ) () c max c Fgure 4 llustrates the nfluence of dfferent compactness levels on the qualty of representaton, usng an example dataset. The nnermost rk-chart boundary provdes effectve representaton but wth 0 support vectors all of whch are n BL S ( 0), whereas the outermost rk-chart boundary 16
acheves compactness wth just support vectors ( 1). Whle the parameters nnd nnd mn mn( x ), max max( x ), c are calculated emprcally, the compactness parameter and the rk-chart boundary scalng parameter (employed for constructng the scaled nner rk-b L ) need to be pre-specfed by the user or requre repeated trals (typcally set around 0.95). Fgure 4: Influence of on rk-chart representaton. Once the optmal value s calculated based on the desred degree of compactness, one can construct nner and outer boundary representatons by correspondngly changng the radus of the rk-chart hyper-sphere. Fgure 5 llustrates ths procedure for the same dataset from Fgure 4. It s clear that as the radus s changed the overall geometrc shape s mantaned whle the scale changes. Fgure 5: Influence of scalng rk-chart hyper-sphere radus (usng parameter ) on boundary representaton. 17
III.3. Learnng from abnormal data There are occasons when observaton samples from abnormal or out-of-control process states are avalable n the tranng set. Not explotng ths nformaton can have a major detrmental effect on the performance of the control chart (n terms of Type-I and Type-II errors). The earler rk- Chart formulaton can be modfed as follows to explot any avalable examples from out-ofcontrol states: r C C (3) Mn o j j Subject to: x c and r x c (4) j r j where Co s the penalty value for an out-of-control state data pont fallng nsde the representaton boundary. The Lagrange formulaton of ths new problem s as follows: L( r, c, ξ, α, γ) r C Co j j r ( x x cx cc) j j j (5) To satsfy KKT condton, we once agan take dervates of the cost functon w.r.t. the parameters and equate them to zero, leadng to the followng equatons: 1 (6) c x x (7) j j j Substtutng (6) and (7) n (5), leads us to followng Lagrange formulaton: L( r, c, ξ, α, γ) ( x x ) ( x x ) ( x x ) j j I J, ji ( x x ) ( x x ) j j j j I, jj J, ji, jj ( x x ) j j (8) 18
where, I s the class of n-control state examples and J s the class of out-of-control state examples. Ths formulaton can be smplfed by labelng out-of-state classes as '-1' and ncontrol class as '+1'. y 1 1 x x I J (9) y (30) ' Substtutng (9) and (30) n (8) results n the followng: Max ( x x ) ( x x ) (31) ' ' ' j j { I, J }, j{ I, J } 0 C (3) 0 j Co (33) 1 (34) As can be seen from equatons (31-34), the formulaton essentally remans the same wth an addtonal constrant for the correspondng Lagrange multpler value of the out-of-control state example. IV. Expermental results The applcaton of rk-chart wll be dscussed n two subsectons: Frst, rk-chart s evaluated usng datasets that follow common probablty dstrbutons (. e. normal, lognormal and exponental). We then evaluate rk-chart usng a benchmarkng dataset (. e. the Smth dataset) (Smth 1994) that has been used extensvely for methodologcal development and evaluaton n the process control lterature (Chnnam 00). Frst, normal, lognormal and exponental dstrbutons are employed to generate data. Parameters for the dstrbutons and a sample of dataset are gven n Table and Fgure 6, respectvely. For a typcal X chart, t s recommended to have at least 0-5 patterns (Woodall 19
and Montgomery 1999). In our experment, 50 samples are generated for each dstrbuton. Samples are grouped wth sze of 10 and two features, the mean and the standard devaton, are calculated for each sample group, resultng n only 5 two-dmensonal data ponts. Table : Parameters of dstrbutons for evaluaton experments. Normal Behavor (Nor) Small Mean Shft (SM) Large Mean Shft (LM) Small Varance Shft (SV) Large Varance Shft (LV) Normal Lognormal Exponental 0 0 1 1 1-1 1 1 1-3 3 4 1 1-0 0 - - 0 0-3 3-5 Exponental Dstrbuton.5 Normal Dstrbuton 1 Lognormal Dstrbuton 4.5 4 1.5 10 3.5 3 1 0.5 8.5 0 6-0.5 1.5-1 4 1 0.5-1.5-0 0 0 40 60 80 100 10 140 (a) 1 -.5 0 0 40 60 80 100 10 140 (b) 0, 1 0 0 0 40 60 80 100 10 140 (c) 0, 1 Fgure 6: Example tme Seres. a) IID Exponental, b) IID Normal, c) IID Lognormal. We traned rk-chart n two ways: Frst, wth only n-control data, and second, wth n-control and lmted out-of-control data as well. In the latter case, we used 10 abnormal patterns. As mentoned before, rk-chart does not requre out-of-control data, but havng some helps to mprove the accuracy of the method. Type-I and Type-II errors are defned as rejectng a true hypothess and acceptng a false hypothess, respectvely. In a statstcal process control context, Type-I error refers to rejectng an n-control process as f t s out-of-control and Type-II error refers to acceptng an out-of-control process as f t s n-control. The format of reportng Type-I and Type-II errors s gven n the Table 3. The results of rk-chart mplementaton on processes that follow normal, lognormal, and exponental dstrbutons are summarzed n Table 3. 0
Actual Table 3: Type-I and Type-II errors defned. Estmated Hypothess Test In-control Out-of-control In-control Correct Type-I error Out-of-control Type-II error Correct Table 4: Classfcaton accuracy for: 1) (Left sde of the table) Tranng only wth 5 n-control patterns or sub-groups, each of whch sub-group sample sze of 10. ) (Rght sde of the table) Tranng wth 5 ncontrol and 10 out-of-control patterns, each wth a sub-group sample sze of 10; a) Normal b) Lognormal c) Exponental dstrbutons; SM: Small Mean Shft, SV: Small Varance Shft, LM: Large mean shft, LV: Large varance shft. Actual Actual Actual Actual a) Normal Dstrbuton 1) Tranng only wth n-control data ) Tranng wth n-control and lmted outof-control data Estmated Estmated a.1 Out-ofcontrocontrol a. Out-of- In-control In-control In-control 88.3% 11.7% In-control 86.6% 13.4% SM 7.4% 9.6% SM 9.0% 91.0% LM 0.0% 100.0% LM 0.0% 100.0% SV 6.4% 93.6% SV 4.1% 95.9% LV 0.% 99.8% LV 0.1% 99.9% Out-ofcontrol Out-ofcontrol Estmated Estmated b.1 Out-ofcontrocontrol b. Out-of- In-control In-control In-control 73.% 6.8% In-control 81.3% 18.7% SM 4.5% 95.5% SM 9.5% 90.5% LM 0.0% 100.0% LM 0.0% 100.0% SV 14.8% 85.% SV 16.9% 83.1% LV 4.1% 95.9% LV 3.0% 97.0% Actual b) Lognormal Dstrbuton Out-ofcontrol Out-ofcontrol Estmated Estmated c.1 Out-ofcontrocontrol c. Out-of- In-control In-control In-control 88.8% 11.% In-control 80.8% 19.% 3.9% 76.1% 16.0% 84.0% olsm olsm LM 0.7% 99.3% LM 0.4% 99.6% Actual Outofcontr Outofcontr c) Exponental Dstrbuton As seen from Table 4, the non-parametrc rk-chart technque s able to detect out-of-control processes wth no avalable out-of-control data wth Type-I errors rangng from a hghest of 6.8% to a low of 11.%. Type-II errors ranged from 3.9% down to 0% n the case of no avalable out-of-control data. When lmted tranng data from faulty process states are avalable, 1
Type-I errors mprove to a hgh of 19.% and a low of 13.4%. Type-II errors also mprove wth lmted faulty process tranng data, to a hgh of 16.9% and a low of 0%. We wll also demonstrate the effectveness of rk-chart for detectng mean and varance shfts n usng the benchmarkng dataset generated by Smth (Smth 1994) and compare the results from a general support vector machne (SVM) method proposed by (Chnnam 00) and a mult-layerperceptron (MLP) neural network model (Smth 1994). The dataset has 300 samples from ncontrol state and out-of-control states of large mean shft (LM), small mean shft (SM), large varance shft (LV), and small varance shft (SV). The parameters are gven n Table 5. The results wll be reported n three dfferent categores. In the frst category, the results of rk-chart wll be compared wth results of Shewhart chart, MLP and SVM. Note that MLP and SVM create a dstnct model for each abnormalty type (e.g. small mean shft, large varance shft). In the second category, two dfferent rk-chart results are reported: rk-chart (1), whch s traned wth only n-control data and rk-chart (), whch s traned wth n-control and lmted out-of-control data. Note that MLP and SVM cannot be mplemented n case of absence of out-of-control samples. In the thrd category, the results of rk-chart that s traned wth very lmted n-control (.e. 5 samples) and out-of-control data (.e. 10 samples) are reported. Table 5: Parameters of testng datasets. State Label Normal Behavor (Nor) 0 1 Small Mean Shft (SM) 1 1 Large Mean Shft (LM) 3 1 Small Varance Shft (SV) 0 Large Varance Shft (LV) 0 3 Table 6: Classfcaton Accuracy of rk-chart versus MLP, Shewhart Charts and SVM charts usng the benchmarkng dataset from Smth (1994). *n: In-control data, out: Out-of-control data. rk-chart MLP Shewhart Control Charts SVM Test Tran Test Test Test Tran Small Shft 91% 9% 7% 73% 93% 91% Large Shft 96% N/A 100% 100% 100% 99%
In the frst category, classfcaton accuracy s reported n methods such as MLP, Shewhart chart and SVM nstead of Type-I and Type-II errors. Thus, the classfcaton accuracy of rk-chart s calculated n order to compare the results wth these methods. The weghted average of Type-I and Type-II errors are calculated, consderng the number of patterns used for n-control and outof-control data. As seen from Table 6, rk-chart performs better than MLP and Shewhart charts. SVM s only 1% to 4% better than rk-chart. Even though SVM method gves better results than rk-chart, there are two dffcultes of mplementng SVM and MLP n real world settngs: 1. A dstnct model s developed for each of the out-of-control state for SVM and MLP. There are four out-of-control states (SM, LM, SV, LV) resultng n four SVM and MLP models. Even though each model works well for developed out-of-control states, they cannot effectvely work for other out-of-control states that are yet to have a model. In addton, SVM and MLP may be senstve to an undefned out-of-control state. In contrast, rk-chart characterzes the n-control state of the process and t has the ablty to detect any undefned and unseen type of out-of-control state.. SVM and MLP are traned wth 300 samples from n-control state and 300 samples from each modeled out-of-control state resultng n 4 models usng a total of 400 samples. On the other hand, rk-chart uses only 300 samples from n-control state and 00 samples from out-of-control states. Table 7: Type-I and Type-II errors for the benchmarkng dataset from Smth (1994) usng rk-chart. In*: n-control state, Out*: Out-of-control state. rk-chart (1) : Tranng wth 300 normal samples. rk-chart () : Tranng wth 300 normal and 00 abnormal samples (100 small mean shft, 100 small varance shft) Actual Out* Estmated rk-chart (1) rk-chart () In* Out* In* Out* In* Out* In* Out* Tran Test Tran Test In-cont. 100% 0% 84% 16% 88% 1% 90% 10% SM N/A N/A 5% 95% 1% 99% 7% 93% LM N/A N/A 0% 100% N/A N/A 0% 100% SV N/A N/A 15% 85% 4% 96% 9% 91% LV N/A N/A 15% 85% N/A N/A 8% 9% 3
In the second category, we wll report rk-chart results wth no and lmted out-of-control data. In the former case (.e. rk-chart (1) ), only 300 samples of n-control data are used for tranng, n the latter case (.e. rk-chart () ) 300 samples of n-control and 100 samples of small mean shft and 100 samples of small varance shft are employed for tranng. The Type-I and Type-II errors as gven n Table 7 are promsng wth 16% Type-I error and hghest 15% Type-II error n rk- Chart (1) and 1% Type-I error and hghest 9% Type-II error wth rk-chart (). In the thrd category, rk-chart s mplemented wth very lmted n-control data. We mplemented rk-chart wth 5 tranng patterns from the benchmarkng Smth (Smth 1994) dataset and results are shown n Table 8. Table 8: Process state classfcaton for non-correlated Smth data wth lmted number of n-control and out-of-control samples (. e. 5 n-control and 10 out-of-control patterns) Actual Data sze of 5 out rk-chart n* out* n 88% 1% SM 9% 91% LM 1% 99% SV 8% 9% LV 4% 96% As seen from Table 8, n the worst of the cases tested, rk-chart had the power to effectvely detect out-of-control condtons 91% of the tme (9% Type-II error), wth false alarm rate of only 1% (Type-I error) even wth lmted data sze. As for parameter selecton n ntalzng rk-charts, when the control chart s ntalzed wth just n-control data, two parameters are requred: a penalty value for msclassfcaton of ncontrol data ( C ) and compactness ( ). The recommended default parameter values are 1.0 and 0., for C and, respectvely. An addtonal penalty parameter ( C 0 ) s requred for cases wth any avalable out-of-control data for ntalzaton. The recommended default parameter value s once agan 1.0. All the expermental results reported here are based on these default settngs. Extensve expermentaton wth these parameters suggests that the rk-chart s relatvely robust 4
wth respect to parameter value selecton and that the recommended default values work well n most cases. However, ndvdual applcatons can call for dfferent types of tradeoffs between Type-I and Type-II errors, whch call for approprate selecton or tunng of rk-chart parameters. V. CONCLUSION A new process control technque based on support vector machne prncples, called robust kernel-dstance control chart (rk-chart), s proposed for both unvarate and multvarate processes. rk-chart has several advantages over the conventonal SPC technques and pattern recognton methods n the lterature. rk-chart does not make any assumpton about the data dstrbuton, whch s a fundamental restrcton for conventonal SPC technques. In addton, conventonal SPC technques cannot beneft from avalable out-of-control data, whereas rk-chart can learn from out-of-control samples, where applcable. Pattern recognton methods used for process control n the lterature based on other support vector machne (SVM) prncples, radalbass functon (RBF) networks, and mult-layer-perceptron (MLP) neural networks requre excessve amount of n-control data as well as out-of-control data. Furthermore, a dstnct model for each type of out-of-control process needs to be created and traned usng both n-control and out-of-control data wth SVM, RBF and MLP. These models are not senstve to out-of-control condtons other than those on whch they are traned. In contrary, rk-chart characterzes ncontrol-processes and requres only n-control data. However, t makes an explct provson to accommodate any avalable out-of-control data. Thus, t s senstve to all types of out-of-control processes. It s also shown that rk-chart s able to gve very reasonable results wth lmted ncontrol and out-of-control data when tested usng a varety of datasets. ACKNOWLEDGEMENTS Ths research s partally funded by Natonal Scence Foundaton under grant DMI-030013. 5
REFERENCES Aradhye, H. B., Baksh, B. R., Strauss, R. A. and Davs, J. F. (001). Multscale statstcal process control usng wavelets: Theoretcal analyss and propertes. Columbus, OH, Oho State Unversty. Baksh, B., Multscale PCA wth applcaton to multvarate statstcal process montorng. Journal of the Amercan Insttute of Chemcal Engneers, 1998, 44, 1596-1610. Chakrabort, S., Van der Laan, P. and Bakr, S. T., Nonparametrc control charts: An overvew and some results. Journal of Qualty Technology, 001, 33, 304-315. Chakrabort, S., Van der Laan, P. and Van de Wel, M., A class of dstrbuton-free control charts. Journal of the Royal Statstcal Socety, Seres C, 004, 55(3), 443-46 Chen, Q., Kruger, U., Meronk, M. and Leung, A. Y. T., Synthetc of t and q statstcs for process montorng, control engneerng practce. Control Engneerng Practce, 004, 1, 745-755. Chnnam, R. B., Support vector machnes for recognzng shfts n correlated and other manufacturng processes. Internatonal Journal of Producton Research, 00, 40, 4449-4466. Chnnam, R. B. and Kolark, W. J., Automaton and the total qualty paradgm, n Proceedngs of the 1st IERC, 199, Chcago, IL, IIE. Cook, D. F. and Chu, C., Usng radal bass functon neural networks to recognze shfts n correlated manufacturng process parameters. IIE Transactons, 1998, 30, 7-34. Crstann, N. and Taylor, J. S., An ntroducton to support vector machnes and other kernel-based learnng methods. (Cambrdge: Cambrdge Unversty Press). Eckelmann, N. and Anant, A., Statstcal process control: What you don't measure can hurt you! Software, IEEE, 003, 0, 49-51. Hotellng, H. (1947). Multvarate qualty control-llustrated by the ar testng of sample bombsghts. Technques of statstcal analyss. C. Esenhart, M. W. Hastay and W. A. Walls. New York, McGraw-Hll: 111-184. Kuhn, H. and Tucker, A., Nonlnear programmng, n Proceedngs of nd Berkeley Symposum on Mathematcal Statstcs and Probablstcs, 1951, Berkely, CA, Unversty of Calforna Press, 481-49. Lu, R. Y. and Sngh, K., A qualty ndex based on data depth and multvarate rank tests. Journal of the Amercan Statstcal Assocaton (JASA), 1993, 88, 5-60. 6
Lowry, C. A. and Montgomery, D. C., A revew of multvarate control charts. IIE Transactons, 1995, 7, 800-810. Lowry, C. A., Woodall, W. H., Champ, C. W. and Rgdon, S. E., A multvarate exponentally weghted movng average control chart. Technometrcs, 199, 34, 46-53. MacGregor, J. F. and Kourt, T., Statstcal process control of multvarate processes. Control Engneerng Practce, 1995, 3, 403-414. Manabu, K., Shnj, H., Ior, H. and Hromu, O., Evoluton of multvarate statstcal process control: Applcaton of ndependent component analyss and external analyss. Computers & Chemcal Engneerng, 004, 8, 1157-1166. Mangasaran, O. L., Nonlnear programmng. (Phladelpha, PA: Socety for Industral and Appled Mathematcs). Martn, E. B., Morrs, A. J. and Zhang, J., Process performance montorng usng multvarate statstcal process control. IEE Proceedngs, 1996, 143, 13-144. Mercer, J., Functons of postve and negatve type and ther connecton wth the theory of ntegral equatons. Phlosophcal Transactons of the Royal Socety of London, Seres A, 1909, 09, 415-446. Messaoud, A., Wehs, C. and Herng, F. (004). A nonparametrc multvarate control chart based on data depth. Dortmund, Germany, Department of Statstcs, Unversty of Dortmund. Montgomery, D. C., Introducton to statstcal qualty control. (New York: Wley). Müller, K. R., Mka, S., Rätsch, G., Tsuda, K. and Schölkopf, B., An ntroducton to kernel-based learnng algorthms. IEEE Neural Networks, 001, 1, 181-01. Nga, H. and Zhang, J., Multvarate cumulatve sum control charts based on projecton pursut. Statstca Snca, 001, 11, 747-766. Nmmo, I., Adequately address abnormal stuaton operatons. Chemcal Engneerng Progress, 1995, 91, 36-45. Polansky, A. M., A smooth nonparametrc approach to multvarate process capablty. Technometrcs, 001, 43, 199-11. Pugh, G. A., A comparson of neural networks to spc charts. Computers and Industral Engneerng, 1991, 1, 53-55 7
Rach, A. and Cnar, A., Statstcal process montorng and dsturbance dagnoss n multvarate contnuous processes. Journal of the Amercan Insttute of Chemcal Engneers, 1996, 4, 995-1009. Rose, K., Mathematcs of success and falure. Crcuts and Devces, IEEE, 1991, 7, 6-30. Runger, G. C. and Prabhu, S. S., A markov chan model for the multvarate exponentally weghted movng averages control chart. Journal of the Amercan Statstcal Assocaton (JASA), 1996, 91, 1701-1706. Runger, G. C. and Testk, M. C., Multvarate extensons to cumulatve sum control charts. Qualty and Relablty Engneerng Internatonal, 004, 0, 587-606. Schllng, E. G. and Nelson, P.R., The effect of non-normalty on the control lmts of X charts. Journal of Qualty Technology, 1976, 8, 183-187. Shewhart, W. A., Qualty control charts. Bell System Techncal Journal, 196,, 593-603. Smth, A. E., X-bar and r control chart nterpretaton usng neural computng. Internatonal Journal of Producton Research, 1994, 3, 309-30. Stoumbos, Z. G. and Reynolds, M. R., On shewhart-type nonparametrc multvarate control charts based on data depth. Fronters n Statstcal Qualty Control, 001, 6, 07-7. Sun, R. and Tsung, F., A kernel-dstance-based multvarate control chart usng support vector methods. Internatonal Journal of Producton Research, 003, 41, 975-989. Tax, D. M. (001). One class classfcaton. Delft, The Netherlands, Delft Techncal Unversty. Tax, D. M. J. and Dun, R. P. W., Support vector doman descrpton. Pattern Recognton Letters, 1999, 0, 1191-1199. Tax, D. M. J. and Dun, R. P. W., Support vector data descrpton. Machne Learnng, 004, 54, 45-66. Teppola, P. and Mnkknen, P., Wavelet-pls regresson models for both exploratory data analyss and process montorng. Journal of Chemometrcs, 000, 14, 383-399. Testk, M. C. and Borror, C. M., Desgn strateges for the multvarate exponentally weghted movng average control chart. Qualty and Relablty Engneerng Internatonal, 004, 0, 571-577. 8
Vapnk, V., Statstcal learnng theory. (New York: Wley). Woodall, W. H. and Montgomery, D. C., Research ssues and deas n statstcal process control. Journal of Qualty Technology, 1999, 31, 376-386. Yoon, S. and MacGregor, J. F., Prncpal-component analyss of multscale data for process montorng and fault dagnoss. Journal of the Amercan Insttute of Chemcal Engneers, 004, 50, 891-903. Yuan, C. and Casasent, D., Support vector machnes for class representaton and dscrmnaton, n Internatonal Jont Conference on Neural Networks, 003, Portland, OR, 1610-1615. 9