FOR understanding behavior of people in a large area of

Transcription

1 No commercal use ^_^ IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 35, NO. 3, MARCH Redentfcaton by Relatve Dstance Comparson We-Sh Zheng, Member, IEEE, Shaogang Gong, and Tao Xang Abstract Matchng people across nonoverlappng camera vews at dfferent locatons and dfferent tmes, known as person redentfcaton, s both a hard and mportant problem for assocatng behavor of people observed n a large dstrbuted space over a prolonged perod of tme. Person redentfcaton s fundamentally challengng because of the large vsual appearance changes caused by varatons n vew angle, lghtng, background clutter, and occluson. To address these challenges, most prevous approaches am to model and extract dstnctve and relable vsual features. However, seekng an optmal and robust smlarty measure that quantfes a wde range of features aganst realstc vewng condtons from a dstance s stll an open and unsolved problem for person redentfcaton. In ths paper, we formulate person redentfcaton as a relatve dstance comparson (RDC) learnng problem n order to learn the optmal smlarty measure between a par of person mages. Ths approach avods treatng all features ndscrmnately and does not assume the exstence of some unversally dstnctve and relable features. To that end, a novel relatve dstance comparson model s ntroduced. The model s formulated to maxmze the lkelhood of a par of true matches havng a relatvely smaller dstance than that of a wrong match par n a soft dscrmnant manner. Moreover, n order to mantan the tractablty of the model n large scale learnng, we further develop an ensemble RDC model. Extensve experments on three publcly avalable benchmarkng datasets are carred out to demonstrate the clear superorty of the proposed RDC models over related popular person redentfcaton technques. The results also show that the new RDC models are more robust aganst vsual appearance changes and less susceptble to model overfttng compared to other related exstng models. Index Terms Person redentfcaton, feature quantfcaton, feature selecton, relatve dstance comparson Ç 1 INTRODUCTION FOR understandng behavor of people n a large area of publc space covered by multple nonoverlappng (dsjont) cameras, t s crtcal that when a target dsappears from one vew, he/she can be redentfed n another vew at a dfferent locaton among a crowd of people. Solvng ths ntercamera people assocaton problem, known as redentfcaton, enables trackng of the same person through dfferent camera vews located at dfferent physcal stes [26], [15], [32], [17], [8]. Despte the best efforts from computer vson researchers n the past fve years, the person redentfcaton problem remans largely unsolved. Ths s due to a number of reasons. Frst, n a busy uncontrolled envronment montored by cameras from a dstance, person verfcaton relyng upon bometrcs such as face and gat s nfeasble and unrelable. Second, as the transton tme between dsjont cameras 1 vares greatly from ndvdual to ndvdual wth uncertanty, 1. The tme gap between a person dsappearng n one camera vew and reappearng n another.. W.-S. Zheng s wth the School of Informaton Scence and Technology, Sun Yat-sen Unversty, Guangzhou, Guangdong , P.R. Chna. E-mal: wszheng@eee.org.. S. Gong and T. Xang are wth the School of Electronc Engneerng and Computer Scence, Queen Mary Unversty London, Mle End Road, London E1 4NS, Unted Kngdom. E-mal: {sgg, txang}@eecs.qmul.ac.uk. Manuscrpt receved 7 Sept. 2011; revsed 9 Feb. 2012; accepted 30 May 2012; publshed onlne 20 June Recommended for acceptance by B. Schele. For nformaton on obtanng reprnts of ths artcle, please send e-mal to: tpam@computer.org, and reference IEEECS Log Number TPAMI Dgtal Object Identfer no /TPAMI t s hard to mpose accurate temporal and spatal constrants. Therefore, the person redentfcaton problem s made harder stll as a model can only rely on mostly appearance features alone. Thrd, the vsual appearance features, extracted manly from the clothng and shapes of people, are ntrnscally ndstnctve for matchng people (e.g., most people n wnter wear dark clothes). In addton, a person s appearance often undergoes large varatons across nonoverlappng camera vews due to sgnfcant changes n vew angle, lghtng, background clutter, and occluson (see Fg. 1), resultng n dfferent people appearng more alke than that of the same person across dfferent camera vews (see Fgs. 6 and 7). Gven a query mage of a person, n order to fnd the correct match among a large number of canddate mages captured from dfferent camera vews, two steps need to be taken. Frst, a feature representaton s computed from both the query and each of the gallery mages. Second, the dstance between each par of potental matches s measured, whch s then used to determne whether a gallery mage contans the same person as the query mage. Most exstng studes have focused on the frst step, that s, seekng a more dstnctve and relable feature representaton of people s appearance, rangng wdely from color hstogram [26], [15], graph model [10], spatal co-occurrence representaton model [32], prncpal axs [17], rectangle regon hstogram [6], part-based models [1], [4], to combnatons of multple features [15], [8]. After feature extracton, these methods smply choose a standard dstance measure such as l 1 -norm [32], l 2 -norm-based dstance [17], or Bhattacharyya dstance [15]. However, under severe changes n vewng condtons that can cause sgnfcant appearance varatons (e.g., vew /13/$31.00 ß 2013 IEEE Publshed by the IEEE Computer Socety

2 No commercal use ^_^ 654 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 35, NO. 3, MARCH 2013 Fg. 1. Typcal examples of appearance changes caused by cross-vew varatons n vew angle, lghtng, background clutter, and occluson. Each column shows two mages of the same person from two dfferent camera vews. angle and lghtng condton changes, occluson), computng a set of features that are both dstnctve and relable s extremely hard f not mplausble. Moreover, gven that certan features could be more relable than others under a certan condton, applyng a standard dstance measure s undesrable as t essentally treats all features equally wthout dscardng bad features selectvely n each ndvdual matchng crcumstance. In ths paper, we focus on the second step of person redentfcaton. That s, gven a set of features extracted from each person mage, we seek to quantfy and dfferentate these features by learnng the optmal dstance measure that s most lkely to gve correct matches. Ths s sgnfcantly dfferent from most exstng approaches n that t requres model learnng from a set of tranng data. In essence, mages of each person n a tranng set form a class. Ths learnng problem can be framed as a dstance learnng problem whch always searches for a dstance that mnmzes ntraclass dstances whle maxmzng nterclass dstances. However, the person redentfcaton problem has four characterstcs. 1. The ntraclass varaton can be large and, more mportantly, can vary sgnfcantly for dfferent classes as t s caused by large and unpredctable vewng condton changes (see Fg. 1). 2. The nterclass varaton also vares drastcally across dfferent pars of classes and there are often severe overlaps between classes n a feature space due to smlar appearance (e.g., clothng) of dfferent people. 3. The tranng set for learnng the model conssts of mages of matched people across dfferent camera vews. In order to capture the large ntra and ntervaratons, the number of classes s necessarly large, typcally on the order of hundreds. Ths represents a large scale learnng problem that challenges exstng machne learnng algorthms. 4. Annotatng a large number of matched people across camera vews s not only tedous, but also nherently lmted n ts usefulness. Typcally, each annotated class contans only a handful of mages of a person from dfferent camera vews,.e., the data are nherently undersampled for buldng a representatve class dstrbuton. Due to these ntrnsc characterstcs of the redentfcaton problem, especally the problem of a large number of undersampled classes, a learnng model could easly be overftted and/or be ntractable f t s learned by mnmzng ntraclass dstance and maxmzng nterclass dstance smultaneously by brute-force, as s typcally done by exstng popular dstance learnng technques. To allevate ths nherently ll-posed dstance learnng problem n person redentfcaton, we formulate the problem as a relatve dstance comparson (RDC) problem. That s, we perform feature quantfcaton by learnng a relatve dstance comparson model. More specfcally, a novel relatve dstance comparson model s formulated n order to dfferentate the smlarty score of a par of true match (.e., two mages of person A) from that of a par of related wrong match (.e., two mages of dfferent people A and B, respectvely) so that the latter one can always be smaller. In other words, the model ams to learn an optmal dstance n the sense that for a gven query mage, the true match s desred to be ranked hgher than the wrong matches among the gallery mage set. The model cares less about how large the absolute dstance between the par of mages for the true match. Ths dffers conceptually from a conventonal dstance learnng approach whch ams to mnmze ntraclass varaton n an absolute sense (.e., makng all mages of person A more smlar or closer n a features space) whle maxmzng nterclass varaton (.e., makng two mages of person A and B more dssmlar). A conventonal approach thus attempts to maxmze the margn between two classes or, n the context of person redentfcaton, enforces a harder dscrmnant constrant that the true match s not only ranked hgher but also has as small a dstance to the query mage as possble compared to that of wrong matches. One of the key advantages of our relatve dstance comparson-based method s that our model s not easly based by large varatons across many undersampled classes as t ams to seek an optmzed ndvdual comparson between any two data ponts rather than comparson among data dstrbuton boundares or among clusters of data. Ths allevates the overfttng problem n person dentfcaton gven undersampled tranng data. Computatonally, learnng the proposed relatve dstance comparson model can be a nonconvex optmzaton problem. It s also a large scale learnng problem even gven a moderate tranng data sze. Ths s because the dstance between each par of mages n a tranng set needs be compared exhaustvely durng model learnng and the feature space for person redentfcaton s typcally of hgh dmenson. To address ths problem, a novel teratve optmzaton algorthm s developed n ths work for learnng the RDC model. The algorthm s theoretcally valdated and ts convergence s guaranteed. Furthermore, n order to allevate the large space complexty (memory usage cost) and the local optmum learnng problem due to the proposed teratve algorthm for solvng hgh-order nonlnear optmzaton crteron, we develop an ensemble RDC n ths work. The am s to learn a set of weak RDC models, each computed on a small subset of data, and then combne them nto a stronger RDC usng ensemble learnng. Extensve experments are conducted on three publcly avalable large person redentfcaton datasets, ncludng the ETHZ [7], -LIDS [37], and VIPeR [14] datasets. The results demonstrate that 1) by formulatng the person redentfcaton problem as a relatve dstance comparson

3 No commercal use ^_^ ZHENG ET AL.: REIDENTIFICATION BY RELATIVE DISTANCE COMPARISON 655 TABLE 1 Man Development of Person Redentfcaton learnng problem based on logstc functon modelng, sgnfcant mprovement on matchng accuracy can be obtaned aganst related popular person redentfcaton technques; and 2) our RDC models outperform not only related dstance learnng methods but also related learnng methods based on boostng and rank support vector machnes (SVMs), both n terms of matchng accuracy and tractablty. 2 RELATED WORKS The problem of matchng people across dsjont camera vews has receved ncreasng attenton n recent years. Exstng works predomnantly focus on the problem of feature extracton and representaton wth a bag-of-word representaton of color and texture features beng the most common choce. Table 1 summarzes the features and representatons employed by exstng methods reported n the lterature. In addton to matchng based on smlarty of vsual appearance, contextual cues can also be exploted. Brghtness transfer functon s ntroduced to explctly compensate for the lghtng condton changes between cameras [3], [27], [18]. However, to learn a brghtness transfer functon one has to not only annotate a set of matched people but also segment each person from the mage, whch sgnfcantly ncreases the already large annotaton cost. The temporal relatonshps between camera vews can be exploted for object taggng. By modelng the transton tme between two camera vews one can reduce the number of potental matches whle also usng the probablty dstrbuton of transton tme as a feature [12], [25], [24], [22]. However, transton tme nformaton could be unrelable when camera vews are sgnfcantly dsjont or feature a large number of movng objects. Nevertheless, when t can be obtaned relably, t has been exploted to good effect (see Table 1, column 4). Such contextual constrants can also be easly employed to the proposed RDC models ether as part of the representaton or a postprocessng step. Snce not all features are equally relable and nformatve for person redentfcaton, Gray and Tao [15] propose a boostng approach based on Adaboost to select a subset of optmal features for matchng people. However, n a boostng framework, good features are only selected ndvdually and ndependently n the orgnal feature space where dfferent classes can be heavly overlapped. Such selecton may not be globally optmal. Rather than selectng features ndvdually and ndependently (local selecton), we consder nstead quantfyng all features jontly (global selecton). Crtcally, the Adaboost-based feature selecton method n [15] could be based by large varatons between the appearance of people as ts modelng shares smlar sprt wth a typcal dscrmnant model that tres to maxmze the dfference between two mages of dfferent people. It s thus prone to model overfttng, as shown n our experments (see Secton 6). In contrast, the proposed RDC model can be seen as a soft dscrmnant approach. Our model thus s less susceptble to overfttng and more tolerant to ntra and nterclass varatons and severe overlappng of dfferent classes n a multdmensonal feature space. Relatve dstance comparson s a specal case of learnng to rank or machne-learned rankng. Rankng technques such as RankSVM [16] and RankBoost [9] have been wdely used n text document analyss and nformaton retreval. In our early work [28], the prmal RankSVM [2] s appled to solve the problem of global feature quantfcaton for person redentfcaton. The prmal RankSVM solves the hgh computatonal cost problem for large scale constrant optmzaton n a standard RankSVM formulaton. Compared to RankSVM and RankBoost, the proposed new model n ths paper s more prncpled and tractable n three aspects: 1) RDC s a second-order feature quantfcaton model, takng nto account the joned effect between dfferent features, whereas both RankSVM [2] and Rank- Boost [9] are a frst-order model unable to explot correlatons among dfferent features. 2) RDC utlzes a logstc functon to provde a soft margn measure between the dfference vectors of dfferent types whle RankSVM does not, and such a formulaton of our objectve functon makes RDC more tolerant to large ntra and nterclass varatons and better suted for copng wth data undersamplng. 3) Usng a prmal RankSVM, one must determne the weght between the margn functon and the rankng error cost functon, whch s computatonally costly. In contrast, our RDC model does not suffer from such a problem, leadng to lower computatonal cost. A more detaled dscusson on the

4 No commercal use ^_^ 656 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 35, NO. 3, MARCH 2013 dfferences between RDC and related rankng models s gven n Secton 5. Extensve experments are presented n Secton 6.6 to valdate the advantages of RDC over RankSVM and RankBoost. Although t has not prevously been exploted for person redentfcaton, dstance learnng n general s a wellstuded problem [35], [13], [36], [34], [15], [29], [33], [20], [5]. The proposed RDC model s related to several exstng dstance learnng methods. In partcular, our model shares the same sprt wth a number of recent works that explot the dea of relatve dstance comparson [29], [33], [20]. However, the relatve dstance comparson formulatons n these works are not quantfed usng logstc functon for soft measure, and crucally they are used as an optmzaton constrant rather than an objectve functon. Therefore, as analyzed n more detal n Secton 5, these approaches, ether mplctly [29], [20] or explctly [33], stll am to learn a dstance by whch each class becomes more compact whle beng more separable from each other n an absolute sense. We demonstrate through extensve experments that, n practce, they reman susceptble to model overfttng and poor tractablty for person redentfcaton. In summary, the man contrbutons of ths work are three-fold. 1. For the frst tme, the person redentfcaton problem s formulated as a relatve dstance comparson learnng problem, wth strong ratonale both conceptually and computatonally. 2. We propose a novel logstc functon-based relatve dstance comparson model for feature quantfcaton whch overcomes the lmtatons of exstng dstance learnng technques gven undersampled data wth large ntra and nterclass varatons. 3. A novel teratve optmzaton algorthm and an ensemble RDC model are proposed to mprove the tractablty of the RDC model and make t more sutable for large scale learnng. An early verson of ths work appeared n [38]. In addton to gvng a more detaled descrpton of the RDC model, the man changes nclude 1) an ensemble RDC model proposed to mprove the scalablty and tractablty of the orgnal RDC model, 2) more n-depth dscusson and analyss on ts relatonshp to alternatve learnng methods, and 3) more extensve expermental evaluatons ncludng the ntroducton of a new dataset. 3 QUANTIFYING FEATURES FOR PERSON REIDENTIFICATION 3.1 Proposed Relatve Dstance Comparson Learnng We formally cast the person redentfcaton problem nto the followng dstance comparson problem, where we assume each nstance of a person s represented by a feature set (e.g., the representaton descrbed n Secton 6.2). For an nstance z of person A, we wsh to learn a redentfcaton model to successfully dentfy another nstance z 0 of the same person captured elsewhere n space and tme. Ths s acheved by learnng a dstance functon fð; Þ so that fðz; z 0 Þ <fðz; z 00 Þ, where z 00 s an nstance of any other person except A. To ths end, gven a tranng set Z¼fðz ;y Þg N ¼1, where z 2R q s a multdmensonal feature vector representng the appearance of a person n one vew and y s ts class label (person ID), we defne a parwse set O ¼fO ¼ðx p ; xn Þg, where each element of a par-wse data O tself s computed usng a par of sample feature vectors. More specfcally, x p s a dfference vector computed between a par of relevant samples (of the same class/person) and x n s a dfference vector from a par of related rrelevant samples,.e., only one sample for computng x n s one of the two relevant samples for computng x p and the other s a msmatch from another class (e.g., x p and x n share the same z n the followng (1), whle they have dfferent z 0 ). The dfference vector x between any two samples z and z 0 s computed by x ¼ d z; z 0 ; z; z 0 2R q ; ð1þ where d s an entry-wse dfference functon that outputs a dfference vector between z and z 0. The specfc form of functon d wll be descrbed n Secton 3.4. Gven the parwse set O, a dstance functon f wll take the dfference vector as nput and can be learned based on relatve dstance comparson so that a dstance between a relevant sample par (fðx p Þ) s wshed to be smaller than that between a related rrelevant par (fðx n Þ). In order to dfferentate these two types of dfference vectors, we propose a logstc functon based modelng to descrbe how a dstance between a relevant par dffers from the one between a related but rrelevant par as follows: C f x p ; xn ¼ 1 þ exp f x p f x n 1: ð2þ We assume the events of dstance comparson between a relevant par and a related rrelevant par are ndependent 2. Then, we wsh to mnmze the rsk of learnng f va all the above relatve dstance comparsons as follows: mn rðf; OÞ; rðf; OÞ ¼log Y C f x p ;! xn : f O The dstance functon f s parameterzed as a Mahalanobs (quadratc) dstance functon: ð3þ fðxþ ¼x T Mx; M 0; ð4þ where M s a semdefnte matrx. The dstance learnng problem thus becomes learnng M usng (3). Drectly learnng M usng semdefnte program technques s computatonally expensve for hgh-dmensonal data [33]. In partcular, we found out n our experments that gven a dmensonalty of thousands, typcal for vsual object representaton, a dstance learnng method based on learnng M becomes ntractable. To overcome ths problem, we perform egenvalue decomposton on M: M ¼ AA T ¼ WW T ; W ¼ A 1 2 ; ð5þ where the columns of A are orthonormal egenvectors of M and the leadng dagonal of contans the correspondng nonzero egenvalues. Note that the columns of W form a set 2. Note that we do not assume the data are ndependent.

5 No commercal use ^_^ ZHENG ET AL.: REIDENTIFICATION BY RELATIVE DISTANCE COMPARISON 657 of orthogonal vectors. Therefore, learnng a functon f s equvalent to learnng such a matrx W ¼ðw 1 ;...; w l ;... ; w L Þ such that mn r W; O ; s:t: w T w j ¼ 0; 8 6¼ j W r W; O ¼ X log 1 þ exp W T x p 2 W T x n 2 : O We call ths relatve dstance comparson learnng for person redentfcaton. RDC s based on a logstc functon rangng from 0 to 1 n value. Ths s desgned to avod dramatc changes n the response to dfferent relatve dstance comparsons. 3.2 An Iteratve Optmzaton Algorthm It s mportant to pont out that our optmzaton crteron (6) may not be a convex optmzaton problem aganst the orthogonal constrant due to the logstc functon-based relatve comparson modelng. It means that dervng an global soluton by drectly optmzng W s not straghtforward. In ths work, we formulate an teratve optmzaton algorthm to learn an optmal W, whch also ams to seek a low-rank and nontrval soluton automatcally. Ths s crtcal for reducng the model complexty, thus allevatng the overfttng problem gven a large number of undersampled classes. Startng from an empty matrx, after teraton a new estmated column w s added to W. The algorthm termnates after L teratons when a stoppng crteron s met. Each teraton conssts of two steps as follows: Step 1. Assume that after teratons a total of orthogonal vectors w 1 ;...; w have been learned. To learn the next orthogonal vector w þ1, let a þ1 ¼ exp X ( ) j¼0 kw T j xp;j k 2 kw T j xn;j k 2 ð6þ ; ð7þ where we defne w 0 ¼ 0, and x p; and x n; are the dfference vectors at the th teraton defned as follows: x s; ¼ x s; 1 ~w 1 ~w T 1 xs; 1 ; s 2fp; ng; ¼ 1;...; O ; where 1 and ~w 1 ¼ w 1 =kw 1 k. Note that we defne x s;0 ¼ x s, s 2fp; ng, and ~w 0 ¼ 0. Step 2. Obtan x p; þ1, x n; þ1 by (8). Let O þ1 ¼fO þ1 ¼ ðx p; þ1 ; x n; þ1 Þg. Then, learn a new optmal projecton w þ1 on O þ1 as follows: w þ1 ¼ arg mn r þ1 w; O þ1 ; ð9þ w where r þ1 w; O þ1 ¼ X O þ1 log 1 þ a þ1 exp w T x p; þ1 2 w T x n; þ1 2 : We seek a soluton by a gradent descent method w þ1 w þ1 ; 0; ð8þ þ1 ¼ X O þ1 2 a þ1 1 þ a þ1 x p; þ1 exp kw T þ1 xp; þ1 k 2 kw T þ1 xn; þ1 k 2 exp kw T þ1 xp; þ1 k 2 kw T þ1 xn; þ1 k 2 x p; þ1 T x n; þ1 x n; þ1 T w þ1 ; where s a step length automatcally determned at each gradent update step usng smlar strategy n [23]. Accordng to the descent drecton n (10), the ntal value of w þ1 for the gradent descent method s set to w þ1 ¼jO þ1 j -1X O þ1 x n; +1 x p; +1 : ð11þ Note that the update n (8) deducts nformaton from each sample x s; -1 affected by w 1 as w T 1 xs; ¼ 0 so that the next learned vector w wll only quantfy the part of the data left from the last step,.e., x s;. In addton, a þ1 ndcates the trends n the change of dstance measures for x p and x n over prevous teratons and serve as a pror weght for learnng w. The teraton of the algorthm (for >1) s termnated when the followng crteron s met: r w ; O r +1 w +1; O þ1 <"; ð12þ where " s a small tolerance value set to 10-6 n ths work. The algorthm s summarzed n Algorthm Theoretcal Valdaton The followng two theorems valdate the clam that the proposed teratve optmzaton algorthm learns a set of orthogonal vectors fw g that teratvely decrease the objectve functon n Crteron (6). Theorem 1. The learned vectors w, ¼ 1;...;L, are orthogonal to each other. Proof. Assume that 1 orthogonal vectors fw j g 1 j¼1 have been learned. Let w be the optmal soluton of Crteron (9) at the teraton. Frst, we know that w s n the range space 3 of fx p; g[fx n; g accordng to (10) and (11),.e., w 2 spanfx s; ;¼ 1;...; j Oj;s2fp; ngg. Second, accordng to (8), we have 3. Ths can also be explored by usng Lagrangan equaton for (9) for a nonzero w.

6 No commercal use ^_^ 658 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 35, NO. 3, MARCH 2013 w T j xs;jþ1 ¼ 0; s2fp; ng; j¼ 1;...; 1 spanfx s; ;¼ 1;...; j Oj;s2fp; ngg span x s; 1 ;¼ 1;...; j Oj;s2fp; ng span x s;0 ;¼ 1;...; j Oj;s2fp; ng : Hence, w s orthogonal to w j, j ¼ 1;...; 1. ð13þ Theorem 2. rðw þ1 ; OÞ rðw ; OÞ, where W ¼ðw 1 ;...; w Þ, 1. That s, the algorthm teratvely decreases the objectve functon value. Proof. Let w þ1 be the optmal soluton of (9). By Theorem 1, t s easy to prove that for any j 1, w T j xs;j ¼ w T j xs;0 ¼, s 2fp; ng. Hence, we have w T j xs r þ1 w þ1 ; O þ1 ¼ X O þ1 log 1 þ a þ1 ¼ rðw þ1 ; OÞ: exp kw T þ1 xp; þ1 tu k 2 kw T þ1 xn; þ1 k 2 Also r þ1 ð0; O þ1 Þ¼rðW ; OÞ. Snce w þ1 s the mnmal soluton, we have r þ1 ðw þ1 ; O þ1 Þr þ1 ð0; O þ1 Þ, and therefore rðw þ1 ; OÞ rðw ; OÞ. tu Snce Crteron (9) may not be convex, a local optmum could be obtaned n each teraton of our algorthm. However, even f the computaton was trapped n a local mnmum of (9) at the þ 1 teraton, Theorem 2 s stll vald f r þ1 ðw þ1 ; O þ1 Þr ðw ; O Þ; otherwse the algorthm wll be termnated by the stoppng crteron (12). To allevate the local optmum problem at each teraton, multple ntalzatons could be deployed n practce. In ths work, we formulate an ensemble algorthm n Secton 4 to allevate the problem of local optmum. 3.4 Learnng n an Absolute Data Dfference Space To compute the data dfference vector x defned n (1), most exstng dstance learnng methods use the followng entrywse dfference functon, x ¼ dðz; z 0 Þ¼z z 0 ; ð14þ to learn M ¼ WW T n the normal data dfference space denoted by DZ ¼ fx j ¼ z z z j ; z j 2Zg. The learned dstance functon s thus wrtten as fðx j Þ¼ z z j T M z z j ¼kW T x j k 2 : ð15þ In ths work, we compute the dfference vector by the followng entry-wse absolute dfference functon: x ¼ dðz; z 0 Þ¼ z z 0 ; xðkþ ¼ zðkþz 0 ðkþ ; ð16þ where zðkþ s the kth element of the sample feature vector. M s thus learned n an absolute data dfference space, denoted by DZ ¼fjxj j¼jz z j j z ; z j 2Zg, and our dstance functon, whch s a symmetrc Premetrcs, becomes fðjx j jþ ¼ jz z j j T Mjz z j j¼kw T jx j j k 2 : ð17þ We now explan why learnng n an absolute data dfference space s more sutable to our relatve comparson model. Frst, we note that jz ðkþz j ðkþj jðz ðkþz j 0ðkÞj jðz ðkþz j ðkþþ ðz ðkþz j 0ðkÞÞj; ð18þ hence we have jx j jjx j 0j: jx j x j 0j, where : san entry-wse. As jx j j; jx j 0j0, we thus can prove jxj jjx j 0j xj x j 0 : ð19þ Ths suggests that the varaton of jx j j gven the same sample space Z s always less than that of x j. Specfcally, f z ; z j ; z j 0 are from the same class, the ntraclass varaton s smaller n jdzj than n DZ. On the other hand, f z j and z j 0 belong to a dfferent class than z, the varaton of nterclass dfferences s also more compact n the absolute data dfference space. Snce the varatons of both relevant and rrelevant sample dfferences x p and x n are smaller, the learned dstance functon usng (6) would yeld more consstent dstance comparson results, therefore benefttng our RDC model. Specally, for the same semdefnte matrx M, by combnng (19) and the Cauchy nequalty, we have upperð W T ðjx j jjx j 0jÞ Þupperð W T ðx j x j 0Þ Þ; where upperðþ s the upper bound operaton. Ths ndcates that n the latent subspace nduced by W, the maxmum varaton of jx j j T Mjx j j s lower than that of x T j Mx j. We show notable beneft of learnng RDC n an absolute data dfference space n our experments. 4 ENSEMBLE LEARNING FOR LARGE SCALE COMPUTATION The proposed RDC s based on the comparson between each relevant and related rrelevant pars and optmzed by an teratve algorthm. However, there are the two followng remanng ssues could stll hnder the tractablty of the proposed model. 1. Frst, the number of comparsons can thus be very hgh gven even a moderate tranng data sze. Specfcally, the amount of these parwse comparson could lead to a consderably large space complexty (memory usage cost). For nstance, let us assume there are N mages n total n a tranng set belongng to L people. Assumng there are N L mages for each person, we can learn an RDC wth a space complexty of Oðq ðð 1 L 1 L ÞN 3 þð 1 2 L 1ÞN2 ÞÞ, where q s the dmenson of the feature space. Ths hgh space complexty s thus caused by both the N 3 term and the typcally hgh feature dmenson q. 2. Second, although the proposed teratve optmzaton algorthm can effectvely handle the hgh order nonconvex optmzaton problem, t could stll be trapped nto a local optmum. To allevate these two problems, rather than learnng a batch mode RDC, we propose learnng a set of weak RDC models, each computed usng a small subset of the data, and then combnng them to buld a stronger RDC usng ensemble learnng. More specfcally, by usng the dea of ensemble learnng, a strong RDC model f s ðxþ s constructed by a set of H weak RDC models f w; ðxþ as follows:

7 No commercal use ^_^ ZHENG ET AL.: REIDENTIFICATION BY RELATIVE DISTANCE COMPARISON 659 f s ðxþ ¼ XH f w; ðxþ; ¼1 ð20þ where f w; ðxþ are defned as n (4) and s the weght of each weak RDC model. Learnng weak RDC models f w; Each weak RDC model s learned usng a dfferent subset of the tranng samples. More specfcally, to learn H weak models, the tranng dataset s dvded nto H groups. Assumng there are n total L people/classes C¼fC 1 ;...; C L g, we frst equally dvde them nto H groups G 1 ;...;G H wthout overlap,.e., C¼ S H ¼1 G T and 8 6¼ j, G Gj ¼;. Subsequently, the tranng dataset Z s dvded nto H subsets Z 1 ;...; Z H as follows: Z ¼fðx ;y Þjy 2 G g: ð21þ Then for each subset Z, another subset of samples O s randomly selected from the remanng samples (.e., B percent of the data n ZZ ). In ths paper, H and B are set to be 50 and 40, respectvely. Fnally, these two subsets Z and O are merged to form the fnal tranng set for learnng the th weak model usng the batch-mode method descrbed n Secton 3.1. Note that Z and O are formed n a dfferent way n that O s drawn randomly. By ntroducng a random component n the data subset we ensure that the feature space s to some extent well sampled for each weak model. Learnng Suppose H weak RDC models ff w; g H ¼1 have been learned from the prevous step. We now explore boostng to learn the weght on the whole dataset Z teratvely (see Algorthm 2). Specfcally, at the tth step, we frst select the best weak dstance model f w;kt that mnmzes the followng cost functon: k t ¼ arg mn X O j D j t f w; x p j >fw; x n j ; ð22þ where D j t s the weght of parwse dfference vectors at the tth step, P j Oj j¼1 Dj t ¼ 1, and s a Boolean functon. Then, Dj t s updated as follows: D j tþ1 ¼ F 1 D j t exp t f w;kt x p j fw;kt x n j ; ð23þ where F s the normalzer such that P j Oj j¼1 Dj tþ1 ¼ 1. The weght t for the selected weak model f w;kt s then determned by t ¼ 0:5 log 1 þ r 1 r ; r ¼ Xj Oj j¼1 D j t fw;kt x n j fw;kt x p j : ð24þ Accordng to [9], n order to ensure that the ensemble algorthm converges, each nput weak RDC model f w; s normalzed by max j fw; ðx p j Þf w;ðx n j Þ,.e., f w; ðþ max j fw; x p j fw; x n j -1 f w; ðþ; ð25þ so that f w; ðx p j Þf w;ðx n j Þ2½1; þ1š. By learnng RDC n an ensemble way, each weak model s learned on a smaller set of data and the fnal dstance functon of the ensemble model s based on the score values of each weak model. Defne N + ðz Þ (N - ðz Þ) as the number of relevant (rrelevant) observatons for query z n the tranng set. Note that the space complexty (memory cost) of creatng all the tranng samples x p and x n s! O XN ¼1 q N + ðz ÞN - ðz Þ ; ð26þ where N - ðz Þ¼N N + ðz Þ-1, q s the number of features to descrbe each data sample. Assumng there are N L mages for each person, we then have N + ðz Þ¼ N L -1. Therefore, to generate each weak RDC model n learnng an ensemble RDC, the space complexty s reduced to Oðq ðð b2 L b L ÞN 3 +ð b 2 L -b2 ÞN 2 ÞÞ, where b s the percentage of all tranng samples used for buldng a weak RDC. 4 After generatng the weak RDCs, the ensemble learnng process tself has a space complexty of OðH ðð 1 L - 1 L ÞN 3 þð 1 2 L 1ÞN 2 ÞÞ, where H s the number of groups (.e., the total number of weak RDC models). As H q, the boostng process has much less memory usage durng tranng. Apart from reducng the space complexty of RDC, ensemble learnng also allevates the local optmum problem of the teratve algorthm proposed to solve the RDC optmzaton problem n Secton 3.2. Note that each RDC model we descrbed above s weak because t s only learned on a small set of tranng data and t may stll suffer from the local optmum problem. As the ensemble learnng theory n [9] ensures the matchng error s mnmzed, the ensemble learnng ntroduced above thus s able to allevate the effect of beng trapped n a local optmum. Our experments show that the Ensemble RDC can generally yeld equal or better performance as compared to the proposed batch mode RDC for large scale computng and s wth reduced memory usage. 5 RELATIONS TO ALTERNATIVE MODELS Gven the RDC model and ts ensemble formulaton, we shall now dscuss the relatons between these models and alternatve models, specfcally rankng models and dstance learnng models. Relatons to exstng rankng models. Our RDC model s a specal rankng model, concerned wth only two ranks,.e., the true match beng ranked hgher than any msmatches. In our early work [28], we nvestgated the use of a rank support vector machne (RankSVM)-based rankng model for person redentfcaton. In partcular, the prmal 4. The value of b s always smaller than 50 percent n our experments under the aforementoned settng of H and B.

8 No commercal use ^_^ 660 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 35, NO. 3, MARCH 2013 RankSVM proposed by Chapelle and Keerth [2] s adopted, whch s more sutable for large-scale learnng compared to a standard RankSVM. The prmal RankSVM ams to solve the followng rankng optmzaton problem: mn w 1 2 kw Xj Oj k2 þ ¼1 max 0; 1 w T x p 2; xn ð27þ where s a postve mportance weght on the rankng performance, and x p and x n are also computed n the absolute data dfference space. Comparng ths optmzaton problem wth the one our RDC model attempts to solve (6), one can note the followng fundamental dfferences between the two models: 1. RDC s able to explore the second-order nformaton extracted from data due to the quadratc formulaton n (4), learnng weghts for not only each ndvdual feature but also the combnaton of each par of features, whle prmal RankSVM only computes the weghts w based on the frst-order nformaton, gnorng the correlatons between features. Ths dfference s due to the dstance learnng formulaton of RDC and the lnear SVM formulaton of prmal RankSVM. 2. Wth the hnge loss functon, prmal RankSVM s essentally a large margn-based optmzaton model due to the offset 1 and mnmzaton of kwk n (27). In contrast, our RDC model enforces a softer constrant by usng logstc functon modelng. Ths enables the RDC model to be more tolerant to large ntra and nterclass varatons and less prone to underfttng gven undersampled data. 3. Dfferng from RDC, there s a free parameter n the cost functon of prmal RankSVM whch determnes the relatve weghtng between the margn functon and the rankng error functon. Determnng the optmal value of s crtcal and can be acheved by cross-valdaton. However, person redentfcaton based on learnng to rank s typcally a large scale learnng problem. Usng cross-valdaton would further ncrease the computatonal cost a lot, makng the model less tractable. Another related rankng model one can consder s RankBoost based on the boostng technque. Comparng RDC to RankBoost [9], the major dfference s that RDC quantfes the jont combnaton of dfferent features rather than quantfyng each feature ndependently. Ths ndvdual local selecton process makes the RankBoost model computatonally much more expensve than ether RDC or RankSVM, as demonstrated by our experments (see Secton 6.6). It s worth pontng out that although boostng technque s also used n our ensemble verson of RDC, the objectve s completely dfferent: We am to combne a handful of weak RDC models together rather than quantfyng features ndvdually and ndependently. Relatons to exstng dstance learnng models. Among varous exstng dstance learnng methods, the methods n [29], [33], and [20] are the most relevant ones to our model as they also explot the dea of relatve dstance comparson. However, there s a fundamental dfference n ther dstance learnng formulaton; that s, n ther models relatve dstance comparson s used as a constrant rather than as part of the cost functon as n the RDC model. In some work, a common form of the constrant n these related models [29], [20] s as follows: x T n Mx n x T p Mx p 1; where x p s the dfference between relevant samples, x n s that of the related rrelevant ones, and M s the dstance matrx. Hence, when those models mnmze the kmk F,ts 1 equvalent to maxmzng the margn kmk between a F relevant par and the correspondng related rrelevant one wth a normalzed dstance matrx fm ¼ M kmk. In [33], the F model explctly mnmzes the ntraclass varaton and maxmzes the nterclass varaton. As a result, these relatve dstance comparson models stll ether mplctly [29], [20] or explctly [33] am to learn a dstance by whch each class becomes more compact whle beng more separable from each other n an absolute sense. In contrast, RDC s only concerned wth the relatve dstance comparson and usng the comparson error tself as ts cost functon. Ths enables a dstance to be learned wth a softer constrant wth the beneft of beng more tolerant to ntra and nterclass varatons and undersamplng. 6 EXPERIMENTS 6.1 Datasets and Settngs Three publcly avalable person redentfcaton datasets, ETHZ [7], -LIDS Multple-Camera Trackng Scenaro (MCTS) [37], [31], and VIPeR [14] were used for evaluaton. The ETHZ dataset was orgnally desgned for person detecton and trackng n mage sequences captured from a movng camera n a busy street scene. Schwartz and Davs [30] converted t nto a person redentfcaton dataset by extractng mages of a set of people selected from the vdeo sequences 5 (.e., those mages of each person were assumed to have been taken from dfferent camera vews). Ths resulted n 146 people and 8,555 mages n total. To make t more realstc to a multcamera setup, we randomly chose sx mages for each person for tranng n the dataset for our experments. The mage sze s normalzed to pxels. The challenges of ths dataset are the llumnaton changes and occlusons on people s appearance whle the vew angle change s small (see Fg. 5). In the -LIDS MCTS dataset, whch was captured ndoor at a busy arport arrval hall, there are 119 people wth a total 476 person mages captured by multple nonoverlappng cameras wth an average of four mages for each person. The mages were normalzed to a sze of pxels. Many of these mages undergo large llumnaton change, consderable vew angle change, and are subject to large occlusons (see Fg. 6). The VIPeR dataset 6 s a person redentfcaton dataset avalable consstng of 632 people captured outdoor wth two mages for each person wth normalzed sze at pxels. Vew angle change 5. The dataset can be downloaded at ~schwartz/datasets.html. 6. The dataset can be downloaded at

9 No commercal use ^_^ ZHENG ET AL.: REIDENTIFICATION BY RELATIVE DISTANCE COMPARISON 661 Fg. 2. Performance comparson usng CMC curves on ETHZ the dataset. was the most sgnfcant cause of appearance change wth most of the matched mage pars contanng one front/back vew and one sde-vew (see Fg. 7). Illumnaton change could also be drastc, but there was lttle occluson. It s noted that these three datasets have dfferent characterstcs (e.g., outdoor/ndoor, large/small varatons n vew angle, presence/absence of occluson) and therefore are deal for evaluatng person redentfcaton algorthms gven dfferent challenges. Among them, the ETHZ dataset s consdered to be the easest one due to the fact that t was not actually captured by multple nonoverlappng vew cameras and thus lack of vew angle change. Note that across the three datasets, the average number of tranng mages of each person ranges from two (VIPeR) to sx (ETHZ), hghlghtng the undersampled class dstrbuton typcal for the person redentfcaton problem. In our experments, we randomly selected all mages of p people (classes) to set up the test set, and the rest of the people (classes) were used for tranng. Dfferent values of p were used to evaluate the matchng performance of models learned wth dfferent amounts of tranng data. Each test set was composed of a gallery set and a probe set. The gallery set conssted of one mage for each person, and the remanng mages were used as the probe set. Ths procedure was repeated 10 tmes. Durng tranng, a par of mages of each person formed a relevant par, and one mage of hm/her and one of another person n the tranng set formed a related rrelevant par, and together they formed the parwse set O defned n Secton 3. For evaluaton, we use the average cumulatve match characterstc (CMC) curves [14] over 10 trals to show the ranked matchng rates. A rank r matchng rate ndcates the percentage of the probe mages wth correct matches found n the top r ranks aganst the p gallery mages. Rank 1 matchng rate s thus the correct matchng/recognton rate. Note that, n practce, although a hgh rank 1 matchng rate s crtcal, the top r ranked matchng rate wth a small r value s also mportant because the top matched mages wll normally be verfed by a human operator [14]. 6.2 Feature Representaton We apply our RDC model as well as other models to an appearance representaton of people captured by a set of dfferent basc features. We start wth a mxture of color and texture hstogram features smlar to those used n [15], [28] and let our model automatcally dscover an optmal feature dstance. Specfcally, we dvded a person mage nto sx horzontal strpes. For each strpe, the RGB, YCbCr, HSV color features, and two types of texture features extracted by Schmd and Gabor flters were computed across dfferent raduses and scales, and n total, 13 Schmd flters and 8 Gabor flters were obtaned. In total, 29 feature channels were constructed for each strpe and each feature channel was represented by a 16D hstogram vector. The detals can be referred to n [15], [28]. Each person mage was thus represented by a feature vector n a 2,784D feature space Z. Snce the features computed for ths representaton nclude low-level features wdely used by exstng person redentfcaton technques, ths representaton s consdered generc and representatve. 6.3 RDC versus Baselne Methods. We frst compared our RDC wth baselne methods, namely nonlearnng based l 1 -norm dstance and Bhattacharyya dstance, whch were used by most exstng person redentfcaton work. Our results (Fgs. 2, 3, and 4, Tables 2, 3, and 4) show clearly that wth the proposed RDC, the matchng performance for all three datasets s mproved sgnfcantly, more so when the tranng set sze ncreases. The mprovement s partcularly dramatc on the VIPeR dataset. In partcular, Table 5 shows that a fourfold ncrease n correct matchng rate (r ¼ 1) s obtaned aganst both l 1 -norm and Bhattacharyya dstances when p ¼ 316. The results valdate the mportance of performng dstance learnng. Examples of matchng people usng RDC for the three datasets are shown n Fgs. 5, 6, and 7 respectvely. 6.4 RDC versus Adaboost and PLS The Adaboost algorthm was formulated n [15] and the partal least squares (PLS) method was proposed n [30]. They are the only learnng-based person redentfcaton methods we are aware of. In our experments, the suggested settngs n [15] and [30] were used. The Adaboost method n [15] s motvated by the observaton that not all features are equally dstnctve and relable for matchng people and ams to learn the weghtng of dfferent features. The proposed RDC algorthm also ams to compute the mportance weght, but t dffers n that 1) RDC performs a rankng-based soft dscrmnant feature selecton whle Adaboot n [15] performs large margnbased dscrmnant selecton; 2) RDC s able to evaluate the mportance of dfferent combnatons of features (second

10 No commercal use ^_^ 662 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 35, NO. 3, MARCH 2013 Fg. 3. Performance comparson usng CMC curves on the -LIDS MCTS dataset. Fg. 4. Performance comparson usng CMC curves on the VIPeR dataset. TABLE 2 Top Ranked Matchng Rate (Percent) on ETHZ p s the sze of the gallery set (larger p means smaller tranng set) and r s the rank. TABLE 3 Top Ranked Matchng Rate (Percent) on -LIDS MCTS p s the sze of the gallery set and r s the rank. order nformaton), whle Adaboost assumes dfferent features are ndependent and selects them ndvdually. As shown n Fgs. 2, 3, and 4, and Tables 2, 3, and 4, our RDC model clearly outperforms the Adaboost-based method n all three datasets. The advantage s partcularly sgnfcant on the more challengng -LIDS and VIPeR datasets. For nstance, for the VIPeR dataset, the rank 1 matchng rate of RDC s twce of that of Adaboost for all three tranng/testng splts. Ths result hghlghts the mportance of quantfyng features globally rather than locally (ndvdually). Although PLS does not quantfy features ndvdually as Adaboost does, t does not perform well for person redentfcaton n our experments. Ths s because PLS s a regresson method and t can only be learned on the gallery dataset. Snce there are only lmted samples per

11 No commercal use ^_^ ZHENG ET AL.: REIDENTIFICATION BY RELATIVE DISTANCE COMPARISON 663 TABLE 4 Top Ranked Matchng Rate (Percent) on VIPeR p s the number of classes n the testng set; r s the rank. TABLE 5 RDC versus Prmal RankSVM (Percent) on ETHZ, -LIDS, and VIPeR Fg. 5. Examples of person redentfcaton on ETHZ usng RDC. In each row, the left-most mage s the probe, mages n the mddle are the top 20 matched gallery mages wth a hghlghted red box for the correctly matched, and the rght-most shows a true match. person for tranng PLS and the people s appearance vares largely, PLS s senstve to the learned data and may not generalze to new data very well. In contrast, our RDC model and the Adaboost model are learned usng an ndependent tranng set consstng of dfferent people from those n the gallery set. Ths not only contrbutes to better performance but also makes the methods more general applcable (.e., applcable even wth only a sngle gallery mage per person). 6.5 RDC versus Related Dstance Learnng Methods We also compared RDC wth four alternatve popular dscrmnant dstance learnng methods, namely, Xng s method [35], LMNN [33], ITM [5], and MCC [13]. Among the four methods, only LMNN explots relatve dstance comparson, but t s used as an optmzaton constrant rather than the man objectve functon, and moreover a hard rather than a soft margn measure s used to quantfy each relatve dstance comparson. MCC s based on Bayesan modelng, but t s not a relatve dstance comparson-based method. Note that snce MCC needs to select the best dmenson for matchng, we performed crossvaldaton by selectng ts value n f½1 :1:10Š;dg, where d s the maxmum rank MCC can learn. Due to the space lmtaton, the standard dervatons of all methods are not shown n the table. In our experments, the standard dervatons of all methods are manly around 2-4 percent, where the proposed RDC s always around 2.5 percent and MCC s always between 3-4 percent. The frst thng we dscovered n our experments was that none of the four models were tractable due to the hgh dmensonalty of the nput data. PCA was thus performed

12 No commercal use ^_^ 664 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 35, NO. 3, MARCH 2013 Fg. 6. Examples of person redentfcaton on -LIDS MCTS usng RDC. Fg. 7. Examples of person redentfcaton on VIPeR usng RDC. to reduce the dmensonalty whle preservng 100 percent of the data. Our results (Fgs. 2, 3, and 4, Tables 2, 3, and 4) clearly show that our model yelds the best rank 1 matchng rate and overall much superor performance compared to the compared models. The advantage of RDC s partcularly apparent when a tranng set s small (learnng becomes more dffcult) and a test set s large ndcated by the value of p (matchng becomes harder). Table 4 shows that on VIPeR when 100 people are used for learnng and 532 people for testng (p ¼ 532), the correct matchng rate for RDC s almost more than doubled aganst any alternatve dstance learnng methods. It s noted that, beneftng from beng a Bayesan modelng, MCC gves the most comparable results to RDC when the tranng set s large. However, ts performance degrades dramatcally when the sze of the tranng data decreases (see columns under p ¼ 120 n Table 2, p ¼ 80 n Table 3, and p ¼ 532 n Table 4). Overall the results suggest that overfttng to undersampled tranng data s the man reason for the nferor performance of the compared alternatve learnng approaches. 6.6 RDC versus Related Rankng Methods We frst compare RDC wth the prmal RankSVM method used n [28]. Dfferent from RDC, RankSVM has a free parameter whch determnes the relatve weghts between the margn functon and the rankng error functon. We cross-valdated the parameter n f0:0001; 0:005; 0:001; 0:05; 0:1; 0:5; 1; 10; 100; 1,000g for prmal RankSVM. As shown n Table 5, the two methods all perform very well compared to non-learnng-based methods and the four dstance learnng-based methods. Our RDC yelds overall better performance, especally at lower rank matchng rate and gven less tranng data over the more challengng -LIDS and VIPeR datasets. The better performance of RDC s manly due to the logstc functon-based modelng that enforces a softer constrant on relatve dstance comparson and explotng second-order rather than frst-order feature quantfcaton. It s dscovered that tunng the free parameter for prmal RankSVM s not a trval task and the performance can be senstve to the tunng especally gven

13 No commercal use ^_^ ZHENG ET AL.: REIDENTIFICATION BY RELATIVE DISTANCE COMPARISON 665 TABLE 6 RDC versus RankBoost (Percent) on ETHZ, -LIDS, and VIPeR TABLE 7 RDC versus Ensemble RDC (Precent) on ETHZ, -LIDS, and VIPeR p s the number of classes n the testng set; r s the rank. undersampled data. Importantly, ths results n more computatonal cost. The tranng of prmal RankSVM took about 2.5 hours for each tral on -LIDS and VIPeR, and about 8 hours for each tral on ETHZ. Hence, learnng prmal RankSVM s costly and could potentally be a serous problem for large-scale learnng (e.g., matchng n a camera network comprsng hundreds of cameras). In contrast, the tranng of our RDC model was at least 10 tmes faster. (See Secton 6.9 for more dscusson on computatonal cost.) In addton, a more advanced development, namely, ensemble RDC, would acheve better performance than RDC n challengng cases. We also compare RDC wth RankBoost [9]. However, t turned out that RankBoost s ntractable for our hghdmensonal feature space (2,784D). Wthout access to specal hardware, RankBoost was only tractable for the smallest tranng dataset settng for all three datasets. The man reason for ths hgh computatonal cost s because RankBoost needs to learn an optmal weak classfer at each teraton, whch has to determne a threshold parameter optmally over a large number of parwse comparson (OðN 3 Þ wth N the number of tranng mages). Table 6 shows the results. It can clearly be seen that Rankboost performs much worse than our RDC. The possble reasons nclude: 1) The weak ranker n RankBoost s too weak based on a sngle feature, and 2) all features are treated ndependently. 6.7 Evaluaton of Ensemble RDC Ensemble RDC s proposed as an extenson to RDC n order to allevate the large scale computaton problem n RDC. Table 7 shows that the ensemble RDC yelds smlar matchng performance to RDC on ETHZ. But on the two more challengng datasets, ensemble RDC outperforms RDC. As expected, the ensemble RDC has much less space complexty than the batch model RDC. For nstance, n the case of p ¼ 316 for VIPeR, ensemble RDC took at most 2G RAM for learnng the weak classfer whle RDC requred at least 10.4G RAM n our experments. The better performance of ensemble RDC s lkely due to the fact that the ensemble learnng process can effectvely allevate the local optmum of the teratve algorthm for optmzng RDC. As we explaned earler, the formulated teratve algorthm n Secton 3.2 may be trapped n a local optmum. Wth the boostng-based learnng, an RDC that s partcularly weak because of beng trapped n a local optmum wll be gven a smaller weght. It thus allevates the local optmum problem. 6.8 Further Evaluatons of RDC In ths secton, we further evaluate the proposed RDC methods n the followng three aspects. Effect of usng logstc functon. We frst evaluate the usefulness of the logstc functon based modelng. Wthout a logstc functon, Crteron (6) becomes mn W r0 ðw; OÞ; s:t: w T w j ¼ 0; 8 6¼ j; where r 0 ðw; OÞ ¼ X O kw T x p k2 kw T x n k2 : ð28þ Ths s smlar to the maxmum margn crteron (MMC) for feature extracton [21], whch we call RDC-MMC n our experments. The performance of RDC-MMC s compared wth RDC n Table 8. The results show that wthout the logstc modelng for dfferentatng the margn n the dfference nformaton from dfferent types, the RDC-MMC model performs much worse for person redentfcaton. Ths hghlghts the mportance of usng a logstc functon for learnng a person redentfcaton model. Effect of learnng n an absolute data dfference space. We have shown n Secton 3.4 that n theory our relatve dstance comparson learnng method can beneft from learnng n an absolute data dfference space. To valdate ths expermentally, we compare RDC wth RDC raw whch learns n the normal data dfference space DZ (see Secton 3.4). The result n Table 9 ndcates that learnng n an absolute data dfference space does mprove the matchng performance. Note that most exstng dstance learnng models are based on learnng n the normal data

14 No commercal use ^_^ 666 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 35, NO. 3, MARCH 2013 TABLE 8 RDC versus RDC-MMC (Percent) on ETHZ, -LIDS, and VIPeR p s the number of classes n the testng set; r s the rank. TABLE 9 Effect of Learnng (Percent) n an Absolute Data Dfference Space TABLE 10 Average Rank of W Learned by RDC dfference space DZ. It s possble to reformulate some of them n order to learn n an absolute data dfference space. In Table 9, we show that when ITM and MCC are learned n the absolute data dfference space jdzj, termed ITM abs and MCC abs, respectvely, ther performances become worse as compared to ther results n Tables 2, 3, and 4. Ths ndcates that the absolute dfferent space s more sutable for our relatve comparson dstance learnng, whch makes the dstance comparson more consstently. 6.9 Computatonal Cost Though RDC s teratve, t has relatvely low cost n practce. In our experments, for VIPeR wth p ¼ 316, t took around 15 mnutes for an Intel dual-core 2.93 GHz CPU and 48 GB RAM server to learn RDC for each tral. We observed that the low cost of RDC s partally due to ts ablty to seek a sutable low rank of W (.e., converge wthn very few teratons), as shown n Table 10. In comparson among the compared other methods, Adaboost was one of the most costly whch took over 7 hours for each tral. The prmal RankSVM took more than 2.5 hours. 7 CONCLUSIONS We have formulated the person redentfcaton as a relatve dstance comparson problem. In partcular, we proposed a relatve dstance comparson model, whch ams to maxmze the lkelhood that a par of true match has a smaller dstance than that of a wrong match par under a soft dscrmnant modelng. An ensemble strategy s also ntroduced to develop ensemble RDC n order to overcome lmtatons n RDC on both space complexty and local mnmum. We have demonstrated that the proposed person redentfcaton models can allevate the bas of large varatons durng optmzaton of learnng smlarty measurement. Our experments valdate that the proposed approach outperforms the related popular person redentfcaton technques and related methods n terms of matchng performance and tractablty. It would be nterestng to nvestgate how nformaton on groups of people can assst person redentfcaton as contextual nformaton. Ths s motvated by the observaton that humans often rely on the people surroundng the target person for dentfcaton f the target s occluded or has undstngushable appearance. Ths contextual nformaton s useful n certan publc spaces such as the -LIDS arport arrval scene where people typcally walk wth the same group of people even when they do not know each other, as demonstrated n our prevous work [37]. However, how to automatcally detect a group of people n practcal scenaros s stll an open problem whch needs to be solved n order to utlze nformaton of group of people as contextual nformaton for person redentfcaton. Also, groups of people may merge, splt, or undergo occluson, and all these ssues may affect the use of group nformaton for helpng person redentfcaton on target people. Hence, we consder that the key problem s on explorng the most relable and robust features for group representaton based on technques such as context quantfcaton [39]. It s worth pontng out although our RDC model s formulated specfcally for addressng the person redentfcaton, t can be appled to solve other pattern recognton problems. In partcular, there are other vson problems that share smlar characterstcs as person redentfcaton,.e., large ntra and nterclass varatons, large number of classes wth few samples per class. Such problems nclude gat

15 No commercal use ^_^ ZHENG ET AL.: REIDENTIFICATION BY RELATIVE DISTANCE COMPARISON 667 recognton and large scale object recognton where there exsts a large number of rare classes, each contanng only a handful of samples. Extendng RDC to address other vson problems s part of our ongong work. Fnally, n the current work, no attempt has been made to remove the background nformaton from a person mage whch could typcally have an negatve effect on the performance of person redentfcaton. The dea was to rely on the proposed feature quantfcaton technque to select the best features n order to elmnate the negatve effect of background nformaton. Nevertheless, t wll be nterestng to ntegrate an explct background segmentaton step nto the proposed framework n the future. ACKNOWLEDGMENTS Ths research was partally funded by the EU FP7 project SAMURAI wth grant no We-Sh Zheng was addtonally supported by the Natonal Natural Scence of Foundaton of Chna (No ), the NSFC-GuangDong (No. U , U ), Specalzed Research Fund for the Doctoral Program of Hgher Educaton (No ), the Natural Scence Foundaton of Guangdong Provnce (No. S ), and the Fundamental Research Funds for the Central Unverstes (No. 12lgpy28, ) for ths work. REFERENCES [1] S. Bak, E. Corvee, F. Brémond, and M. Thonnat, Person Re- Identfcaton Usng Spatal Covarance Regons of Human Body Parts, Proc. IEEE Int l Conf. Advanced Vdeo and Sgnal Based Survellance, pp , [2] O. Chapelle and S.S. Keerth, Effcent Algorthms for Rankng wth SVMS, Informaton Retreval, vol. 13, pp , June [3] K. Chen, C. La, Y. Hung, and C. Chen, An Adaptve Learnng Method for Target Trackng across Multple Cameras, Proc. IEEE Conf. Computer Vson and Pattern Recognton, [4] D. Cheng, M. Crstan, M. Stoppa, L. Bazzan, and V. Murno, Custom Pctoral Structures for Re-Identfcaton, Proc. Brtsh Machne Vson Conf., [5] J. Davs, B. Kuls, P. Jan, S. Sra, and I. Dhllon, Informaton- Theoretc Metrc Learnng, Proc. Int l Conf. Machne Learnng, [6] P. Dollar, Z. Tu, H. Tao, and S. Belonge, Feature Mnng for Image Classfcaton, Proc. IEEE Conf. Computer Vson and Pattern Recognton, [7] A. Ess, B. Lebe, and L. Van Gool, Depth and Appearance for Moble Scene Analyss, Proc. IEEE Int l Conf. Computer Vson, [8] M. Farenzena, L. Bazzan, A. Perna, M. Crstan, and V. Murno, Person Re-Identfcaton by Symmetry-Drven Accumulaton of Local Features, Proc. IEEE Conf. Computer Vson and Pattern Recognton, [9] Y. Freund, R. Iyer, R.E. Schapre, and Y. Snger, An Effcent Boostng Algorthm for Combnng Preferences, J. Machne Learnng Research, no. 4, pp , [10] N. Ghessar, T. Sebastan, and R. Hartley, Person Redentfcaton Usng Spatotemporal Appearance, Proc. IEEE Conf. Computer Vson and Pattern Recognton, [11] A. Glbert and R. Bowden, Trackng Objects Across Cameras by Incrementally Learnng Inter-Camera Colour Calbraton and Patterns of Actvty, Proc. European Conf. Computer Vson, [12] A. Glbert and R. Bowden, Incremental, Scalable Trackng of Objects Inter Camera, Computer Vson and Image Understandng, vol. 111, no. 1, pp , [13] A. Globerson and S. Rowes, Metrc Learnng by Collapsng Classes, Proc. Advances n Neural Informaton Processng Systems, [14] D. Gray, S. Brennan, and H. Tao, Evaluatng Appearance Models for Recognton Reacquston, and Trackng, Proc. IEEE Int l Workshop Performance Evaluaton of Trackng and Survellance, [15] D. Gray and H. Tao, Vewpont Invarant Pedestran Recognton wth an Ensemble of Localzed Features, Proc. European Conf. Computer Vson, [16] R. Herbrch, T. Graepel, and K. Obermayer, Large Margn Rank Boundares for Ordnal Regresson, Proc. Advances n Neural Informaton Processng Systems, pp , [17] W. Hu, M. Hu, X. Zhou, J. Lou, T. Tan, and S. Maybank, Prncpal Axs-Based Correspondence between Multple Cameras for People Trackng, IEEE Trans. Pattern Analyss and Machne Intellgence, vol. 28, no. 4, pp , Apr [18] O. Javed, K. Shafque, Z. Rasheed, and M. Shah, Modelng Inter- Camera Space-Tme and Appearance Relatonshps for Trackng across Non-Overlappng Vews, Computer Vson and Image Understandng, vol. 109, no. 2, pp , [19] O. Javed, K. Shafque, and M. Shah, Appearance Modelng for Trackng n Multple Non-Overlappng Cameras, Proc. IEEE Conf. Computer Vson and Pattern Recognton, [20] J. Lee, R. Jn, and A. Jan, Rank-Based Dstance Metrc Learnng: An Applcaton to Image Retreval, Proc. IEEE Conf. Computer Vson and Pattern Recognton, [21] H. L, T. Jang, and K. Zhang, Effcent and Robust Feature Extracton by Maxmum Margn Crteron, IEEE Trans. Neural Networks, vol. 17, no. 1, pp , Jan [22] G. Lan, J. La, and W.-S. Zheng, Spatal-Temporal Consstent Labelng of Tracked Pedestrans across Non-Overlappng Camera Vews, Pattern Recognton, vol. 44, no. 5, pp , [23] C.-J. Ln, Projected Gradent Methods for Nonnegatve Matrx Factorzaton, Neural Computaton, vol. 19, no. 10, pp , [24] C. Loy, T. Xang, and S. Gong, Tme-Delayed Correlaton Analyss for Mult-Camera Actvty Understandng, Int l J. Computer Vson, vol. 90, no. 1, pp , [25] D. Makrs, T. Ells, and J. Black, Brdgng the Gaps between Cameras, Proc. IEEE Conf. Computer Vson and Pattern Recognton, [26] U. Park, A. Jan, I. Ktahara, K. Kogure, and N. Hagta, Vse: Vsual Search Engne Usng Multple Networked Cameras, Proc. Int l Conf. Pattern Recognton, [27] B. Prosser, S. Gong, and T. Xang, Mult-Camera Matchng under Illumnaton Change over Tme, Proc. ECCV Workshop Mult- Camera and Mult-Modal Sensor Fuson Algorthms and Applcatons, [28] B. Prosser, W.-S. Zheng, S. Gong, and T. Xang, Person Re- Identfcaton by Support Vector Rankng, Proc. Brtsh Machne Vson Conf., [29] M. Schultz and T. Joachms, Learnng a Dstance Metrc from Relatve Comparsons, Proc. Advances n Neural Informaton Processng Systems, [30] W. Schwartz and L. Davs, Learnng Dscrmnatve Appearance- Based Models Usng Partal Least Squares, Proc. Brazlan Symp. Computer Graphcs and Image Processng, [31] UK, Home Offce -LIDS Multple Camera Trackng Scenaro Defnton, [32] X. Wang, G. Doretto, T. Sebastan, J. Rttscher, and P. Tu, Shape and Appearance Context Modelng, Proc. IEEE Int l Conf. Computer Vson, [33] K. Wenberger, J. Bltzer, and L. Saul, Dstance Metrc Learnng for Large Margn Nearest Neghbor Classfcaton, Proc. Advances n Neural Informaton Processng Systems, [34] S. Xang, F. Ne, and C. Zhang, Learnng a Mahalanobs Dstance Metrc for Data Clusterng and Classfcaton, Pattern Recognton, vol. 41, no. 12, pp , [35] E. Xng, A. Ng, M. Jordan, and S. Russell, Dstance Metrc Learnng wth Applcaton to Clusterng wth Sde-Informaton, Proc. Advances n Neural Informaton Processng Systems, [36] L. Yang, R. Jn, R. Sukthankar, and Y. Lu, An Effcent Algorthm for Local Dstance Metrc Learnng, Proc. 21st Nat l Conf. Artfcal Intellgence, pp , [37] W.-S. Zheng, S. Gong, and T. Xang, Assocatng Groups of People, Proc. Brtsh Machne Vson Conf., [38] W.-S. Zheng, S. Gong, and T. Xang, Person Re-Identfcaton by Probablstc Relatve Dstance Comparson, Proc. IEEE Conf. Computer Vson and Pattern Recognton, 2011.

16 No commercal use ^_^ 668 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 35, NO. 3, MARCH 2013 [39] W.-S. Zheng, S. Gong, and T. Xang, Quantfyng and Transferrng Contextual Informaton n Object Detecton, IEEE Trans. Pattern Analyss and Machne Intellgence, vol. 34, no. 4, pp , Apr We-Sh Zheng receved the PhD degree n appled mathematcs from Sun Yat-Sen Unversty n 2008, and has been a postdoctoral researcher on the EU FP7 SAMURAI Project at Queen Mary Unversty London. He joned Sun Yat-sen Unversty under the one-hundredpeople program n Hs research nterests nclude object assocaton and categorzaton n vsual survellance. He s a member of the IEEE. Tao Xang receved the PhD degree n electrcal and computer engneerng from the Natonal Unversty of Sngapore n He s a senor lecturer (assocate professor) at Queen Mary Unversty London. Hs research nterests nclude computer vson, statstcal learnng, vdeo processng, and machne learnng, wth focus on nterpretng and understandng human behavor.. For more nformaton on ths or any other computng topc, please vst our Dgtal Lbrary at Shaogang Gong receved the DPhl degree n computer vson from Keble College, Oxford Unversty n He s a professor of vsual computaton at Queen Mary Unversty London and s a fellow of the Insttuton of Electrcal Engneers and the Brtsh Computer Socety. Hs work focuses on moton and vdeo analyss; object detecton, trackng, and recognton; face and expresson recognton; gesture and acton recognton; vsual behavor recognton.