Wildfire Smoke Detection Using SpatioTemporal Bag-of-Features of Smoke

Transcription

1 Wldfre Smoke Detecton Usng SpatoTemporal Bag-of-Features of Smoke JunOh Park, ByoungChul Ko, Jae-Yeal Nam Dept. of Computer Engneerng Kemyung Unversty Daegu, Korea wnsdhd, nceko, Abstract Ths paper presents a wldfre smoke detecton method based on a spatotemporal bag-of-features (BoF) and a random forest classfer. Frst, canddate blocks are detected usng key-frame dfferences and non-parametrc color models to reduce the computaton tme. Subsequently, spatotemporal three-dmensonal (3D) volumes are bult by combnng the canddate blocks n the current key-frame and the correspondng blocks n prevous frames. A hstogram of gradent (HOG) s extracted as a spatal feature, and a hstogram of optcal flow (HOF) s extracted as a temporal feature based on the fact that the dffuson drecton of smoke s upward owng to thermal convecton. Usng these spatotemporal features, a codebook and a BoF hstogram are generated from tranng data. For smoke verfcaton, a random forest classfer s bult durng the tranng phase by usng the BoF hstogram. The random forest wth BoF hstogram can ncrease the detecton accuracy and allow smoke detecton to be carred out n near real-tme. 1. Introducton Wldfres rank among varous natural dsasters such as storms, droughts, floods, landsldes, and tsunams, resultng n not only extensve loss of lfe and property but also ecologcal problems. Therefore, an early warnng of wldfres s crucal to reduce the potental for casualtes and property damage. Recently, wth the rapd development of nformaton technology, automatc wldfre detecton has grown to become a new research feld. The conventonal approach has been to watch for a sgn of fre or smoke wth the naked eye. To overcome the lmtatons of tradtonal methods, nfrared (IR) sensors and lght detecton and rangng (LIDAR) systems are used to dentfy the heat flux of frelght and to measure the laser lght backscattered by smoke partcles. However, these optcal sensor-based wldfre detecton methods generate many false alarms due to atmospherc condtons, lght reflectons, and the vast dstance between the sensors and the burnng pont [1]. SooYeong Kwak Dept. Electronc and Control Engneerng, Hanbat Natonal Unversty Daejeon Korea sykwak@hanbat.ac.kr In general, a charge-coupled devce (CCD) camera for a wldfre detecton system s nstalled atop a mountan to montor a wde area and to provde more dependable fre detecton results than sensor-based methods: The equpment and management costs are lower. One camera covers a wde range. The response tme for fre and smoke detecton s shorter. The manager can confrm a fre and montor the status of the fre and smoke wthout vstng the locaton. However, methods based on CCD cameras stll have problems such as envronmental llumnaton, varatons n smoke color tones, and low qualty of mages of a wde outdoor area [2]. Wldfre detecton can be dvded nto two research categores: smoke detecton and flame detecton. Smoke detecton s partcularly mportant for early warnng systems, because smoke usually rses before flames arse. Therefore, our research focuses on daytme wldfre smoke detecton. The man smoke detecton methods are summarzed as follows: Krstnć et al. [2] proposed a pxel-level mage analyss and segmentaton of smoke-colored pxels for automatc forest fre detecton. They proved that good color space canddates for smoke detecton nclude HIS and ts dervatves as well as the RGB color space. They also proposed a performance evaluaton method to fnd an effcent combnaton of a color space and a pxel-level smoke segmentaton algorthm. Töreyn and Cetn [3] proposed a part-based smoke detecton algorthm usng four sub-algorthms: (1) slow-movng vdeo object detecton, (2) gray regon detecton, (3) rsng object detecton, and (4) shadow elmnaton. These four sub-algorthms detect the presence of smoke ndvdually, and the decsons of the sub-algorthms are combned by an adaptve weghted majorty algorthm. Ham et al. [4] proposed a wldfre smoke detecton algorthm usng fuzzy fnte automata (FFA) and temporal vsual features. For smoke verfcaton usng FFA, three two-dmensonal probablty densty functons are estmated from nformaton on ntensty, wavelet energy, and moton /12/$ /13/$ IEEE 200

2 Habboglu et al. [5] used background subtracton and color thresholds to fnd the smoke-colored slow-movng regons n vdeo. Canddate regons are dvded nto spatotemporal blocks and correlaton features are extracted from the blocks. Then, a bnary support vector machne (SVM) classfer wth spatotemporal correlaton descrptors s used to classfy smoke-colored and non-smoke-colored objects. Ko et al. [6] extracted spatotemporal characterstcs such as color, wavelet coeffcents, moton orentaton, and hstogram of orented gradents (HOG) from canddate smoke blocks and the precedng 100 assocated frames. The man dea of ths method s to tran two ndvdual random forests usng ndependent temporal and spatal feature vectors. Fnally, a canddate block s declared a smoke block f the average probablty of two random forests n the smoke class s maxmum. In ths study, we ntroduce a wldfre smoke detecton method based on a spatotemporal bag-of-features (BoF) and a random forest classfer. Frst, we detect canddate blocks from key-frames n vdeo and prepare spatotemporal 3D volumes by combnng the canddate blocks n the current key-frame wth the correspondng blocks n prevous frames. An HOG s extracted from the current block as a spatal feature, and a hstogram of optcal flow (HOF) s extracted as a temporal feature based on the fact that the dffuson drecton of smoke s upward owng to thermal convecton. Second, the random forest classfer, whch s an ensemble of decson trees, s bult durng the tranng phase by usng the BoF hstogram. The random forest wth the BoF hstogram can ncrease the detecton accuracy and allow smoke detecton to be carred out n near real-tme. The remander of ths paper s organzed as follows. Secton 2 descrbes the canddate smoke block detecton algorthm as the preprocessng step. Secton 3 ntroduces our smoke verfcaton method usng a spatotemporal BoF and random forests. Secton 4 presents an expermental evaluaton of the accuracy and applcablty of our proposed wldfre smoke detecton method. Secton 5 presents our conclusons and dscusses scope for future work. 2. Canddate smoke block detecton For real-tme processng, t s effcent to use canddate regons wthout usng every mage pxel. Therefore, we frst detect slow-movng regons by key-frame detecton, and then we confrm the canddate regons by usng a smoke color model. The nput vdeo sequences are dvded nto blocks accordng to the aspect rato n the MPEG standardzaton, and all subsequent procedures are appled to these unts. One of the man characterstcs of wldfre smoke s the relatvely low apparent spreadng speed, as the survellance cameras are nstalled at vast dstances. Thus, canddate smoke regons cannot be detected usng only a smple background subtracton model. To overcome the lmtatons of smple background subtracton, we select key-frames from a vdeo sequence used n [6] whenever the frame dfference exceeds a certan threshold. After detectng key-frames and ther movng blocks, the non-smoke-colored blocks must be fltered out to reduce the computaton tme for smoke verfcaton. In general, the smoke color s dstrbuted wdely n the RGB color space dependng on the burnng materal. On the other hand, smoke color n the HSI color space s dstrbuted wth a low level of saturaton (S), a hgh level of ntensty (I), and no hue (H) [4]. Thus, we remove non-smoke-colored blocks usng the probablty densty functon (PDF) of a smoke color model. The parameters of the PDFs are learned from tranng data. In contrast to typcal supervsed learnng n whch the underlyng dstrbuton functons are known, smoke color does not follow a known dstrbuton or parametrc denstes [8]. Therefore, we construct the PDF to have multmodal denstes, rather than unmodal denstes, by usng a non-parametrc method. In ths paper, we use a smoother kernel functon n the form of a Gaussan n order to obtan the followng smoother kernel densty model: 2 N 1 1 x x n p( x) d exp (1) N n1 (2h ) 2h where N s the number of data and h s the parameter that determnes the wdth of the effectve Gaussan wndow along each dmenson d. After PDF are generated, the lkelhood of the feature vector of block b s denoted by p( b Smoke), and can be estmated from the defned PDFs. These PDF are then used to confgure real canddate blocks. From the PDF, the lkelhood p( b Smoke) that a block ( b ) has the smoke color s obtaned by summng the probabltes for all pxels n b usng the followng formula: n 1 f p( b j Smoke ) T1 b (2) j1 0 otherwse If the result exceeds a pre-defned threshold, the block b s declared a canddate smoke block. Fgure 1 shows canddate smoke blocks after key-frame dfferencng and flterng of non-smoke-colored blocks. After detectng the canddate smoke blocks, the mage s 201

3 scanned to group the blocks nto clusters based on block connectvty. Once all of the N clusters have been determned, some clusters are removed f the number of blocks s less than three. 3. Smoke verfcaton usng spatotemporal BoF and random forest Ths secton presents our smoke verfcaton method usng spatotemporal features such as HOGs, HOFs, and bag-of-features. Fnally, a feature hstogram s estmated from the BoF, and the random forest classfer uses the feature hstogram to determne whether the canddate block represents real smoke Spatotemporal BoF In vdeo-based acton recognton, sparse spatotemporal features have recently shown good performance [9, 10, 11]. Fgure 1: Examples of canddate blocks detected by key-frame dfferencng and flterng of non-smoke-colored blocks. In ths study, we use canddate blocks that are selected from prevous steps nstead of nterest ponts. In a manner smlar to that for acton recognton, we frst prepare spatotemporal 3D volumes by combnng the canddate blocks wth t correspondng blocks n prevous frames, as shown n Fgure 2 (a). Each volume ( x, y, t ) has the same wdth and heght, wth canddate blocks (10, 10), and the tme duraton t s 100. From each volume, we compute the moton and appearance of local spatotemporal features as shown n Fgure 2 (b). For a spatal feature, an HOG s generated from the current block. Snce the dffuson drecton of smoke s upward owng to thermal convecton, the gradent dstrbuton of a smoke boundary has a dstngushable pattern. By usng ths characterstc, we extract an HOG as a spatal feature. In order to extract an HOG from a canddate block, gradent orentatons are estmated at each pxel and a hstogram of each orentaton n a canddate block s calculated n a manner smlar to that n [6]. For a temporal feature, an HOF s generated from 100 blocks wthn the same volume. Snce smoke usually drfts contnually upwards due to hot arflows, the orentaton of the moton s estmated from the 100 blocks n each volume. Then, two features are normalzed and concatenated nto one. After extractng a set of spatotemporal features from tranng data, we construct a spatotemporal BoF as shown n Fgures 2 (c) and (d). A BoF s desgned to depct each mage as an orderless collecton of local features. Vsual vocabulares (codebooks) are bult to descrbe a BoF usng tranng descrptors and k-means clusterng. Each cluster s treated as a vsual word (codeword) n the vsual vocabulary. By mappng the local features of an mage to the vsual vocabulary, we can descrbe the mage as a feature hstogram accordng to the presence (or count) of each vsual word [12]. Gven a random subset of volumes ncludng smoke and non-smoke regons from the tranng set, we learn the vsual vocabulary by performng k-means clusterng. In our experments, the sze of the vsual vocabulary was determned as k = 400. Fgure 2 (c) shows the process of codebook generaton Generaton of BoF hstogram Once the codebook of the BoF s bult, two knds of BoF hstograms of vsual words should be estmated usng the smoke and non-smoke classes. The BoF hstogram assgns each feature to the closest vsual word and computes the hstogram of vsual word occurrences over a space-tme volume [13]. To assgn a weght to the hstogram, bnary weghtng ndcates the presence or absence of a vsual word by values 1 or 0, respectvely. However, we adopt the soft-weghtng approach [14] to emphasze the sgnfcance of vsual words: nstead of searchng only for the nearest vsual word, we select the N nearest vsual words and assgn dfferent weghts accordng to the sum of dstances. Fgure 2 (d) shows an example of a BoF feature hstogram Smoke verfcaton usng random forest classfer In contrast to prevous heurstc methods and smple pattern classfers for smoke verfcaton, ths study uses a random forest classfer [15] to determne whether the canddate smoke clusters represent real smoke. A random forest s a decson tree ensemble classfer, wth each tree grown usng some types of randomzaton. Random forests have a capacty for processng large amounts of data wth hgh tranng speeds, based on a decson tree. The structure of each tree n the random forest s bnary and s created n a top down manner, as shown n Fgure 2 (e). In the tranng procedure, the random forest s started by choosng a random subset I from the local BoF tranng 202

4 Fgure 2: Flow dagram of BoF generaton and random forest learnng: (a) 3D volume generaton from canddate blocks, (b) spatotemporal feature extracton, (c) codebook generaton, (d) BoF hstogram generaton, and (e) random forest generaton. data I. At node n, the tranng data I n s teratvely splt nto left and rght subsets I and I by usng Equaton (3) l r wth the threshold t and splt functon f ( v ) for the feature vector v. The threshold t s randomly chosen n the range t mn f ( v ),max f ( v )) by the splt functon f v ) ( [6]. Il { In f ( v) t}, (3) Ir In \ Il. The overall procedure for tranng one decson tree of a random forest s summarzed below. 1. Choose the maxmum tree depth. 2. Grow each tree for an ndvdual random forest by the followng steps: a. Select n new bootstrap samples from the tranng set S n and grow an unpruned tree usng these n bootstrap samples. b. For each node, randomly select m varables at each nternal node and determne the best splt functon usng only these varables. c. Grow the tree wthn the maxmum tree depth. After the ndvdual decson trees are traned, the ensemble of trees s assembled nto a random forest classfer as shown n Fgure 2 (e). The number of trees T s set to 100, whch has been shown emprcally to yeld good results and computaton tmes comparable wth those for related methods. After the random forest classfer has been learned, the BoF hstogram of the test blocks s created and s used as nput to the traned random forests. The fnal class dstrbuton s generated by arthmetc averagng of each ( dstrbuton of all trees, L = ( l 1, l 2,, l T ), usng Equaton (4). In Equaton (4), T s the number of trees, and we choose c as the fnal class of an nput mage f p( c L) has the maxmum value. T 1 P( c L) P( c lt) (4) T t1 4. Experment We performed experments usng the KMU Fre & Smoke Database ( whch ncludes 38 dverse fre vdeos, such as Indoor-outdoor flame, Indoor-outdoor smoke, Wldfre smoke, and Smoke & fre lke movng object. The frame rates of the vdeo data vared from 15 to 30 Hz, whle the sze of each nput mage was pxels. To perform the tranng, 517 blocks were randomly selected from ten vdeos: 130 blocks of dense smoke, 131 blocks of tenuous smoke, 120 blocks of tenuous smoke-colored clouds and fog, and 136 blocks of dense smoke-lke clouds and fog. The tranng vdeo sequences conssted of fve vdeos that ncluded wldfre smoke and fve vdeos that ncluded smoke-colored clouds and fog. In ths study, we used three crtera for performance evaluaton: average true postve rate (ATPR), average false postve rate (AFPR), and average mss rate (AMR). We compared the performance of the proposed algorthm wth that of prevous algorthms. Among the exstng methods, those of Töreyn and Cetn [3] and Ham et al. [4] 203

5 Fgure 3: Comparson between results of proposed method and two related methods performed well usng the same ten test vdeos contanng fve smoke samples and fve smoke-colored samples. Fgure 3 shows the comparson of the results of the three methods. Note that the proposed method outperformed the methods of Ko et al., Töreyn et al. and Ham et al., wth an ATPR of 95.1% versus 94.2%, 75.6% and 81.0%, an AFPR of 2.3% versus 6.3%, 7.5% and 14.5%, and an AMR of 4.1% versus 1.5%, 16.9% and 4.5%, respectvely. In partcular, the method of Töreyn and Cetn produced a hgher AMR, snce t detected canddate smoke regons n every frame by usng the frame dfference. However, snce wldfre smoke, especally that n vdeos 1, 3, and 4, appears to move very slowly due to the vast dstances between the camera and the locatons of the smoke, many true smoke regons were mssed. The method of Ham et al. exhbted hgher AFPR of 14.5% for test vdeos. The man reason for the hgher error was that smoke was confused wth movng smoke-colored objects and swayng trees. Smlarly, the method of Töreyn and Cetn gave several false alarms for vdeo 6 because movng clouds were confused wth fre-smoke. However, by usng spatotemporal features wth a random forest classfer, our method reduced the false alarms caused by movements of smoke-colored clouds. Moreover, snce a BoF represents vdeos as a collecton of local propertes calculated from a set of 3D volumes, our method can dstngush real smoke from a smoke-colored cloud n dynamc moton. Fgure 4 shows the results of smoke detecton obtaned wth our proposed method usng a set of ten tests. As shown n Fgure 4, our proposed method detected smoke correctly and elmnated false alarms for test vdeos contanng smoke and non-smoke objects wth dfferent speeds and colors. Fgure 4: Smoke detecton results obtaned usng the proposed algorthm 5. Concluson Ths paper ntroduced a wldfre smoke detecton method based on a spatotemporal BoF and a random forest classfer. After canddate blocks are detected, we frst prepare the spatotemporal 3D volumes by combnng the canddate blocks wth t correspondng blocks n prevous frames. An HOG s extracted from the current block as a spatal feature, and an HOF s extracted as a temporal feature based on the fact that the dffuson drecton of smoke s upward owng to thermal convecton. For classfcaton, the random forests that comprse the ensemble of decson trees are bult durng the tranng phase by usng the BoF hstogram. Compared wth related methods, our algorthm can ndeed provde mproved smoke detecton performance. In future work, n order to mprove the detecton performance especally reduce the AMR, we shall optmze the parameters of the BoF and random forest, such as the codebook sze, 3D volume length, tree depth, and number of trees. Acknowledgement Ths research was partally supported by Basc Scence Research Program through the Natonal Research Foundaton of Korea (NRF) funded by the Mnstry of Educaton, Scence and Technology ( ) and t was also fnancally supported by the Mnstry of Educaton, Scence Technology (MEST) and Natonal Research Foundaton of Korea(NRF) through the Human Resource 204

6 Tranng Project for Regonal Innovaton ( A ) References [1] B. C. Ko and S. Y. Kwak. A survey of computer vson-based natural dsaster warnng systems. Optcal Engneerng, 57(1): , [2] D. Krstnć, D. Stpančev, and T. Jakovčevć. Hstogram-based smoke segmentaton n forest fre detecton system. Informaton Technology and Control, 38(3): , [3] B. U. Töreyn and A. E. Cetn. Wldfre detecton usng LMS based actve learnng. IEEE Internatonal Conference on Acoustcs, Speech and Sgnal Processng, , [4] S. J. Ham, B. C. Ko, and J. Y. Nam. Vson based forest smoke detecton usng analyzng of temporal patterns of smoke and ther probablty models. Image Processng: Machne Vson Applcatons IV, 7877:1 6, [5] Y. H. Habboglu, O. Gunay, and A. E. Cetn. Real-tme wldfre detecton usng correlaton descrptors. 19th European Sgnal Processng Conference, , [6] B. C. Ko, J. Y. Kwak, and J. Y. Nam. Wldfre smoke detecton usng temporal-spatal features and random forest classfers. Optcal Engneerng, 51(1): , [7] Q. Tan and S. Zhang. Descrptve vsual words and vsual phrases for mage applcatons. ACM Multmeda, 19 24, [8] B. C. Ko, J. W. Gm, and J. Y. Nam. Automatc whte blood cell segmentaton usng stepwse mergng rules and gradent vector flow snake. Mcron, 42: , [9] M. Marszalek, C. Schmd, and B. Rozenfeld. Learnng realstc human actons from moves. IEEE Conference on Computer Vson and Pattern Recognton, 1 8, [10] P. Dollar, V. Rabaud, G. Cottrell, and S. Belonge. Behavor recognton va sparse spato-temporal features. IEEE Internatonal Workshop on Vsual Survellance and Performance Evaluaton of Trackng and Survellance, 65 72, [11] J. C. Nebles, H. Wang, and L. Fe-Fe. Unsupervsed learnng of human acton categores usng spato-temporal words. Brtsh Machne Vson Conference, 1 11, [12] Y. G. Jang C. W. Ngo, and J. Yang. Towards optmal bag-of-features for object categorzaton and semantc vdeo retreval. 6th ACM Internatonal Conference on Image and Vdeo Retreval, , [13] I. Laptev, M. Marszalek, C. Schmd, and B. Rozenfeld. Learnng realstc human actons from moves. IEEE Internatonal Conference on Computer Vson and Pattern Recognton, 1 8, [14] A. Agarwal and B. Trggs. Hyperfetures-multlevel local codng for vsual recognton. IEEE Internatonal Conference on Computer Vson and Pattern Recognton, 30 43, [15] L. Breman. Random forests. Machne Learnng, 45:5 32, [16] Actual Author Name..". The frobncatable foo flter, Face and Gesture apper ID 324. [17] Authors. Frobncaton tutoral, Some URL al tr.pdf. 205