Smar Surveillance: Applicaions, Technoloes and Implicaions Arun Hampapur, Lisa Brown, Jonahan Connell, Shara Pankani, Andrew Senior and Yingli Tian. Absrac Smar surveillance, is he use of auomaic video analysis echnoloes in video surveillance applicaions. This paper aemps o answer a number of quesions abou smar surveillance: Wha are he applicaions of smar surveillance? Wha are he sysem archiecures for smar surveillance? Wha are he key echnoloes? Wha are he some of he key echnical challenges? and Wha are he implicaions of smar surveillance, boh o securiy and privacy? 1. Inroducion Recen world evens have creaed a shif in he securiy paradigm from "invesigaion of incidens" o "prevenion of poenially caasrophic incidens". Exising dial video surveillance sysems provide he infrasrucure only o capure, sore and disribue video, while leaving he ask of hrea deecion exclusively o human operaors. Human monioring of surveillance video is a very labor-inensive ask. I is generally agreed ha waching video feeds requires a higher level of visual aenion han mos every day asks. Specifically vilance, he abiliy o hold aenion and o reac o rarely occurring evens, is exremely demanding and prone o error due o lapses in aenion [12]. One of he conclusions of a recen sudy by he US Naional Insiue of Jusice [6], ino he effeciveness of human monioring of surveillance video, is as follows These sudies demonsraed ha such a ask[..manually deecing evens in surveillance video], even when assigned o a person who is dedicaed and well-inenioned, will no suppor an effecive securiy sysem. Afer only 20 minues of waching and evaluaing monior screens, he aenion of mos individuals has degeneraed o well below accepable levels. Monioring video screens is boh boring and mesmerizing. There are no inellecually engang simuli, such as when waching a elevision program. Clearly oday s video surveillance sysems while providing he basic funcionaliy fall shor of providing he level of informaion need o change he securiy paradigm from invesigaion o preempion. Auomaic visual analysis echnoloes can move oday's video surveillance sysems from he invesigaive o prevenive paradigm. Smar Surveillance Sysems provide a number of advanages over radiional video surveillance sysems, including "he abiliy o preemp incidens -- hrough real ime alarms for suspicious behaviors "enhanced forensic capabiliies -- hrough conen based video rerieval IBM T.J. Wason Research Cener, 19 Skyline Drive, Hawhorne, NY 10532 arunh@us.ibm.com siuaional awareness hrough join awareness of locaion, ideniy and aciviy of objecs in he moniored space. Secion 2 provides a shor inroducion o various applicaions of smar surveillance sysems. Secion 3 discusses he archiecures for smar surveillance sysems. Secion 4 presens he key echnoloes for smar surveillance sysems. Secion 5 briefly discusses he challenges in smar surveillance. Secion 6 discusses he implicaions of smar surveillance echnoloes. Conclusions are presened in secion 7. 2. Applicaions of Smar Surveillance In his secion we describe a few applicaions of smar surveillance echnology. In his secion, we describe a few applicaions. We group he applicaions ino hree broad caegories, real ime alers, auomaic forensic video rerieval, and siuaion awareness. 2.1 Real Time Alers: There are wo ypes of alers ha can be generaed by a smar surveillance sysem, user defined alers and auomaic unusual aciviy alers. 2.1.1 User Defined Alers: Here he sysem is required o recognize a variey of user defined evens ha occur in he moniored space and noify he user in real ime, hus providing he user wih an opporuniy o evaluae he siuaion and ake prevenive acion if necessary. Following are some ypical evens. 1. Generic Alers: These are alers which depend solely on he movemen properies of objecs wihin he moniored space. Following are a few common examples. 1. Moion Deecion: This aler deecs movemen of any objec wihin a specified zone. 2. Moion Characerisic Deecion: These alers deec a variey of moion properies of objecs, including specific direcion of objec movemen (enry hrough exi lane), objec velociy bounds checking (objec moving oo fas). 3. Abandoned Objec Aler: This deecs objecs which are abandoned, e.g., a piece of unaended baggage in an airpor, or a car parked in oading zone. 4. Objec Removal: This deecs movemens of a user-specified objec ha is no expeced o move, for example, a paining in a museum. 2. Class Specific Alers: These are alers which use he ype of objec in addiion o he objec s movemen properies. Following are a few common examples. 1. Type Specific Movemen Deecion: Consider a camera ha is monioring runways a an airpor. In such a scene, he sysem could provide an aler 1
d i d i on he presence or movemen of people on he armac bu no hose of aircrafs. 2. Saisics: Example applicaions include, alers based on people couns (e.g., more han one person in securiy locker) or people densiies (e.g., discoheque crowded beyond an accepable level). 3. Behavioral Alers: These alers are generaed based on adherence o, or deviaion from, learn models of moion paerns. Such models are ypically rained by analyzing movemen paerns over exended periods of ime. These alers end o be very applicaion specific and use a significan amoun of conex informaion, for example, 1. Deecing shopping groups a reail checkou couners, and alering he sore manager when he lengh of he queue a a couner exceeds a specified number. [8] 2. Deecing suspicious behavior in parking los, for example, a person sopping and rying o open muliple cars. 4. High Value Capure: This is an applicaion which augmens real ime alers by capuring seleced clips of video based on pre-specified crieria. This becomes highly relevan in he conex of smar camera neworks which use wireless communicaion. 2.1.2 Auomaic Unusual Aciviy Alers: Unlike he user defined alers, here he sysem generaes alers when i deecs aciviy ha deviaes from he norm. The smar surveillance sysem achieves his based on learning normal aciviy paerns [17]. For example, a smar surveillance sysem ha is monioring a sree learns ha vehicles move abou on he road and people move abou on he side walk. Based on his paern he sysem will provide an aler when a car drives on he side walk. Such unusual aciviy deecion is he key o effecive smar surveillance, as all he evens of ineres canno be manually specified by he user. 2.2 Auomaic Forensic Rerieval (AFVR): The capabiliy o suppor forensic video rerieval is based on he rich video index generaed by auomaic racking echnology. This is a criical value-add from using smar surveillance echnoloes. Typically he index consiss of such measuremens as objec shape, size and appearance informaion, emporal rajecories of objecs over ime, objec ype informaion, in some cases specific objec idenificaion informaion. In advanced sysems, he index may conain objec aciviy informaion. The Washingon, DC sniper inciden is a prime example of where AFVR could be a break-hrough echnology. During he inciden he invesigaive agencies had access o hundreds of hours of video surveillance fooage drawn from a wide variey of surveillance cameras covering he areas in he viciniy of he various incidens. However, he ask of manually sifing hrough hundreds of hours of video for invesigaive purposes is almos impossible. However if he collecion of videos were indexed using visual analysis, i would enable he following ways of rerieving he video 1. Spaio-Temporal Rerieval: An example query in his class would be, Rerieve all clips of video where a blue car drove in fron of he 7/11 Sore on 23 rd sree beween he 26 h of July 2pm and 27 h of July 9am a speeds > 25mph. 2. Surveillance Mining: In he case of he Washingon sniper inciden, he surveillance video mining applicaion would aemp o presen he users wih a se of poenial movemen paerns of cars over a se of cameras covering muliple inciden locaions, his would enable he invesigaive agencies o answer quesions like Was here a single car ha appeared in all of he inciden locaions?. 2.3 Siuaion Awareness: Ensuring oal securiy a a faciliy requires sysems ha can perpeually rack he ideniy, locaion and aciviy of people and vehicles wihin he moniored space. For example, he exising surveillance echnology canno answer quesions such as: did a single person loier around near a high securiy building on muliple occasions? Such perpeual racking can be he basis for very high levels of securiy. Typically surveillance sysems have focused on racking locaion and aciviy, while biomerics sysems have focused on idenifying individuals. As smar surveillance echnoloes maure [7], i becomes possible o address all hese hree key challenges in a single unified frame work ving rise o, join locaion ideniy and aciviy awareness, which when combined wih he applicaion conex becomes he basis for siuaion awareness. 3. Smar Surveillance Archiecures: In his secion we discuss how smar surveillance echnoloes are incorporaed ino a complee surveillance sysem. We discuss hree differen ypes of smar surveillance archiecures 3.1 Basic Smar Surveillance Archiecure (BSSA): Dial Recorder Smar Surveillance Channel Selecion Swicher Rich Index Index User Terminal Real Time Alers Forensic Rerieval Figure 1: Block diagram of a smar surveillance sysem. Figure 1 shows he block diagram of a basic smar surveillance sysem. The oupus of video cameras are recorded dially and simulaneously analyzed by he smar surveillance server, which produces real ime alers and a rich video index. The ypes and parameers of he alers are user configurable. The user can use he rich index o rerieve video from he archive for forensic. In he BSSA 2
d i d i d i di di he video recording and analysis is cenralized requiring he cameras o be wired o a cenral locaion. The role of auomaic visual analysis in he BSSA is primarily analyical. 3.2 Acive Smar Surveillance Archiecure (ASSA): Figure 2 shows he block diagram for an acive smar surveillance archiecure. The key difference beween his and a BSSA is he addiion of acive camera conrols. Here he auomaic visual analysis is used no only o undersand wha is going on in he scene bu also o selecively pay more aenion o auomaically deeced aciviies or evens of ineres. The ASSA could be used for many differen applicaions. Below we describe wo examples. 1. Face Caaloger: This is a sysem which aims o non inrusively acquire a high-resoluion face images of all people passing hrough a space, Here ASSA deecs and racks people and uses he acive cameras o zoom in and acquire high resoluion face picures. 2. Muli-scale : This is a sysem which auomaically allocaes higher resoluion o porions of he scene which have cerain predeermined ypes of aciviy. For example, all cars ha are moving wih high speed hrough a parking lo may be imaged a a higher resoluion hrough he acive cameras. Compuer Conrolled PTZ Dial Recorder Camera Conrol Acive Camera Smar Surveillance Channel Selecion Swicher Rich Index Index User Terminal Real Time Alers Forensic Rerieval Figure 2: Block diagram of an Acive Smar Surveillance Sysem. The key echnical difference beween he ASSSA and BSSA are Acive Camera Resource Managemen: This includes echniques for deciding which of he acive cameras are o be used o mee he overall goals of he sysem. Acive Camera Image Analysis: This includes analysis of he acive camera images in order o conrol he movemen and zoom of he cameras. 3.3 Disribued Smar Surveillance Archiecure (DSSA) Smar Disribued Camera Coordinaor High Value & Index User Terminal Real Time Alers Forensic Rerieval Figure 3: Block diagram of a disribued smar surveillance archiecure using smar cameras. One of he bigges coss in deploying a surveillance sysem is he infrasrucure (e.g., wiring) required o move he video from he cameras o a cenral locaion where i can be analyzed and sored. Furher, he wiring capaciy is no easily scalable and because of involvemen of manual labor, insananeous/porable insallaion of nework is difficul. There is an increasing rend o building cameras which in addiion o generaing images also analyze hem wih onboard processing. Such cameras are ypically called smar camera. The goal of a smar camera sysem is o minimize he cos of deploymen. In such archiecure (fig 3), he camera would ypically use wireless communicaion o coordinae wih a cenral camera coordinaor. A smar camera would use he auomaic visual analysis o deermine how o use he limied sorage and bandwidh o effecively ransmi only high value video o he server. 4. Technoloes for Smar Surveillance. Among he hree archiecures presened in he previous secion, he BSSA is likely o be he mos ubiquious in he near fuure. The BSSA is an enhancemen of he archiecure of curren day surveillance sysems, which have cameras mouned in differen pars of a faciliy, all of which are wired o a cenral conrol room. In his secion, we will discuss various echnoloes ha enable he BSSA and discuss he challenges involved as such sysems ge widely deployed. Figure 4 shows he inernal srucure of a smar surveillance server which is one of he key componens in he BSSA. The video from a camera is processed o deec moving objecs of ineres, hese objecs are racked as hey move abou in he moniored space. The racked objec becomes 3
a fundamenal inernal represenaion in he sysem upon which a number of processes ac, including classificaions and real ime aler module. In his secion we discuss each of he key componens of he smar surveillance server. Inpu Objec Deecion Module Muli-objec Tracking Module Objec Classificaion Objec Tracks Objec Type Foreground Objecs Acive Objec Track Sore Rich Index Aler definiions from User Real Time Aler Module Real Time Alers Figure 4: Inernal srucure of he smar surveillance enne. 4.1 Objec Deecion: Mos of he work on objec deecion relies heavily on he assumpion of a saic camera [9]. There is some work which has looked a deecing independen moion in moving camera images where he camera moion is well modeled [18]. Our approach o objec deecion has been wo pronged each of which we discuss briefly below. 4.1.1: Adapive Background Subracion wih Healing: The background subracion module combines evidence from differences in color, exure, and moion. The use of muliple modaliies improves he deecion of objecs in cluered environmens. The resuling saliency map is smoohed using morpholocal operaors and hen small holes and blobs are eliminaed o generae a clean foreground mask. The background subracion module has a number of mechanisms o handle channg ambien condiions and scene composiion. Firs, i coninually updaes is overall RGB channel noise parameers o compensae for channg ligh levels. Second, i esimaes and correcs for AGC (auomaic gain conrol) and AWB (auomaic while balance) shifs induced by he camera. Thirdly, i mainains a map of high aciviy reons and slowly updaes is background model only in areas deemed as relaively quiescen. Finally, i auomaically eliminaes occasional spurious foreground objecs based on heir moion paerns. 4.1.2: Salien Moion Deecion: This is a complemenary approach o background subracion. Here we approach he problem from a moion filering perspecive. Consider he following figure 5, he image on he lef shows a scene where a person is walking in from of a bush which is waving in he wind. The nex image in figure 5 shows he oupu of a radiional background subracion algorihm (which per is design correcly classifies he enire bush as a moving objec). However, in his siuaion, we are ineresed in deecing he person as opposed o he moving bush. Our approach uses opical flow as he basis for deecing salien moion. We use a emporal window of N frames (ypically 10-15) o assess he coherence of opic flow a each pixel over he enire emporal window. Pixels wih coheren opical flow are labeled as candidaes. The candidaes from he moion filering are hen subjeced o a reon growing process o obain he final deecion. Orinal Image Background Subracion Image Difference Opical Flow Figure 5 Illusraion of salien moion deecion. Final Resul Background subracion and salien moion deecion are complemenary approaches, each wih is srenghs and weakness. Background subracion is more suied for indoor environmens where lighing is fairly sable and disracing moions are limied, whereas salien moion deecion is well suied o deec coheren moion in oudoor siuaions. 4.2 Muli-Objec Tracking: Muli-objec racking aemps o associae objecs wih one anoher over ime, by using a combinaion of he objecs appearance and movemen characerisics. This has been a very acive area of research [2,3,4,5,8] in he pas several years. Our approach o muli-objec racking has focused heavily on handling occlusions [15]. Following is a brief descripion of our approach. The muli-objec blob racking relies on appearance models which are image-based emplaes of objec appearance. New appearance models are creaed when an objec eners a scene. In every new frame, each of he exising racks is used o ry o explain he foreground pixels. The fiing mechanism used is correlaion, implemened as minimizaion of sum of absolue pixel differences beween he deeced foreground area and an exising appearance model. During occlusions, foreground pixels may represen appearance of overlapping objecs. Color similariy is used o deermine occlusion informaion (relaive deph ordering) for he objec racks. Once his relaive deph ordering is esablished, he racks are correlaed in order of deph. The correlaion process is gaed by he explanaion map which holds a each pixel he ideniies of he racks explaining he pixels. Thus foreground pixels ha have already been explained by a rack do no paricipae in he correlaion process wih models of he objecs which are more disan. The explanaion map is now used o updae he appearance models of objecs associaed wih each of he exising racks. Reons of foreground pixels ha are no explained by exising racks are candidaes for new racks. A deailed discussion of he 2D muli-blob racking algorihm can be found in [15]. The 2D muli- objec racker is capable of racking muliple objecs moving wihin he field of view of he camera, while mainaining an accurae models of he shapes and colors of he objecs. 4
Orinal Image Wih Occlusion Tracked Objec Models Resolved Pixel Map Figure 6: Occlusion handling in objec racking: Lef Orinal image wih occlusion. Middle: Objecs being racked by he sysem. Righ: Classificaion of pixels ino models. Track Observer is deleed immediaely upon he exi of he objec from he scene. 2. Direcional Moion Objec Track Observer: This is he process ha is charged wih he job of measuring he direcion of moion of he objec and comparing i o he user specified direcion. When ever he objec moion direcion maches he user specified direcion, he objec rack observer issues a real ime aler. The applicaion uses he aler o signal he user ha one of he specified aler condiions has been me. Track Manager Raw Segmened Objec 4.3 Objec Classificaion: Moving foreground objecs are classified ino relevan caegories. Saisics abou he appearance, shape, and moion of moving objecs can be used o quickly disinguish people, vehicles, cars, animals, doors opening/closing, rees moving in he breeze, ec. Our sysem classifies objecs ino vehicles, individuals, and groups of people based on shape feaures (compacness and ellipse parameers), recurren moion measuremens, speed and direcion of moion (see Fig 7). From a small se of raining examples, we are able o classify objecs in similar fooage using a Fisher linear discriminan classfier and emporal consisency informaion. Figure 7: Lef: Oupu of objec classificaion algorihm. Righ Inermediae seps in objec classificaion 4.4 Real Time Aler Module: The real ime aler module uses he informaion produced by he oher modules, namely, objec deecion, racking and classificaion o deec user specified alers. The key feaure of he real ime aler module is is exensible design. Figure 8 shows he generic srucure of he real ime aler module. This srucure is insaniaed by mos of he aler ypes. In order o illusrae he srucure of he module we presen he design of he direcional moion aler as an example. The following processes are insaniaed when he user specifies a direcional moion aler. 1. Direcional Moion Aler Manager: Each moion aler user definiion insaniaes a direcional moion aler manager, which is responsible for ensuring correc monioring of he scene. The aler manager ensures ha for every objec being racked here is a corresponding Objec Track Observer ha is insaniaed. And ha he Objec AlerManager Objec Track Observer Real Time Aler Module User Aler Definiion Aler Condiion Saisfied Invoke Aler Callback Yes Figure 8: Inernal srucure of he real ime aler module. The exac naure of he objec rack observer depends on he paricular aler i is implemening. However he general srucure of many ypes of alers is similar o ha described above. 5. Challenges There are wo ypes of challenges ha we highligh in he fuure developmen of smar surveillance sysems. 1. Technical Challenges: There are a number of echnical challenges ha sill need o be addressed in he underlying visual analysis echnoloes. These include challenges in robus objec deecion, racking objecs in crowded environmens, challenges in racking ariculaed bodies for aciviy undersanding, combining biomeric echnoloes like face recogniion wih surveillance o achieve siuaion awareness. 2. Challenges in Performance Evaluaion: This is a very significan challenge in smar surveillance sysem. Evaluaing performance of video analysis sysems requires significan amouns of annoaed daa. Typically annoaion is a very expensive and edious process. Addiionally, here can be significan errors in annoaion. All of hese issues make performance evaluaion a significan challenge. 6. Implicaions of Smar Surveillance Smar surveillance is a echnology ha has many differen applicaions and poenially has significan implicaions o each of hese. We look a implicaions primarily in he surveillance applicaion, namely, securiy and privacy. Securiy Implicaions: Clearly, he abiliy o provide real ime alers, capure high value video and provide sophisicaed forensic video rerieval has he poenial o enhance securiy in various public and privae faciliies. However, he value of he echnology is ye o be proven in he field. As more and more smar surveillance sysems ge deployed he exac value will be known. In paricular, sysems mus be analysed for heir effeciveness in 5
deecing evens of ineres, while generaing few false alarms. In he firs insance smar surveillance sysems are inended o assis securiy guards, and will be measured on heir abiliy o improve vilance and o reduce labor and sorage coss. Privacy Implicaions: Smar surveillance sysems have he abiliy o monior video a evel which is humanly impossible. This provides he monioring agencies wih a significanly enhanced level of informaion abou he people in he space leading o higher concerns abou individual privacy and abuse of sensiive individual informaion. However, he same smar surveillance echnology by virue of indexing he video provides novel ways of enhancing privacy in video based sysems which was hihero no possible. Furher deails on he privacy preserving aspecs of smar surveillance echnoloes can be found in [16]. 7. Conclusions While i is difficul o foresee a fuure where he surveillance of he moniored space is compleely auomaic, here is clearly an urgen need o augmen he exising surveillance echnology wih beer ools o aid efficacy of he human operaors. Wih he increasing availabiliy of he inexpensive compuing, video infrasrucure and beer video analysis echnoloes smar surveillance sysems will compleely replace exising surveillance sysems. The degree of smarness will vary wih he level of securiy offered by such sysems. 8. Acknowledgemens All he echnical discussions in his paper are based on ongoing work in visual racking a IBM research. The auhors wish o acknowledge he PeopleVision projec a IBM T.J.Wason Research Cener [14]. References 1. Lisa Brown and Yingli Tian, Comparaive Sudy of Coarse Head Pose Esimaion, IEEE Workshop on Moion and Compuing, Orlando FL, Dec. 5-6, 2002 2. Anjum Ali, J. K. Aggarwal: Segmenaion and Recogniion of Coninuous Human Aciviy. IEEE Workshop on Deecion and Recogniion of Evens in 2001. 3. Collins, Lipon, Fujiyoshi, and Kanade, Algorihms for cooperaive mulisensor surveillance, Proc. IEEE, Vol. 89, No. 10, Oc. 2001. 4. D. Comaniciu, V. Ramesh, P. Meer: Real-Time Tracking of Non-Rid Objecs using Mean Shif,, IEEE Conf. Compuer Vision and Paern Recogniion (CVPR'00), Hilon Head Island, Souh Carolina, Vol. 2, 142-149, 2000 5. Trevor Darrell, David Demirdjian, Neal Checka, Pedro Felzenszwalb: Plan-View Trajecory Esimaion wih Dense Sereo Background Models. ICCV 2001: 628-635. 6. Mary W. Green, The Appropriae and Effecive Use of Securiy Technoloes in U.S. Schools, A Guide for Schools and Law Enforcemen Agencies, Sandia Naional Laboraories, Sepember 1999, NCJ 178265 7. A. Hampapur e al., Face Caaloger: Muli-Scale Imang for Relaing Ideniy o Locaion, IEEE Inernaional Conference on Advanced and Signal Based Surveillance, Miami, FL, July 03. 8. Hariaolu and Flickner, Deecion and Tracking of Shopping Groups in Sores, CVPR 2001. 9. T. Horpraser, D. Harwood, and L. Davis. A Saisical Approach for Real-Time Robus Background Subracion and Shadow Deecion. Proceedings of IEEE Frame-Rae Workshop, Kerkyra, Greece, 1999. 10. A. K. Jain, R. Bolle, and S. Pankani (eds.), Biomerics: Personal Idenificaion in Neworked Sociey, Kluwer Academic, 1999. (ISBN 0792383451) 11. G. Kogu, M. Trivedi, "A Wide Area Tracking Sysem for Vision Sensor Neworks," 9h World Congress on Inelligen Transpor Sysems, Chicalgo, Illinois,Ocober,2002. 12. Miller e al, Crew faigue and performance on US coas guard cuers, Oc 1998, US Dep of Transporaion. 13. Anurag Mial and Larry S. Davis, M2Tracker: A Muli-View Approach o Segmening and Tracking People in a Cluered Scene. Inernaional Journal of Compuer Vision. Vol. 51 (3), Feb/March 2003. 14. PeopleVision: Demo videos of on going work a IBM research: www.research.ibm.com/peoplevision 15. A. Senior e al., Appearance Models for Occlusion Handling. Proceedings of IEEE Workshop on Performance Evaluaion of Tracking and Surveillance Sysems, 2001. 16. A Senior e al, Enabling video privacy hrough compuer vision, To appear, IEEE Securiy and Privacy Magazine. 17. C. Sauffer, Auomaic hierarchical classificaion using ime-based co-occurrences, IEEE Conf on CVPR99, 333--339, 1999, 18. H. Tao, H.S. Sawhney and R. Kumar, Dynamic Layer Represenaion wih Applicaions o Tracking, Proc. of he IEEE Compuer Vision & Paern Recogniion, Hilon Head, SC, 2000. 6