A cooperative perception system for multiple UAVs: Application to automatic. detection of forest fires

Transcription

1 A cooperatve percepton system for multple UAVs: Applcaton to automatc detecton of forest fres Lus Merno 1, Fernando Caballero 2, J. R. Martínez-de Dos 2, Joaquín Ferruz 2 and Aníbal Ollero 2 Robotcs, Vson and Intellgent Control Group 1 Dpt. Envronmental Scences, Pablo de Olavde Unversty, Sevlle, Span 2 Dpt. Systems Engneerng and Automatc Control, Unversty of Sevlle, Sevlle, Span ABSTRACT: Ths paper presents a cooperatve percepton system for multple heterogeneous UAVs. It consders dfferent knd of sensors: nfrared and vsual cameras and fre detectors. The system s based on a set of multpurpose low-level mage-processng functons ncludng segmentaton, stablzaton of sequences of mages and geo-referencng, and t also nvolves data fuson algorthms for cooperatve percepton. It has been tested n feld experments that pursued autonomous mult-uav cooperatve detecton, montorng and measurement of forest fres. Ths paper presents the overall archtecture of the percepton system, descrbes some of the mplemented cooperatve percepton technques and shows expermental results on automatc forest fre detecton and localzaton wth cooperatng UAVs. KEYWORDS: mult-uav system, cooperatve percepton, automatc forest fre detecton, feld expermentaton. 1

2 1. Introducton In the last decade unmanned aeral vehcles (UAVs) have attracted a sgnfcant nterest n many feld robotcs applcatons. The hgher moblty and maneuverablty of UAVs respect to ground vehcles have made aeral vehcles the natural way to approach a target to get nformaton or even to perform some actons such as the deployment of nstrumentaton. Aeral robotcs seems a useful approach to perform tasks such as data and mage acquston of targets and areas naccessble usng ground means, localzaton of targets, trackng, map buldng and others. UAVs have been wdely used for mltary applcatons but, recently they are beng extended to cvlan applcatons such as natural and human made dsasters scenaros, search and rescue, law enforcement, aeral mappng, traffc survellance, nspecton and cnematography (Ollero & Merno, 2004). Many of these applcatons requre robust and flexble percepton systems. The most common percepton devces n UAVs are cameras and range sensors. Range sensors are used for some specfc operatons such as autonomous landng and mappng (Mller & Amd, 1998). Computer vson plays the most mportant role and has been appled for dfferent tasks. It has been used as a method to sense relatve poston, as n the approach by Omd, Kanade & Fujta (1999), where t s mplemented the concept of vsual odometer, n Zhang & Hntz (1995), where a vdeo-based atttude and heght sensor for low alttude aeral vehcles s presented, or n Corke, Skka & Roberts (2001), where a stereo vson system s used for heght estmaton. Vson-based methods have been also consdered for safe landng of a helcopter (Sarpall, Montgomery & Sukhatme, 2003). Lacrox, Jung & Mallet (2001) descrbe Smultaneous Localzaton and Mappng (SLAM) technques wth stereo vson systems on board an autonomous arshp. UAV SLAM wth vson s also presented n Km & Sukkareh (2003). Furthermore, computer vson has been used for detecton and montorng. Thus, algorthms for dense moton estmaton have been appled to traffc montorng wth an UAV (Farnebäck & Nordberg, 2002). Vdal, Sastry, Km, Shakerna & Shm (2002) used computer vson to detect 2

3 evaders. Other applcatons nclude road dentfcaton and trackng (Bueno et al., 2002) and nspecton of power lnes (Del-Cerro, Barrento, Campoy & García, 2002). Many of the above-mentoned systems and methods nvolve only one UAV. However, the complexty of some applcatons requres cooperaton between UAVs or between UAVs and other robots. Systems wth multple UAVs are very scarce and have been appled manly for mltary applcatons. The coordnaton of multple homogeneous UAVs n close-formaton flght has been usually studed usng control approaches; for example (Hall & Pachter, 1999) and (Gulett, Pollne & Innocent, 2000). In ths paper we consder the cooperaton of multple heterogeneous UAVs. The heterogenety ncreases the complexty of the problem, but also provdes several advantages for the applcaton such us the possblty to explot the complementartes of dfferent UAV platforms wth dfferent moblty attrbutes and also dfferent sensor and percepton functonaltes. It should be noted that many applcatons requre several sensors that can not be carred by only one UAV due to payload lmtatons. In these cases the cooperaton between the UAVs equpped wth dfferent sensors should be establshed also at a percepton level. Ths paper presents a mult-uav cooperatve percepton system. The archtecture of the percepton system allows both sngle-uav and cooperatng UAVs percepton. It consders manly nfrared and vsual cameras, and also a specalzed fre sensor, but can be adapted to other knd of sensors. The system ncludes multpurpose mage-processng functons approprate for a wde range of tasks ncludng among others survellance, detecton, montorng and, measurng. The proposed percepton system has been demonstrated for the autonomous detecton, montorng and measurng of forest fres. Ths s a very relevant applcaton n many countres where forest fres have dsastrous socal, economc and envronmental mpact. Furthermore, forest fre fghtng s a very dangerous actvty that orgnates many casualtes every year. Ths paper presents results of feld experments on fre detecton, confrmaton and precse localzaton wth cooperatng UAVs. 3

4 The work descrbed n the paper has been carred out n the framework of project COMETS: Real-tme coordnaton and control of multple heterogeneous unmanned aeral vehcles (IST ) of the IST Programme of the European Commsson. The objectve of the COMETS project was to desgn and mplement a system for cooperatve actvtes usng heterogeneous UAVs. The heterogenety of the UAVs consdered n the system s manfold. On one hand, complementary platforms are consdered: helcopters, and arshps. The helcopters have hgh maneuverablty and the hoverng ablty to perform effcently nspecton and montorng tasks that requre to mantan a poston and to obtan detaled vews. Arshps have much less maneuverablty and can be used to provde global vews or to act as communcatons relay. On the other hand, the UAVs consdered are also heterogeneous n terms of on board processng capabltes, rangng from fully autonomous aeral systems to conventonal rado controlled systems wth mnmal on-board capabltes requred to record and transmt nformaton. Thus, the plannng, percepton and control functonaltes of the UAVs can be ether mplemented onboard the vehcles, f enough on-board processng power s avalable, or on ground statons when lght, low-cost aeral vehcles are used. Fnally, the UAVs are also heterogeneous respect to the sensors they carry on board. Ths characterstc plays an mportant role n the co-operatve percepton work descrbed n ths paper. In order to acheve ths general objectve, the COMETS project produced a new decsonal archtecture (Gancet, Hattenberger, Alam & Lacrox, 2005a), (Gancet, Hattenberger, Alam & Lacrox, 2005b), (Ollero et al., 2005). Ths archtecture s used to coordnate the fleet of vehcles. It allows to decompose, ether n a centralzed or partally decentralzed way, a complex msson plan nto atomc tasks to be processed by the vehcles. These tasks nclude cooperatve percepton tasks, such as the synchronzed percepton of a target. The cooperatve percepton system s lnked to the decsonal archtecture, and the fleet can react dependng on the data and events rased by the percepton algorthms, through re-plannng. 4

5 Although the COMETS system could gve support to a wde range of applcaton, the specfc problem of forest fre detecton and montorng was chosen for testng and valdaton purposes. UAVs cooperaton s very valuable n ths hghly challengng context. Mssons nvolve fre alarm detecton, confrmaton and localzaton, and fre montorng. Several feld tests wth controlled fres have been carred out durng the past years. Fgure 1 shows some pctures of these experments. Fgure 1: Left Marvn and Helv durng a experment. Rght, Karma flyng over Marvn and Helv n a cloudy day. The followng UAVs were deployed durng the COMETS experments: the helcopter Marvn, the arshp Karma and the helcopter Helv. Marvn s an autonomous helcopter developed by the Real-Tme Systems & Robotcs Group of the Technsche Unverstät Berln (Remuß, Musal & Hommel, 2002). Karma s an autonomous 18m 3 electrcally propelled arshp developed by LAAS (Laboratore d'archtecture et d'analyse des Systèmes) at Toulouse (Lacrox, Jung, Soueres, Hygounenc & Berry, 2003). Helv s the result of the evoluton of a conventonal remotely ploted helcopter whch has been transformed by the Robotcs, Vson and Control 5

6 Group at the Unversty of Sevlle by addng sensng, percepton, communcaton and control functons. Fgure 1 shows the three vehcles durng the feld experments presented n ths paper. The rest of the paper s structured as follows. Secton 2 presents the cooperatve percepton system for UAVs ncludng the hardware and software archtectures and communcatons. Secton 3 descrbes some of the computer vson technques ncluded n the percepton system, wth specal emphass on technques for stablzaton of sequences of mages, mage segmentaton and mage geo-locaton. Secton 4 deals wth the cooperatve percepton algorthms. Secton 5 presents feld experments on autonomous fre detecton, fre alarm confrmaton and localzaton wth cooperatng UAVs. Conclusons and acknowledgements are the fnal sectons. 2. The Percepton System Ths secton presents the mult-uav dstrbuted percepton system wth specal emphass on sensors, ts software archtecture and communcatons. 2.1 Sensors The UAVs are heterogeneous also n the sense of the sensors carred by them. They are equpped wth DGPS, gyroscopes and Inertal Measurement Unts and other sensors requred for navgaton. The man envronment percepton sensors consdered n ths paper are vsual and nfrared cameras, and a specalzed fre sensor. Marvn carres a fre sensor, whose man component s a photodode set-up to lmt ts sensblty to the band of [185, 260] nm, normally assocated to fres. The output of the sensor s a scalar value, proportonal to the radaton energy, receved every 2 seconds. Beng a magntude sensor, t s not possble to determne f a measure s due to a bg fre far away or a nearby small fre. Also, the sensor cannot drectly provde the poston of the fre. Secton 4 wll present the procedure used to detect and localze fres by usng ths sensor. Marvn also carres a Canon S40 dgtal photo camera. 6

7 Helv s equpped wth nfrared and vsual vdeo cameras. Each vdeo camera s connected to a vdeo server whch dgtzes and sends the mage streams usng standard net protocols. The nfrared camera s a low-cost non-thermal OEM mcro-camera (see Fgure 2 rght) n the far nfrared band (7-14 mcrons). The vsual camera s a low-weght color devce wth 320x240 pxel resoluton. Both helcopters, Marvn and Helv, have motorzed pan and tlt unts that allow orentatng the cameras ndependently from the body of the vehcle (see Fgure 2 left). Those unts have encoders that measure the pan and tlt angles. Fnally, Karma carres a stereo bench wth two vsual cameras n order to generate depth maps. These cameras are also used for event montorng. Fgure 2: Left: Infrared and vsual cameras of Helv mounted n the pan and tlt unt. Rght: detal of the nfrared mcro-camera. 2.2 Software archtecture Fgure 3 shows the software archtecture of the Percepton System (PS). Ths system conssts of a dstrbuted subsystem, called Applcaton-Independent Image Processng (AIIP), and two centralzed subsystems (whch deal wth the cooperatve algorthms): Detecton/Alarm Confrmaton, Localzaton and Evaluaton Servce (DACLE) and, the Event Montorng System (EMS). 7

8 Fgure 3: PS subsystems nterconnecton and archtecture. The communcatons system employed allows to locate the AIIP on-board UAVs (n the FS) or on ground (n the GS) transparently. Left: partally dstrbuted confguraton. Rght: fully dstrbuted confguraton. The AIIP subsystem s the processng front-end, the module of the Percepton System closest to the sensors. There s one AIIP module for each camera, and for each UAV, ts AIIPs can be located on-board f they have enough processng capabltes (case of UAV of Fgure 3) or on ground statons, otherwse (case of UAV j of Fgure 3 left). The AIIP apples a frst processng step over the data, reducng ts dmensonalty (and hence, the bandwdth needed to transmt them). The AIIP manly deals wth the low-level mage processng functons that are common to the DACLE and EMS subsystems such as stablzaton of mage sequences, segmentaton and geo-referencng. These functonaltes wll be descrbed n Secton 3. Also, the AIIP acts as a vrtual mage channel, beng able to modfy the resoluton and regon of nterest of the mages. The objectve of the DACLE s to perform fre detecton/alarm confrmaton and localzaton. At ts request, the DACLE subsystem receves nformaton about possble fre alarms and other data from the AIIPs of the UAVs. DACLE apples sensor data fuson technques to explot the complementartes of the nformaton gathered by the dfferent sensors on board the dfferent UAVs. Partcularly, DACLE performs cooperatve relable detecton and ncludes technques for false alarm reducton. It also mproves the localzaton of the alarms by fusng the locatons gven by the sensors of several vehcles and takng nto account ther uncertantes n a statstcal framework. These technques are presented n Secton 4. 8

9 The EMS s n charge of the mult-uav fre montorng functonaltes. Ths subsystem s not descrbed n ths paper due to space lmtatons. 2.3 Communcatons The dstrbuted percepton system employs a custom communcaton system, called BlackBoard Communcaton System (BBCS) as communcaton layer for the dfferent subsystems. The BBCS, developed by the Techncal Unversty of Berln (Remuss, Musal & Brandenburg, 2004), (Remuss & Musal, 2004), s mplemented va a dstrbuted shared memory, called blackboard. The consstency of ths shared memory s ensured by a real-tme aware protocol. The BBCS API also offers a set of functons to deal wth wreless communcatons and nclude functons robust to perods of degraded bandwdth, not nfrequent n forest scenaros. Its hgh confguraton capablty allows mplementng network communcatons wth low delay usng a smple software structure. The BBCS s bult on top of exstng transport layers (UDP, TCP), and can be adapted to dfferent knds of operatng systems and hardware platforms (rangng from PCs to mcrocontrollers), always offerng the same servces and nterfaces. The subsystems of the PS can then be located on board the UAVs or on laptops on the ground, over dfferent archtectures, wthout sgnfcant changes n the confguraton of the network. 3. Low level percepton technques Ths secton presents some of the functonaltes currently consdered wthn the AIIP subsystem (outlned n Fgure 4). These functons are requred for automatc forest fre detecton and localzaton. Although tested for ths specfc scenaro, t should be noted that these tools, as well as the cooperatve technques descrbed n Secton 4, can be adapted to a wde spectrum of applcatons. 9

10 Fgure 4: Scheme of AIIP functonaltes and ther relatons. 3.1 Fre segmentaton Fre segmentaton s a functon of the AIIP essental for fre detecton, carred out by DACLE. The man objectve s to dfferentate fre pxels from background pxels. Two segmentaton technques have been appled dependng on the type of mage: vsual or nfrared. A bnary correcton algorthm s appled n both cases after segmentaton to flter out solated fre and background pxels (Haralck & Shapro, 1992) Fre segmentaton n vsual mages The technque used s a tranng-based algorthm smlar to those descrbed by Kjedlsen & Kender (1996) and Phlps, Shah & da Vtora-Lobo (2002). The method requres some tranng mages n whch an experenced user has determned the pxels that correspond to the fre. In the tranng stage a RGB hstogram s bult by addng Gaussan-type dstrbutons centered at the RGB coordnates of the pxels consdered as a fre pxel n the tranng mages. If the pxel s consdered as background n the tranng mages, a Gaussan-type dstrbuton centered at the RGB coordnates s subtracted from the RGB hstogram. Fnally, ths RGB hstogram s thresholded and a look-up table for the RGB color space s bult. The look-up table contans a 10

11 Boolean value ndcatng whether the color represents fre or background. In the applcaton stage the RGB coordnates of the pxels are mapped n the traned look-up table and are consdered fre pxels f the value n the look-up table s 1 and, background otherwse. Fgure 5 rght shows the mage resultng from segmentng the mage n Fgure 5 left (See Vdeo 1, for a vdeo showng more results). Fgure 5: Left: Vsual mage of a fre experment; Rgth: the resultng segmented mage Fre segmentaton n nfrared mages The nfrared camera used n the experments was a low-cost OEM non-thermal camera. It does not provde temperature measures but estmatons of the radaton ntensty throughout the scene. Black and whte colors represent low and hgh radaton ntenstes, respectvely. Thresholdng s proposed for fre segmentaton. For robust fre segmentaton, the thresholdng technque should consder the partculartes of the applcaton. The soluton adopted was to use the tranng-based thresholdng method descrbed n Martínez-de Dos & Ollero (2004). Its man dea s to extract the partculartes of a computer vson applcaton and use them to supervse a multresoluton hstogram analyss. The technque s appled n two stages: tranng and applcaton, see Fgure 6. The tranng stage requres a set of tranng mages and ther correspondng desred threshold values gven by an experenced user. The tranng stage dentfes the condtons under whch pxels should be consdered to belong to the object of nterest. These partculartes are ntroduced n a system va ANFIS tranng method (Jang, 1993). In the applcaton stage, features of the mage are used to determne a sutable threshold value accordng to these partculartes. A 11

12 detaled descrpton can be found n Martínez-de Dos & Ollero (2004). At Vdeo 2 shows some results. Tranng Images Desred threshold values tranng stage mage 1 mage NI Knowledge extracton desred_th 1 desred_th NI applcaton stage Image Multresoluton analyss Threshold value Fgure 6: General scheme of the tranng-based threshold selecton Characterzaton of the fre segmentaton algorthms The prevous algorthms are used for fre detecton. The vehcles of the fleet wll cooperate to reduce the number of false alarms by means of data fuson (see Secton 4), and ths requres the probablstc characterzaton of the above segmentaton algorthms. The algorthms are modeled by the probabltes P D of detecton and P F of false postve outputs. These values have been expermentally determned for both algorthms wth a large set of mages, some of whch present actual fres. The probabltes have been computed as follows: P D s the rato between the alarms correctly detected and the total number of fre alarms presented n the set of mages. P F s the rato between the number of mages where the algorthm detected fre ncorrectly and the total number of mages of the sequence. Table I shows the obtaned values for the algorthms used for fre segmentaton n vsual and nfrared mages. TABLE I CHARACTERISTIZATION OF FIRE SEGMENTATION ALGORITHMS IR Vsual P D 100% 89.2% P F 8.9% 3.1% 12

13 3.2 Geolocaton The determnaton of the geo-referenced locaton of the objects observed on the mages s requred for many applcatons. Besdes, t s very useful to obtan an estmaton of the uncertanty n the computed locaton. The sensors onboard the dfferent UAVs are used to compute, n a global and common coordnate frame, the poston and orentaton of each UAV tself and also of the sensors that are carred on board (these poston and orentaton wll be denoted by x s ). For the later, the UAV atttude angles measured by the IMU unts have to be combned wth those of the pan and tlt devces. Also, the UAVs provde an estmaton of the covarance matrx C s of the errors of these quanttes. If the camera s calbrated and a dgtal elevaton map, denoted by D, s avalable, t s possble to obtan the geo-referenced locaton x m of an object n the common global coordnate frame from ts poston on the mage plane, o: x = f ( o,, D) (1) m x s The functon f nverts the camera projecton obtaned by, for example, a pn-hole model of the camera. Ths model s obtaned through calbraton for all the cameras, usng the algorthm developed by Zhang (2000). Clearly, the functon f s non-lnear, and n the general case the dependence on the map D cannot be expressed analytcally. Notce that the errors n the poston and orentaton of the camera (represented by C s ) and the errors n the poston of the object on the mage plane (represented by C o ) are propagated nto x m (see Fgure 7) through (1). The covarances C m of these errors are estmated by usng the so-called Unscented Transform (Juler & Uhlmann, 1997), (Schmtt, Hanek, Beetz, Buck & Radg, 2002). The Unscented Transform s chosen because t allows to consder a more general class of functons than the usual frst order expanson. Also the estmated covarance matrx s more accurate than that obtaned by means of a Taylor expanson of f (Juler & Uhlmann, 1997). 13

14 Thus, by usng the geolocaton procedure, each UAV wll provde measures of the form [x m,c m ], where x m s the measured geo-referenced locaton of the event of nterest (for nstance, a segmented fre gven by the segmentaton functons) n the common coordnate frame and C m s the estmated covarance of the errors on ths locaton. Fgure 7: Scheme of the uncertanty propagaton durng the geolocaton process. 3.3 Feature matchng and stablzaton Many applcatons, such as montorng, requre havng moton-free sequences of mages. Thus, tools to compensate the moton nduced on the mage plane by the moton of the UAV are requred. Usng these tools, the AIIP system can provde sequences of stablzed mages. The approach adopted obtans the apparent mage moton by means of a robust nterest pont matchng algorthm, and compensates the moton by warpng the mages to a common mage frame. For specfc confguratons, the mage moton model used for ths warpng s a homography Feature matchng method The computaton of the approxmate ground plane homography needs a number of good matchng ponts between pars of mages n order to work robustly. The mage matchng method adopted s related to that descrbed by the authors n Ferruz & Ollero (2000), wth sgnfcant mprovements (Ollero, Ferruz, Caballero, Hurtado & Merno, 2004). Although the same feature selecton procedure of corner ponts s used, and a combnaton of least resdual correlaton error and smlarty between clusters of features s stll the dsambguaton constrant, a new matchng strategy has been mplemented. Instead of searchng for ndvdual matchng ponts, clusters are bult as persstent structures and searched for a whole. Ths allows to change the dsambguaton 14

15 algorthm from a relaxaton procedure to a more effcent predctve approach. Selected matchng hypothess are used as startng ponts to locate a full cluster; the poston of addtonal cluster members s predcted from the cluster deformaton model. The practcal result of the approach s to drastcally reduce the number of matchng tres, whch are by far the man component of processng tme when a sgnfcant number of features have to be tracked, and large search zones are needed to account for hgh speed mage plane moton. Ths s the case n non-stablzed aeral mages, especally f only vdeo streams of relatvely low frame rate are avalable (see Vdeo 3, for some results). As explaned n Ollero et al. (2004), the detected corners defne mage wndows whch are tracked n subsequent frames; the result of such trackng s a set of wndow sequences. For a cluster of wndows Φ, = {, }, the shape smlarty constrants that must hold are 1 2 n equvalent to assume that the changes n wndow dstrbuton can be approxmately descrbed by eucldean transformaton and scalng. The effects of nose and the nnacuraces of the model are accounted for through tolerance factors. Under the assumpton that such constrants hold, t s easy to verfy that two hypotheszed matchng pars allow to predct the poston of the other members of the cluster. The generaton of canddate clusters for a prevously known cluster can start from a prmary hypothess, namely the matchng wndow proposed for one of ts wndow sequences (see Fgure 8), selected because of the low grey-level resdual error between t and the last known wndow of the sequence. Ths assumpton allows to restrct the search zone for other sequences of the cluster, whch are used to generate at least one secondary hypothess. Gven both hypothess, the full structure of the cluster can be predcted wth the small uncertanty mposed by the tolerance parameters, and one or several canddate clusters can be added to a data base. The creaton of any gven canddate cluster can trgger the creaton of others for neghbour clusters, provded that there s some overlap among them; n Fgure 8, for example, the creaton of a canddate for cluster 1 can be 15

16 used mmedately to propagate hypothess and fnd a canddate for cluster 2. Drect search of matchng wndows s thus kept to a mnmum. At the fnal stage of the method, the best cluster canddates are used to generate clusters n the last mage, and determne the matchng wndows for each sequence. Cluster sze s used as a measure of local shape smlarty; a mnmum sze s requred to defne a vald cluster. If a matchng par cannot be ncluded n at least one vald cluster, t wll be rejected, regardless ts resdual error. Fgure 8: Generaton of cluster canddates Homography computaton By usng the above algorthm, a set of matches between two consecutve mages can be computed. The man dea here s to compute an mage moton model from these matches and, then, nverse ths model to undo the moton nduced n the mage. The moton model selected s a homography, so a planar surface or a pure camera rotaton are assumed as hypotheses. Homography-based technques have been proven to be frequently vald for aeral mages: planar surface model holds f the UAV fles at a suffcently hgh alttude; and pure rotaton model holds for a hoverng helcopter. Thus, f a set of ponts n the scene les n a plane, and they are maged from two vewponts, then the correspondng ponts n mages and j 16

17 are related by a plane-to-plane projectvty or planar homography (Faugeras, Luong & Papadopoulo, 2001), H: where ~ = [ u, v,1] k k k sm ~ = Hm ~, (2) j m s the vector of homogenous mage coordnates for a pont n mage k, H s a 3x3 non-sngular matrx and s s a scale factor. Only four correspondences are needed to determne H. In practce, more than four correspondences are avalable by usng the above matchng procedure, and the overdetermnaton s used to mprove accuracy. A robust outler rejecton procedure s used n ths work, based on LMedS (Least Medan Square Estmator) and further refned by the Far M- estmator (Xu & Zhang, 1996), (Zhang, 1996), (Zhang, 1995). Once the homography matrx H has been computed, the mages are warped to a common frame. The warpng was optmzed due to real-tme constrants (Ollero et al., 2004). The computaton tme for moton compensaton n mages of 384x287 pxels s 30 ms. n a Pentum III at 1GHz (see Vdeo 4, for a stablzed sequence). Fgure 9 shows a mosac of the scenaro of the feld experments of Secton 5 bult from mages gathered by the blmp Karma of the LAAS team usng the stablzaton procedure technques to reduce the global mage postonng error. 17

18 Fgure 9: Mosac of Lousa arfeld (Portugal). Mosac constructed usng more than 500 mages taken by Karma. The square shows a detal of the mosac. 4. Cooperatve fre detecton The objectve of the DACLE subsystem s, from the measures provded by each vehcle of the fleet, to cooperatvely estmate the geographcal locaton of potental fre alarms whle tryng to reduce the number of false alarms. The DACLE subsystem can receve as measures the fre sensor data from Marvn and the geolocated fre alarms from the AIIP subsystems of the UAVs that carry cameras onboard. Ths secton extends the work presented n Merno, Caballero, Martínez-de Dos & Ollero (2005). There, the authors presented the algorthms to deal wth nformaton provded only by cameras. Here, ths work s extended to cope wth fre sensor data and the fnal scheme s presented. Fgure 10 shows a scheme of the DACLE operaton. At tme k, the current nformaton about every alarm stored by DACLE s defned by [ ( k), ( k), p ( k) ] x. where x a (k) s the estmated geo-referenced locaton for alarm at tme k, a C a C a (k) s the estmated covarance matrx of the errors n x a (k) and p (k) s the estmated probablty for ths alarm to be a fre. 18

19 Fgure 10: Scheme of the DACLE functonaltes. The fre detecton procedure conssts of two stages, called detecton and confrmaton. 4.1 Fre detecton In ths stage one or several UAVs are commanded to survey non-overlappng areas searchng potental fre alarms. In ths case, no cooperatve percepton s actually performed, but each UAV sends to the Control Centre the poston of the alarms. Two dfferent data sources come from the UAVs: mages and data from the fre sensor Detecton of fre alarms n mages By usng the fre segmentaton and geolocaton algorthms of the AIIP subsystem, the UAVs equpped wth cameras provde drect estmatons of the locatons x a (k) and the covarance C a (k) of the fre alarms. These estmatons are complemented by the probabltes P D and P F assocated to the fre detecton algorthms. These values are used to compute the ntal probablty p (0) as: p P D ( 0) = (3) PD + PF The justfcaton of ths expresson wll be gven n Secton 4.2, where t wll be proven that the expresson consders an ntal probablty of fre at poston x a of value Detecton of fre alarms wth the fre sensor 19

20 The fre sensor provdes a scalar value ndcatng the presence of fre. Usng a threshold, ths value s used to obtan a Boolean value, s, ndcatng that a fre alarm s present. To estmate the poston of the alarm a grd-based localzaton technque s used. Each cell of the grd s assocated to an area of the searchng zone of the UAV centered at poston x. Cell s assgned wth a value, p(x k), that represents the probablty that fre alarm s present n ts area at tme k (see Fgure 11 left). The values of the grd are updated teratvely wth the new data gathered by the sensor. At tme k=0, wth no nformaton about the presence of fre alarms, all the cells are ntated wth p(x 0)=0.5. When a new measure s k+1 arrves, the condtonal probablty p(x k+1 s k+1 ) for each cell wthn the feld of vew of the sensor s computed. p(x k+1 s k+1 ) s the probablty of havng a fre alarm n cell condtoned to s k+1. p(x k+1 s k+1 ) s computed by usng the well-known Bayes rule: p( x p( s x ) p( x k ) 1 (4) p( x ) dx k + 1 k k + sk+ 1) = p( sk + 1 x k ) The sensor model p(s k+1 x k) used n (4), the probablty of havng the measure s k+1 gven a fre at locaton x, s also characterzed by the probabltes P D and P F of the sensor, as n Secton 3.1 for the mage-based detecton algorthms. The ntegral n (4) s a sum over the two possble states of cell (havng fre,.e. TRUE, or not,.e. FALSE). If s k+1 s TRUE (that s, a fre s detected n the feld of vew of the fre sensor), (4) becomes: p k k PD p( x k ) + 1 sk+ = TRUE) =, (5) P p( x k ) + P ( x k 1 whle f s k+1 =FALSE, then the update equaton s: p D F [ 1 p( x k )] (1 PD ) p( x k ) + 1 sk+ = FALSE) = (6) (1 P ) p( x k ) + (1 P )1 ( x k 1 D F [ p( x k )] The values of the cells of the grd wthn the feld of vew of the fre sensor are recursvely updated usng (5) and (6) as new data gathered by the fre sensor arrve. The feld of vew of the sensor s defned by a maxmum range and the horzontal and vertcal aperture angles (see Fgure 20