Augmented Crime Scenes: Virtual Annotation of Physical Environments for Forensic Investigation

Transcription

1 Augmented Crime Scenes: Virtual Annotation of Physical Environments for Forensic Investigation Andrew P. Gee Dept of Computer Science Walterio Mayol-Cuevas Dept of Computer Science P.J. Escamilla-Ambrosio INAOE, Mexico Andrew Calway Dept of Computer Science Matthew Webb Dept of Elec & Electronic Eng ABSTRACT We describe a system aimed at the in-situ 3D annotation of physical objects and environments by the use of augmented reality. The system integrates absolute positioning technology, in the form of GPS and UWB positioning, with realtime computer vision to create a virtual incident map. The virtual map is constructed collaboratively by multiple operatives and a remote control centre. We demonstrate the system operating in a test environment that covers indoor and outdoor areas and explain how the technology may be used to assist forensic investigators as they collect and process evidence in a crime scene. Mapped points GPS Visual mapping Camera In-Field Operative (IFO) Wireless data network UWB positioning Categories and Subject Descriptors H.5.1 [Multimedia Information Systems]: Artificial, augmented and virtual realities; H.5.3 [Group and Organization Interfaces]: Computer-supported cooperative work General Terms Design Keywords UWB, GPS, Visual SLAM, Augmented Reality 1. INTRODUCTION The documentation of the structure and appearance of physical environments is often a critical process in many applications. This is particularly true in areas of security and criminal justice in which efficient and judicious collection of This work was carried out while this author was within the Dept of Computer Science, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MiFOR 10, October 29, 2010, Firenze, Italy. Copyright 2010 ACM /10/10...$ Virtual map Control centre (CCC) Annotations Figure 1: System overview: multiple operatives equipped with GPS and UWB positioning sensors and a handheld visual SLAM sensor collaborate to construct a distributed virtual map of the environment with augmented reality annotations. data is paramount. One such example is in the area of crime scene investigations, in which forensic scientists are required to collect evidential data, such as finger prints, locations of objects, as well as topological and metric layouts. They are required to do so in a meticulous and efficient manner, with careful and traceable recording of findings, often within constrained time limits. Data collection at most crime scene investigations currently relies heavily on manual recording, in written, photographic and video form [13]. This is inevitably a time consuming task, especially when multiple personnel are operating within multiple teams over wide areas, and where evidential information needs to be fully accounted for and corroborated amongst the teams. It follows that technology capable of making this process more efficient would be a welcome addition to the toolbox of crime scene investigators. The development of one such tool is the aim of the work described in this paper.

2 Our focus is on efficient recording, representation and visualisation of information in the context of the physical environment. We are interested in developing systems whereby collected data can be directly associated with physical structure and visualised in that context, and also shared amongst participating users. The key motivation is that information about an environment can be more readily accessed and assessed when visualised in its spatial context, facilitating greater efficiency whilst maintaining the quality of data collection and logging. We address this through the use of Augmented Reality (AR) and the construction of Virtual Maps (VM) by investigators equipped with wearable computing systems. There are two important capabilities in the system: the logging of the 3-D location of salient information within an evolving shared VM of the environment; and the annotation of objects and structure in the environment using AR, allowing investigators to record and subsequently to visualise information in spatial context. As illustrated in the system overview in Fig. 1, this is achieved by integrating absolute positioning technology, in the form of GPS and Ultra-Wide Band (UWB) positioning, with vision based simultaneous localisation and mapping (SLAM), with the latter providing the localised AR annotation and visualisation capability. An ad-hoc wireless LAN then provides communication between investigators and a control centre, enabling the sharing of the virtual incident map. A significant factor in the work is the robustness of the vision based SLAM framework [5, 4], which enables localised mapping in previously unseen environments and reliable relocalisation when those environments are revisited. This provides the flexibility and tracking accuracy required for this form of AR based annotation and visualisation. As the results demonstrate, it enables the generation of rich wide area virtual maps, providing an efficient means of generating succinct representation of topology and environment content within a spatial context. In the next section we review related work in this area, followed in Section 3 by an overview of the system. The key components of absolute positioning and annotation and visualisation via visual SLAM are then covered in Sections 4, 5 and 6, including results obtained from a demonstration trial. The paper concludes with a discussion on the directions for future work. 2. RELATED WORK The use of new multimedia technologies in the gathering, analysis and presentation of forensic evidence has been considered by a number of authors. Burton et al. [2] describe how AR could be used to present in-situ reconstructions of crime scenes and Schofield [11] highlights additional applications in evidence collection, analysis and processing. Baber et al. have built a prototype wearable system to support evidence recovery using RFID tags and show promising results in user trials [1, 6]. The application of computer vision to forensic scene analysis has often focused on accurate photo-realistic reconstruction of crime scenes, e.g. [8, 12]. Constructing these types of models is not the aim of the system described in this paper. Instead, we are interested in the fusion of visual SLAM techniques [5, 7] with absolute positioning technology in order to produce a shared VM containing AR annotations that can be used to assist evidence collection, analysis and process- GPS UWB (a) Internal Touchscreen Batteries Laptop (b) Worn GPS UWB Camera Figure 2: Hardware setup: (a) GPS and UWB units are carried inside a backpack with batteries; (b) the laptop is carried in an external pouch on the backpack, GPS and UWB antennae are mounted in locations which provide a relatively unobstructed field of view, and the camera is mounted on the front of the touchscreen. ing in the ways described in [2, 11]. In this way, our work is similar to that of Castle et al. [3] who construct local SLAM maps in areas of interest and use them to present AR annotations to users. However, their system does not incorporate any absolute positioning sensors, so the relative location of the SLAM maps and the overall structure of the environment is not known. 3. SYSTEM OVERVIEW Figure 1 shows the high level system structure. The In- Field Operatives (IFOs) are responsible for collecting sensor observations as they move around and interact with the environment. Each IFO is equipped with a touchscreen and backpack containing a camera, UWB and GPS sensors and a laptop computer, as shown in Fig. 2. These sensors provide absolute 3-D localisation of the IFO using UWB (indoors) or GPS (outdoors) and accurate relative localisation of 3-D position and orientation of the handheld device using the camera to run visual SLAM. The observations generated by the IFO are stored in a local database for immediate processing and display to the local user. A Central Control Centre (CCC) manages the flow of data between the IFOs over a wireless LAN, enabling each IFO to view the data collected by the other IFOs. The OGC Sensor Observation Service (SOS) standard [10] provides a standardised interface for sharing observations throughout the system. This enables observations to be exposed to other devices and applications outside the system as well as simplifying the process of defining new sensors and technologies. The CCC acts as a central interface for external access to observations and controls the flow of information around the system to enable it to adapt to changing network conditions. In addition to providing continous localisation of the IFOs, the system enables users to map or annotate an object or region of interest which will subsequently be highlighted using AR on any IFO s touchscreen terminal as well as displayed in the CCC. The workload of the distributed mapping process is shared between the team of operatives, facilitating rapid mapping of the environment. Users who enter the scene are immediately provided with the latest version of the map and are able to view the structure and annotations that have been added by other users in their correct context.

3 IFO AR annotation Window annotation Wall annotation Door annotation Figure 3: Reference image of indoor area of demonstrator site. Locations of UWB base units and WiFi router are indicated by white circles and ellipse respectively. 4. POSITIONING IN 3-D WITH UWB/GPS The system is designed to take advantage of both indoor and outdoor positioning technologies. Outdoor positioning uses a Teseo GPS chipset to provide 3-D positioning accurate to 2m with a 50% confidence limit. Indoors, a UWB positioning system provides 3-D positioning [9]. Switching between the UWB and GPS systems is transparent to the users: in areas where observations from both sensors are available, UWB is preferred for reasons of accuracy. In the system described in this paper, the UWB base units have been surveyed into the building but alternative modes using dropped units to extend the UWB network are also possible. Figure 3 shows a reference image of the demonstrator site and the UWB base unit locations. In a typical indoor environment, the UWB system provides 3-D positioning to at least metre-level accuracy. This accuracy varies according to the coverage of the UWB base units, which is affected by obstructions in the lines of sight between units and reflective surfaces in the environment that add multipath effects. The absolute positioning provided by the UWB/GPS systems is used to locate the IFOs and construct a rough access map indicating the areas that have been covered by their movements as they explore the incident site. This information is used to coordinate the IFOs from the CCC and may also be useful for identifying areas of the site which have not yet been processed. 5. AUGMENTING THE ENVIRONMENT In addition to providing an access map of the site, the system also provides the operatives with tools to augment the map. Operatives are able to sketch the internal structure of a building by adding walls, doors and windows to the virtual map. The aim of this process is not to generate a highly accurate model of the building but to provide contextual information which can help the operatives to navigate around the incident site. Figure 4 shows an example of the type of VM that can be created. In locations where more accurate mapping is required, the VM is supplemented by local Figure 4: Example virtual map. The background map image is a low resolution rasterised version of Ordnance Survey MasterMap vector data covering the demonstrator site. A low resolution occupancy grid access map is generated as the users move around the floorspace. Structural annotations have been added to the map to provide context and structure indoors. AR annotations have been added to the map indoors and outdoors using UWB and GPS respectively for absolute positioning. maps containing AR annotations which highlight important objects and evidence in the incident site. 5.1 Sketching Internal Structure Internal structural features, such as walls and doors, are added semi-automatically to the map by the operatives. An operative stands next to the point of interest and clicks on a button in the user interface that selects the type of structure to be inserted in the map. This records the current absolute position of the operative, as measured by the UWB system, and inserts the chosen structure at this location. Figure 5 shows an example of this process. A team of operatives working collaboratively can rapidly generate a VM containing the essential indoor features. The accuracy of the VM is limited by two main factors: the accuracy of the UWB system and the ability of the operatives to place their UWB sensor in the precise locations of the structures being mapped. However, since the map is solely used to provide context to the human operatives, metre-level accuracy is sufficient to generate a map with useful information. Figure 6 provides a comparison between the generated VM and an architectural plan of the demonstrator site. In addition to annotating the VM with physical structures, the operatives are also able to annotate it with images captured from the camera. Images can be stamped with the 3-D location of the operative taking the photograph and placed into the VM. All of the structural and image annotations are stored in the observation database and distributed to other operatives. 5.2 Annotating Important Objects AR annotations can be inserted into the VM by operatives in order to add information and highlight important locations, as shown in Fig. 7. To insert an annotation, an operative stands facing the area of interest and creates a lo-

4 Insert screenshot Insert door Insert window Insert wall (a) User interface (b) Mark start of wall (c) Mark end of wall (d) Insert wall (e) Insert door (f) Insert window Figure 5: Construction of virtual map: (a) the user interface contains a set of buttons that insert structural features; (b)-(d) linear structures, such as walls, are inserted into the map by marking the start and end points of the structure using the UWB positioning sensor on the handheld device; (e)-(f) point structures, such as doors and windows, are added immediately at the operative s current location. (a) Hammer (b) Door handle Figure 6: Comparison between virtual map and architect s plan of demonstrator site. Overall alignment between the maps is good enough to provide useful contextual information for the indoor AR annotations and is within the expected margin of error of the UWB tracking system. Small offsets are observed where the operative was unable to stand at the precise location of the structure or where the coverage from the UWB was poor. Note that the access map is correctly limited to the balcony and stairs regions of the plan (see Fig. 3 for reference image). (c) Mobile phone (d) Bin Figure 7: Selection of different AR annotations that have been added to the map. Annotations are scaled to match the object of interest and a suitable text label is added. The visual SLAM system operates indoors and outdoors, enabling annotation of objects in both environments.

5 cal visual SLAM map of the region by moving the camera smoothly around the scene. The visual SLAM sensor [5] provides accurate local pose estimation in real time and the new local map is registered in the VM using the UWB/GPS absolute position of the device. The accurate pose estimation in the local map enables annotations to be inserted using triangulation. A crosshair in the centre of the camera image defines the direction of a 3-D semi-infinite ray from the optical centre of the camera through the targeted point in the image plane. Selecting a point of interest from several different views creates a set of intersecting rays and least-squares optimisation is used to define the 3-D position of the point. Figure 8 shows an example of this process. Once a 3-D point in the local map has been defined, any desired annotation can be attached to the specified location. In the current system, a red circle is used to draw the users attention to the point of interest and a text label is attached to provide information about the feature. However, it would also be possible to attach animated 3-D models or any other type of multimedia content that is supported by the underlying software application. Finally, in order to share the new annotation with the other operatives, the local visual SLAM map and the annotation are packaged into observations and inserted into the observation database. It is worth noting that the current system only registers the 3-D translation between the local map origin and the VM origin. This allows the location of the local map to be marked in the VM but leaves an unknown rotation and scaling factor between the two coordinate frames. In theory, it is possible to determine a similarity transformation between the coordinate frames by aligning the camera trajectory in the local map with the UWB/GPS trajectory in the VM. However, this does not always work well in practice because the lengths of the camera trajectories generated in the local maps are of similar magnitude to the error in the UWB/GPS measurements. In practice, we avoid this problem altogether by allowing the user to adjust the scale and orientation of the annotation manually. This does not increase the workload of the user significantly, since it is likely that they would make these types of adjustments anyway. 6. RELOCALISING AND VISUALISING The VM provides contextual information and absolute positioning data that can assist other operatives to navigate to the approximate location of new AR annotations. A screenshot of the annotation provides the operative with a visual cue to point the camera in correct general direction and the relocalisation capability of the visual SLAM system [4] quickly restores full pose estimation when features in the local map become visible. This enables the annotations to be visualised through an AR interface and viewed in context in their relevant location. The annotations can also be amended and added to by other operatives and these changes are distributed through the system. Figure 9 shows an example of this process. When an operative views an AR annotation for the first time, the system automatically loads the relevant visual SLAM map associated with that annotation from the observation database. As the operative explores the area around the annotation with the camera, the local version of the SLAM map is extended normally with new visual features to maintain tracking. This allows the operative to view the annotation from novel viewpoints that were not mapped by Target annotation IFO (a) Navigate to AR marker (c) Visualise annotation (b) Relocalise (d) Relabel annotation Figure 9: Sequence of images showing visualisation of an AR annotation: (a) the virtual map is used to navigate to the location of the AR annotation; (b) the view through the camera shows that the SLAM system needs to relocalise and the thumbnail screenshot provides a visual cue to help the operative position the camera; (c) after relocalisation the annotation can be visualised in its correctly registered location; (d) AR annotations can be modified by other operatives and changes are distributed through the system, for example this annotation has been relabeled. the creator of the annotation. However, these local updates to the map are not shared with the other operatives. This avoids the potentially complex problem of maintaining synchronised versions of the SLAM maps between all of the operatives, but places some additional responsibility onto the initial creator of the map to ensure that it covers a reasonable default set of viewpoints to assist initial relocalisation in the map. One potential problem associated with using visual features for localisation is that the performance will degrade or even fail if the structure of the scene is significantly modified with respect to the original map. However, in practice, distinctive visual descriptors and robust data association techniques provide good protection against small changes, such as the movement, addition or removal of objects, as long as the majority of the scene remains consistent. This also means that the AR annotations could continue to mark the location of pieces of evidence after the evidence has itself been removed from the scene. 7. CONCLUSIONS This paper presents a system for efficient recording, representation and visualisation of information in the context of the physical environment. A robust visual SLAM framework provides accurate registration of AR annotations with the real world and these are combined with absolute positioning technologies to create a virtual incident map that can be used for coordination and navigation. The system supports multiple users and can be used to assist in the collection and processing of forensic data. We demonstrated the system by building a virtual map covering an indoor test site and extending outdoors. Future work should include a user

6 (a) User interface (b) Create SLAM map (c) Define first ray (d) Define second ray AR annotation (e) Insert annotation (f) Resize and relabel (g) Update screenshot (h) Marker added to map Figure 8: Sequence of images showing definition of 3-D annotation: (a) the user interface - as seen from the touchscreen - is switched to show the view through the camera; (b) a new SLAM map is built by moving the camera smoothly around the area of interest (green and red circles indicate visual features that have been added to the SLAM map); (c)-(e) aligning the crosshair in the centre of the image with the object of interest from two different viewpoints triangulates its position and enables an AR annotation to be added to the map (red circle); (f) the annotation is resized to adjust for the unknown scale factor and an appropriate text label is selected; (g) the thumbnail screenshot image of the annotation is updated to capture the new information; (h) a new AR annotation marker appears in the VM. assessment to validate the approach and may also consider the incorporation of inertial sensors and a digital compass to provide absolute orientation and improve the estimation of absolute position in areas with poor UWB/GPS coverage. 8. ACKNOWLEDGMENTS This work was funded by the UK Technology Strategy Board and the UK Engineering and Physical Sciences Research Council. The authors wish to thank all partners in the ViewNet project for their discussions and participation c Crown copyin this work. Ordnance Survey mapping right. 9. REFERENCES [1] C. Baber, P. Smith, J. Cross, D. Zasikowski, and J. Hunter. Wearable technology for crime scene investigation. In Proc. IEEE Int. Symp. on Wearable Computers, [2] A. Burton, D. Schofield, and L. Goodwin. Gates of global perception: forensic graphics for evidence presentation. In Proc. ACM Int. Conf. on Multimedia, [3] R. Castle, G. Klein, and D. Murray. Video-rate localization in multiple maps for wearable augmented reality. In Proc. IEEE Int. Symp. on Wearable Computers, [4] D. Chekhlov, W. Mayol-Cuevas, and A. Calway. Appearance based indexing for relocalisation in real-time visual SLAM. In Proc. British Machine Vision Conf., [5] D. Chekhlov, M. Pupilli, W. Mayol-Cuevas, and A. Calway. Real-time and robust monocular SLAM using predictive multi-resolution descriptors. In Proc. Int. Symp. on Visual Computing, [6] J. Cross, C. Baber, and P. Smith. Multi-platform crime scene investigation field tool. In Proc. IEEE Int. Symp. on Wearable Computers, [7] A. Davison, I. Reid, N. Molton, and O. Stasse. MonoSLAM: real-time single camera SLAM. IEEE Trans. on Pattern Analysis and Machine Intelligence, 29(6), [8] S. Gibson and T. Howard. Interactive reconstruction of virtual environments from photographs, with application to scene-of-crime analysis. In Proc. ACM Symp. on Virtual Reality Software and Technology, [9] D. Harmer, M. Russell, E. Frazer, T. Bauge, S. Ingram, N. Schmidt, B. Kull, A. Yarovoy, A. Nezirovic, L. Xia, V. Dizdarevic, and K. Witrisal. EUROPCOM: emergency ultrawideband radio for positioning and communications. In Proc. IEEE Conf. on Ultra-Wideband, volume 3, [10] A. Na and M. Priest. OpenGIS sensor observation service. Technical Report OGC r6, Open Geospatial Consortium, Inc., [11] D. Schofield. Animating and interacting with graphical evidence: bringing courtrooms to life with virtual reconstructions. In Proc. Int. Conf. on Computer Graphics, Imaging and Visualization, [12] S. Se and P. Jasiobedzki. Instant scene modeler for crime scene reconstruction. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, [13] P. White, editor. Crime scene to court: essentials of forensic science. Royal Society of Chemistry, 4th edition, 1998.