RPV/UAV Surveillance for Transportation Management and Security

RPV/UAV Surveillance for Transportation Management and Security Final Report Prepared by: Demoz Gebre-Egziabher Department of Aerospace Engineering & Mechanics University of Minnesota CTS 08-27

Technical Report Documentation Page 1. Report No. 2. 3. Recipients Accession No. CTS 08-27 4. Title and Subtitle 5. Report Date RPV/UAV Surveillance for Transportation Management and Security December 2008 6. 7. Author(s) 8. Performing Organization Report No. Demoz Gebre-Egziabher 9. Performing Organization Name and Address 10. Project/Task/Work Unit No. Department of Aerospace Engineering & Mechanics University of Minnesota 107 Akerman Hall 110 Union St SE Minneapolis, MN 55455-0153 CTS project # 2005040 11. Contract (C) or Grant (G) No. 12. Sponsoring Organization Name and Address 13. Type of Report and Period Covered Intelligent Transportation Systems Institute University of Minnesota 200 Transportation and Safety Building 511 Washington Ave. SE Minneapolis, Minnesota 55455 15. Supplementary Notes http://www.cts.umn.edu/publications/researchreports/ 16. Abstract (Limit: 250 words) Final Report 14. Sponsoring Agency Code This report describes the results of an investigation into some of the technical and operational issues associated with using Unmanned Aerial Systems (UAS) for the application of surveillance in support of transportation infrastructure management and security. As part of this investigation a low-cost, miniature, hand-launched aerial vehicle and supporting ground systems suitable for surveillance of highways and traffic infrastructure were developed. Except for the ground station software, this system was built from off-the-shelf components. The ground station software developed was used to enhance ground station operators' situational awareness and simultaneously allow analysis of the data transmitted from the aerial vehicle. In addition, a key system that was developed was an open-source Guidance, Navigation and Control (GNC) software suite for autonomous operation of small aerial vehicles. The culmination of this work was a series flight tests where the UAS developed was used as a tool to enhance situational awareness over a simulated traffic incident or emergency situation. The test consisted of defining a series of waypoints around the area of the simulated incident and launching the miniature aerial vehicle to autonomously fly from waypoint to waypoint. 17. Document Analysis/Descriptors 18. Availability Statement UAV, UAS, RPV, ROV, Integrity, Risk, Concept of Operations No restrictions. Document available from: National Technical Information Services, Springfield, Virginia 22161 19. Security Class (this report) 20. Security Class (this page) 21. No. of Pages 22. Price Unclassified Unclassified 37

RPV/UAV Surveillance for Transportation Management and Security Final Report Prepared by Demoz Gebre-Egziabher Department of Aerospace Engineering & Mechanics University of Minnesota December 2008 Published by Intelligent Transportation Systems Institute Center for Transportation Studies 200 Transportation and Safety Building 511 Washington Avenue S.E. Minneapolis, Minnesota 55455 The contents of this report reflect the views of the authors, who are responsible for the facts and the accuracy of the information presented herein. This document is disseminated under the sponsorship of the Department of Transportation University Transportation Centers Program, in the interest of information exchange. The U.S. Government assumes no liability for the contents or use thereof. This report does not necessarily reflect the official views or policy of the Intelligent Transportation Systems Institute or the University of Minnesota. The authors, the Intelligent Transportation Systems Institute, the University of Minnesota and the U.S. Government do not endorse products or manufacturers. Trade or manufacturers names appear herein solely because they are considered essential to this report.

Acknowledgement We would like to acknowledge the support of the ITS Institute at the University of Minnesota, the Minnesota Department of Transportation and SRF Consulting for sponsoring the work discussed in this report. However, the views expressed in this report belong to the author alone and do not necessarily represent the position of any other organization or person. We also wish to acknowledge those who made this research possible. The study was funded by the Intelligent Transportation Systems (ITS) Institute, a program of the University of Minnesotas Center for Transportation Studies (CTS). Financial support was provided by the United States Department of Transportation Research and Innovative Technologies Administration (RITA). The author of this report would also like to acknowledge the work of the students (Romeo Ahohe, Paw Yew Chai, Rohit Pandita, Troy Wigton and Zhiqiang Xing) and research staff (Curtis Olson and Greg Nelson) that were part of this project. The results discussed in this report are based on algorithms and hardware developed by these individuals.

Contents 1 Introduction 1 1.1 Report Organization.................................... 1 2 UAS in ITS Applications 2 2.1 Examples of Strategic Applications............................ 2 2.2 Example of Tactical Applications............................. 3 3 Technical and Operational Challenges 5 4 Project Objectives And Scope 7 4.1 Project Budget...................................... 8 5 Tasks And Outcomes 9 5.1 Task 1: Develop an Autonomous Aerial Platform.................... 9 5.1.1 Airframe and Powerplant Selection....................... 9 5.1.2 Avionics Selection................................ 10 5.2 Task 2: Sensor Suite and Mission Payload Development................. 13 5.2.1 Mission Sensor Payload: EO Camera...................... 13 5.2.2 Data Link Modem................................ 14 5.2.3 Open-Source GNC Software: MicroGear.................... 15 5.3 Task 3: Ground Station Development........................... 18 5.4 Task 4, 5 and 6: Testing and Validation of System.................... 22 6 Summary 25 7 Future Research 26 7.1 Image Processing, Stabilizing and Feature Tracking................... 26 7.2 GPS Attitude Determination............................... 26 References 28

List of Tables 4.1 Work Time-Line...................................... 7 4.2 List of Tasks........................................ 8

List of Figures 5.1 Aerial Vehicle Unicorn................................... 10 5.2 Method for Launching Unicorn.............................. 11 5.3 MicroPilot MP2028 Autopilot............................... 11 5.4 Microbotics MIDG II Integrated GPS/Inertial Navigator................. 12 5.5 Schematic of Guidance, Navigation and Control Avionics for Unicorn.......... 13 5.6 Camera Pod or Blister for Mounting Cameras on Miniature Aerial Vehicles...... 14 5.7 Block Diagram of MicroGear Navigation, Guidance and Control System........ 16 5.8 Crossbow MicroNav GNC Hardware. The MicroGear Open-Source Software Used the MicroNav Sensors................................... 17 5.9 Ground Station System Schematic............................. 18 5.10 Ground Station Shown in Set-Up in a Laboratory..................... 19 5.11 Screen Capture of a Video from a Flight of the Unicorn over Jensen Field, Minnesota.. 20 5.12 MicroGear s Depiction of the Screen Capture Shown in Figure 5.11. Flight Instruments Showing Speed, Altitude, etc are Shown at the Bottom of the Display....... 21 5.13 Side by Side Comparison of Live Video and a MicroGear Generated Display...... 21 5.14 Another MicroGear Depiction of the Scene Captured in Figure 5.12. Boundaries of the Minneapolis Class-B Airspace is Shown........................ 22 5.15 Ground Track of an Autonomous Flight Enabled by MicroGear............. 23 5.16 MicroGear Depiction of the Trajectory Shown in Figure 5.15............... 24

Executive Summary This report describes the work done at the University of Minnesota to investigate the technical and operational issues associated with using Unmanned Aerial Systems (UAS) for the application of surveillance in support of transportation infrastructure management and security. The outcomes of this work are: 1. Development of a low-cost, miniature, hand-launched aerial vehicle and supporting ground systems suitable for surveillance of highways and traffic infrastructure. Except for the ground station software, this system was built from off-the-shelf components. 2. Development of software that enhances ground station operators situational awareness and allows simultaneous analysis of the data transmitted from the aerial vehicle. 3. Development of an open-source Guidance, Navigation and Control (GNC) software suite for autonomous operation of small aerial vehicles. The third outcome was the result of a study to identify the regulatory and technical issues that must be solved before UAS can be routinely used for surveillance in support of transportation infrastructure management and security. One of the conclusions of this effort was that certifying UASs will be easier if certifying agencies have influence over the design of GNC algorithms especially features having to do with fault detection and isolation. An open-source GNC software suite can facilitate building regulatory requirements for GNC systems. The culmination of this work was a series flight tests where the UAS developed was used as a tool to enhance situational awareness over a simulated traffic incident or emergency situation. The test consisted of defining a series of waypoint around the area of the simulated incident and launching the miniature aerial vehicle to autonomously fly from waypoint to waypoint.

Chapter 1 Introduction This report describes the work done at the University of Minnesota to investigate the technical and operational issues associated with using Unmanned Aerial Systems (UAS) for surveillance in support of transportation infrastructure management and security. The work was accomplished under a research grant from the ITS Institute at the University of Minnesota and a research contract from the Minnesota Department of Transportation (via a subcontract from SRF Consulting Inc.). The objectives of this work were twofold. First, it was to develop an UAS from off-the-shelf components (i.e., develop a turn key solution ). The second objective was to identify technical and regulatory issue that need to be addressed before UAS can be routinely and effectively used in Intelligent Transportation System (ITS) applications. The work associate with the first objective resulted in the following outcomes: Development of a low-cost, miniature, hand-launched aerial vehicle and supporting ground systems. Except for the ground station software, this system was built from off-the-shelf components. Development of software which enhances ground station operators situational awareness and allows simultaneous analysis of the data transmitted from the aerial vehicle. The work associated with the second objective resulted in the following outcomes: Development of an open-source Guidance, Navigation and Control (GNC) software suite for autonomous operation of small aerial vehicles. 1.1 Report Organization In what follows the outcomes and the findings of this research project are discussed in detail. To that end, the remainder of this report is organized as follows: First, a detailed discussion of the motivation for using UAS in these applications is presented. The technical and regulatory challenges associated with this application are discussed. Then each project task and its outcome is discussed separately. Concluding remarks and suggestions for future work close the report. 1

Chapter 2 UAS in ITS Applications Uninhabited Aerial Vehicles (UAVs) are defined as aircraft which fly without a human operator onboard. They can be flown autonomously or piloted remotely. If they are piloted remotely, they are sometimes referred to as Remotely Piloted Vehicles or RPV. They are designed as either expendable or recoverable platforms [1]. They are well-suited for use as sensor payload platforms to gather remotely sensed data. They span a wide range in size and complexity with some of the largest UAVs being several thousand pounds in weight and have wing spans on the order of 100 feet [2, 3]. At the other end of the size spectrum are the Micro Aerial Vehicles (MAVs) which have a maximum dimension and weight on the order of 15 cm and 150 grams, respectively [2, 3]. A UAV (or MAV) together with the systems on the ground used to interact with it are normally referred to as an Uninhabited Aerial Systems (UAS). While it is not difficult to imagine military missions for UAS, there are equally many potential civilian and scientific applications [4, 5]. Envisioned civilian applications include forestry surveys, environmental monitoring and power-line inspections. These envisioned applications can be divided into the two broad categories that we in this report call strategic and tactical operations. In simple terms, we define strategic operations to be operations where the aerial vehicle is expected to traverse or cover a large geographical area. These operations are mostly pre-planned and the separation distance between the aerial vehicles and the operators can be very large. This implies that the round-trip travel time for signals between the ground station and the aerial vehicle can range from several milliseconds to a few seconds. Since delays of this magnitude can be dangerous or destabilizing when it comes to control of aerial vehicles, UAS slated for strategic operations must be capable of a high level of autonomy. On the other hand, we define tactical operations are defined to be operations in and around a small geographical area. Operation can be in response to planned or unplanned events. Because of the proximity of the aerial vehicle to the ground operators, it is possible for that tactical operations to entirely conducted by teleoperation (i.e. non-autonomous). In the case of Intelligent Transportation System (ITS) application, the following two applications demonstrate the difference between what we call strategic and tactical operations: 2.1 Examples of Strategic Applications A strategic ITS application for which the use of UAS is envisioned is that of mobile sensor networks. The mobile networks will augment installed or stationary sensor networks in gathering data about highway infrastructure [6, 7, 8]. The economic and social motivation for using UAVs in this 2

application is very compelling. For example, if we consider the state of Minnesota as a typical example, almost 50% of the traffic (measured in miles traveled) is on rural roads (currently 230 thousand rural lane miles), and is increasing at an annual rate of 6% a year. In addition, approximately 70% of all fatalities occur on rural roads (the number of crash related fatalities and rural percentages have remained approximately flat over the last five years) [9, 10]. Rural Minnesota winter weather can present especially hazardous driving conditions-leaving motorists stranded for extended periods. In order to ensure driver safety and rapid incident response, considerable efforts have been made to place ITS traffic sensing and surveillance technologies on specific corridors of Minnesota s rural roadways. Whether they are permanent installations or placed on movable trailers, they have become an integral part of managing traffic during special events, road construction and work zones, etc. However, these systems, while still necessary, cannot by on their own provide wide area coverage in a cost-effective manner. The power or other communications infrastructure may not be available. It is simply not practical to manage and build that much infrastructure. Remotely sensed data gathered from moving platforms such as UAS can address the shortcoming encountered with the traditional method of data collection. UAS used in this manner are a complimentary technology to ground based cameras and other ITS technologies. They can be flown over land and traffic directly to problem areas. In doing so, they fill large coverage holes, providing a big picture of traffic flow over large portions of the road network. While manned aircraft can be used in lieu of UASs for this application, there are economic and operational advantages which make the latter a more attractive option. For example, the use of UASs obviates the need for regional airport to house and provide the landing-takeoff-fueling cycles required by manned assets. With adequate 3D feature and terrain knowledge, they may be safely flown at much lower altitudes than manned aircraft (such is the case during low cloud ceilings, etc). This operation is considered strategic because the coverage holes of interest may be far away from the site where the UAS is launched or where the remote operators of the system are located. 2.2 Example of Tactical Applications An example of a tactical ITS application for UASs is that of on-scene traffic, incident or emergency management. In the event of a traffic incident or general emergency, UASs can be used as force multipliers for law enforcement and emergency response personnel. In these instances the aerial vehicles can provide services such as mobile-local communication nodes or simply be platforms for Electro- Optical (EO) or Infrared (IR) camera. The recent collapse of the I-35 Bridge in Minneapolis, MN provides an example where the use of UASs would serve as a force multiplier for law enforcement and emergency personnel. In this case the UAS could have been used to provide platform for continuous video coverage. This coverage could be used to optimize and organize the rescue and recovery efforts; provide continuous communications between on-site personal and those far away; and finally as camera platforms for photogrammetry reconstruction of the incident. In the case of terrorist attacks that involve, for example, toxic materials appropriately instrumented UAS can be used to map out the boundary of the toxic materials and monitor their spread. Thus, they will prevent the risking the lives of highly skilled personnel that would otherwise have to fly aircraft especially under hostile or toxic conditions. It should be noted that UAS in this instance fit within the recent framework that addresses transportation security and anti-terrorism measures. This framework, published by the Transportation Security Agency (TSA), emphasizes harnessing dual-use technologies that meet the needs of our trans- 3

portation systems as well as for homeland security roles and law enforcement [11, 12]. The use of UAVs in the above described manner assumes that UAVs can be safely and legally operated in and around populated areas as well as potentially alongside other passenger carrying aircraft in the National Airspace System (NAS). This, of course, is not the case. It is not clear that UAS operations do not pose an unacceptable level of risk to manned aircraft in the NAS or the population on the ground. 4

Chapter 3 Technical and Operational Challenges Currently, there is no clear and systematic legal or regulatory frame work which will allow operation of UAS in the manner described above. That is, while there has been a considerable effort focused on identifying operational concepts which use UASs for traffic management and infrastructure security[6, 7, 8]. As such, there is a considerable amount of research currently underway to develop of operational procedures which ensure safe operation of these vehicles in and around populated areas or the NAS [19]. An important subset of the above mentioned work has focused on developing systematic methods for quantifying risks or dealing with certification issues. Up to this point the analysis of risk associated with these operations has been limited to developing and evaluating methods for preventing collisions between UAVs and other manned aircraft [13]. Recently, researchers at the MIT International Center for Air Transportation have proposed a very promising approach for evaluating safety and risk mitigation measures associated with operation of UAVs [14, 15]. The work in [14] and [15] provides a systematic method for analyzing a given UAVs concept of operation to determine if it meets levels of safety established by regulation. However, while the work in [14] and [15] provides general guidelines on how to perform such evaluations, it does not does not provide an evaluation of any particular concept of operation. Related to the challenge of developing operational procedures is a technical challenge of proving that the risk posed by these operations is acceptably small. For aviation operations the regulatory safety requirements are established by the Federal Aviation Administration (FAA) and are normally given as statistical performance metrics. In this context, risk is measured, in part, by the metric known as integrity Systems are deemed to have an acceptable level of integrity if the risk of hazard to humans when using them is less than some prescribed threshold (for example, one-in-ten-million for certain landing systems [16]). The difficulty of the integrity problem can be understood by noting the following: The smaller (size and weight) aerial vehicles are, the risk associated with their operation is less. Thus, small, handlaunched aerial vehicles represent an attractive solution. This in turn, presents a unique challenge which is not encountered with other UAVs. With regard to navigation, guidance and control of these vehicles, there are two major challenges. First, while it is possible to adapt and use avionics developed for piloted aircraft in the larger UAVs, new and miniaturized systems have to be developed for smaller UAVs and especially MAVs. There are currently very few off-the-shelf solutions for navigation, guidance and control of small UAVs that are certified to the exacting FAA integrity standards. Second, because of the size and weight constraints, the physical integration of miniature systems and sensors 5

represents a significant challenge. At these scales the sensors and mission payloads have dimensions and weight is a significant fraction of the overall vehicle dimension and weight, respectively. The objective of the work described in this report was, in part, to determine whether any meaningful statements about integrity can made for off the shelf avionics for small UAVs without extensive analysis. 6

Chapter 4 Project Objectives And Scope As noted earlier, the work described in the report investigated aspects of both the operational and technical challenges with UASs. In support of this objective, the work in this project was organized around the seven tasks shown in Table 4.2. The outcome of this project was a prototype flying platform, sensors, algorithms and operational procedures which enable autonomous monitoring of traffic and highway infrastructures and relay information in real-time to remotely located decision makers. The scope of this work did not include developing new and novel sensing or aerial platform technologies. Instead, it was to demonstrate the effectiveness and practicality of using RPVs or UAVs to monitor traffic and highway infrastructure. The scope of this work, therefore, was limited to modifying an existing aerial platform as required and to instrument it with proven technologies for imaging and geo-referencing. Stated differently, the focus of this work was to identify existing sensors, systems and miniature aerial platforms; show how they can be used to construct an aerial surveillance platform; and demonstrating the capabilities of such a platform. It should be noted that the systems and procedures developed as part of this work are be dual-use technologies which have direct relevance to homeland security applications. Furthermore, it should also be noted that the final product of the work was a demonstration of traffic and highway infrastructure monitoring using UASs and not the equipment for accomplishing this function. Therefore, the hardware and software developed as part of this project was not designed to be put operational service by law enforcement or local government agencies. Instead, the systems developed will be a prototype of what will eventually become a turn-key solution to the problem of traffic and highway infras- Task Duration/Months 1 2 3 4 5 6 7 8 9 10 11 12 1 X X X X 2 X X X X X X 3 X X X X X X X X X 4 X X X 5 X X X X X X X X X 6 X X 7 X Table 4.1: Work Time-Line 7

Task Description Duration 1 Develop the RPV Platform 4 Months 2 Sensor Payload/Suite Development 6 Months 3 Ground Station Development 9 Months 4 Operational Test of Integrated System 3 Months 5 Operational & Test Procedure Development 11 Months 6 Validation Experiment 2 Months 7 Write Final Report 1 Months Table 4.2: List of Tasks tructure monitoring via RPVs and UAVs. It is expected that more work and effort will be required to convert the systems developed into a final product for operational use. Except for Tasks #4, #6 and #7 all proceeded concurrently during the project. This is clearly shown in the work time-line shown in Table 4.1. 4.1 Project Budget The total projet budget was divided into the two categories: (1) Budget for purchase and fabrication of hardware and (2) Personnel salary and other expenses. At the end of the project on December 31, 2007, approximately $21 K of the portion allotted for equipment fabrication and hardware remained. These are funds that were set aside to purchase and fabricate an image processing, stabilizing and feature tracking system/software. These remaining funds are discussed later in the report in relation to recommended future work. 8

Chapter 5 Tasks And Outcomes In what follows, a detailed description of each task and its associated outcome or deliverable is given. Along the way, we will also briefly discuss our findings regarding issues that need further study. 5.1 Task 1: Develop an Autonomous Aerial Platform The objective of this task was to develop an aerial platform that is compact, robust and easy to operate. While it may have been possible to develop an aerial vehicle design that is unique to this specific project (and this was the case with a sister project discussed in [22]), it was deemed less risky (from a project schedule point of view) to purchase an existing aerial vehicle kit and modify it. Important constraints that led the selection of the kit and final modification were that the vehicle had to be compact (i.e., small in size and weight), stable in flight and easy to operate. The first of these constraints (compactness of size and weight) originates from the fact that the aerial vehicle must be easy to move from location to location as dictated by the traffic monitoring functions at hand. Furthermore, the final users of this aerial vehicle will more than likely be government or law enforcement officials. Thus, this means that the aerial vehicle should be able to fit in vehicles used by law enforcement officials such as a squad car or a van. The second constraint (stability of flight) resulted from the fact that one of the important payloads on such miniature aerial vehicles is an Electro-Optical (EO) camera. This is because one of the primary missions of these vehicles is that of being video-image gathering platform. Thus, the amount of video and image processing required to stabilize the images will be less if the camera is also relatively stable. Image stabilization and processing are issues that we believe warrant further examination and, thus, will be discussed in more detail at the end of the report. The third constraint (ease of operation) originates from the fact that the utility of such device increases if it does not require specially trained personnel to operate it. Furthermore, the less complex it is to operate means that the potential for operational errors that can be hazardous (e.g., loss of control of the vehicle) will be minimized. This requirement is at the heart of the issues surrounding regulatory requirements for the operation of these vehicles. 5.1.1 Airframe and Powerplant Selection In view of these design constraints, the final platform selected to meet all these requirements was a kit manufactured by Unicorn Wings [17]. The kit airplane is known as the Unicorn. This aerial 9

platform kit has been used in related projects at other universities including a very successful US Air Force sponsored project at Brigham Young University [18]. The Unicorn is shown in Figure 5.1 and it can be seen that it is a flying wing design. As shown in the photograph of Figure 5.2, it is easy to launch manually. Figure 5.1: Aerial Vehicle Unicorn. 5.1.2 Avionics Selection Central to autonomous or remote operation of an aerial vehicle is a Guidance, Navigation and Control (GNC) system. The GNC system generates high bandwidth information about the aerial vehicle s navigation state vector. The navigation state vector consists of the position vector, velocity vector and three attitude angles (i.e., yaw, pitch and roll) In the course of the project, it was determined that there was not a single off-the-shelf system that would address all the GNC system requirements. Thus, the aerial vehicle we designed used two off-the-shelf GNC systems: the MicroPilot MP2028 and the Microbotics Inc., MIDG II. MicroPilot Model MP2028 The MicroPilot MP2028 is an integrated GNC system shown in Figure 5.3. That is, it incorporates the guidance, navigation and control functions in one unit. With the MicroPilot, the user defines a trajectory that the aerial vehicle will follow via a series of waypoints. Once these are programmed into the MicroPilot, the aerial vehicle will follow the trajectory using information generated by a navigation and attitude determination system based on the fusion of GPS and inertial sensors. While the MicroPilot is an off-the-shelf GNC solution and was used extensively in the Unicorn for autonomous operations, there are some issues associated with using it in a fielded system. The primary 10

Figure 5.2: Method for Launching Unicorn. Micropilot MP 2028 Figure 5.3: MicroPilot MP2028 Autopilot. issue has to do with the fact that the software behind the GNC solution is proprietary and the user does not have access to it. Since the user does not have access to the internal workings of the MicroPilot, it is difficult to assess the level of safety and redundancy. Precisely quantifying the level of risk due to software performance, to a large extent, is dependent on the GNC algorithm s ability to detect and isolate faults in real-time. Assessing a GNC system s ability to do this is essential for certifying GNC systems. Thus, it is was difficult to assess the overall risk and robustness of the system. This was addressed, in part, by the work done in Task 2. This involved developing an open-source GNC software algorithm suite. 11

Microbotics Inc. MIDG II Shown in Figure 5.4, the MIDG II is an integrated GPS/Inertial Navigation System (INS) with an automatic pilot or control system extension which can be purchased separately. In this project we did not use the MIDG autopilot because the performance of MicroPilot controller was found to be satisfactory. The MicroPilot system was also less costly than the MIDG II-based system. However, the attitude solution provided by the MIDG II had higher accuracy and was more robust. This higher accuracy and robustness was required for the ground station interface software and geo-referencing. Figure 5.4: Microbotics MIDG II Integrated GPS/Inertial Navigator. The MIDG II generates a position, velocity and attitude solution by fusing the information derived from a triad of accelerometers, gyros and magnetometers with GPS. The result of this fusion is a relatively accurate and drift-free solution of position, velocity and attitude (orientation) of the aerial vehicle. However, the quality of the attitude solution is still considered marginal for precise georegistering applications. The architecture of the GNC system which features these two systems and was flown extensively in our flight tests is shown in Figure 5.5. Even though both systems were used simultaneously, only the MicroPilot was used required for guidance, navigation and control of the aerial vehicle. Ground station data visualization or operator interface software required the higher attitude accuracy afforded by the MIDG II. To ensure that a failure in the control system does not result in a hazardous situation, the aerial vehicle was equipped with a fail-safe system. This system allows the ground operators to isolate the control system in case it fails or an anomalous condition is detected. Once the control system is isolated, the ground operators can manually fly the aerial vehicle to safety. This system is shown at the center of the schematic in Figure 5.5 labeled Fail Safe System. This component is there because the control system must possess a high level of reliability since a failure of the system leads to an uncontrolled aerial vehicle. 12

GPS Antenna 1 MP2028 Servo controller Board MP2028 Flight Computer Board RC Receiver Data Modem 1 Data Modem 2 Fail Safe Board MIDG Sensor R Aileron Servo GPS Antenna 2 L Aileron Servo Video Camera Video Transmitter Motor Speed Controller Motor Figure 5.5: Schematic of Guidance, Navigation and Control Avionics for Unicorn. 5.2 Task 2: Sensor Suite and Mission Payload Development. In order to accomplish the function of traffic and highway infrastructure monitoring, the miniature aerial vehicle had to be equipped with the appropriate mission sensor payload. For visual surveillance the mission payload is an Electro-Optical (EO) camera. For some law enforcement operations the EO camera may be replaced by an Infrared (IR) camera. To support processing of the data collected from the mission sensor payload, the miniature vehicle must be equipped with a data-link modem. The GNC system generates information about the miniature vehicle s state vector (position, velocity and orientation or attitude) which is used to aid in actively pointing these cameras at the right target. Alternately, in a passive mode of data gathering the navigation state vector is used for geo-referencing the images captured by the onboard camera (i.e. know the location and orientation of features in the captured image relative to some coordinate frame fixed to Earth) and, thereby, facilitate displaying them to the decision makers at the remote site. Finally, all the data collected by the aerial vehicle is transmitted to the ground station for visualization, decision making and archiving. The transmission of the data is accomplished using a radio-modem or data-link. 5.2.1 Mission Sensor Payload: EO Camera In this project, the mission sensor payload was a video camera or an imaging system. The video imaging system consisted of two onboard cameras and associated electronics. The installation was such that the maximum coverage of the scene below the miniature aerial vehicle was be available at all times. Thus, the onboard cameras were be installed in the airframe as to provide the maximum Field of View (FOV). The video images captured by these cameras were transmitted in real-time to the ground station for visualization. The transmission of the video images was accomplished by using a pair of analog video transmitters. The cameras and video transmitters were integrated into a camera pod or blister which can be 13

easily attached or removed to the Unicorn aircraft. As a matter of fact, the camera (or mission sensor payload) pod is shown in Figure 5.6 attached to another airframe. When landing the Unicorn, the camera pod is the first point that comes into contact with the ground. As such, for strength, the pod was constructed from hard plastic. CAMERAS #1 AND #2 VIDEO DATA LINK ANTENNAS & DATA LINK TRANSMITTER Figure 5.6: Camera Pod or Blister for Mounting Cameras on Miniature Aerial Vehicles. 5.2.2 Data Link Modem Navigation data from the aerial vehicle are essential for the data visualization and operator interface. The programs for data visualization and operator interface that run on the ground station computer require the miniature aerial vehicle s position, velocity and attitude information at a rate of no-less than 1 Hz. The ground station must also be able to reliably send data up to the miniature aerial vehicle to effect a manual override of the automatic pilot in case of a malfunction. These requirements drive the need for a data-link modem that is robust to data dropouts and has high data bandwidth. In this project the data-link modem used was a MaxStream RF modem. These were 900 MHz RF modems. In our testing we found that these modems were able to transmit data reliably at a baud rate of up to 20 Kbps and distances upto 1 mile. 14

5.2.3 Open-Source GNC Software: MicroGear As noted earlier, the GNC system must possess a high level of reliability since a failure of the system leads to an uncontrolled aerial vehicle. Thus, the GNC used in the Unicorn was equipped with a fail-safe system. This system allowed the ground operators to isolate the control system in case it fails or an anomalous condition is detected. Once the control system is isolated, the ground operators can manually fly the aerial vehicle to safety. The fail-safe system is described in more detail below. This, of course, is a solution suitable only for a prototype system. In a system that will eventually be fielded in a realistic operational setting, we cannot assume that the operators on the ground are trained to pilot aerial vehicles. Thus, to ensure that the public is not exposed to an unacceptable level of risk, either one of the following will have to be done: 1. A detailed analysis to show that the risk of failure in the control system due to hardware or software malfunctions is below some pre-defined threshold. This threshold is defined by the agency that will be responsible for certifying such aerial vehicles. 2. Design of back up systems to ensure that no single hardware or software malfunction will lead to failure will lead to situations with an unacceptable large level of risk. It should be noted that the failures that need to be considered are not only those associated with the control system but also all other systems on which it relies such as the navigation and guidance systems. Designing potential hardware backup systems as well as analysis techniques for validating the GNC system risk is the topic of an on-going ITS institute sponsored research project. This project will be discussed in more detail later in the report when discussed proposed directions for future work. In order to be able to perform detailed risk analysis, we developed an open GNC solution. This solution used software (which is now an open-source software) with an off-the-shelf flight control unit hardware. The system developed is called MicroGear and combines the flight simulation and visualization features of the open-source flight simulator FlightGear with the open sources GNC algorithm developed as part of this work. The MicroGear software architecture is shown in Figure 5.7 and described below. Guidance System For autonomous operations, the user must define a geographic area or boundaries within which the aerial vehicle must fly. In the system developed for this project, this consisted of defining a series of waypoints. Waypoints are locations in space which, when connected in the appropriate sequence, define a path that a vehicle has to fly. The guidance system provides the interface by which the operator defines the waypoints. This is the input to the guidance algorithms as shown in Figure 5.7. The algorithms generate steering instructions that are used by the control system to guide the aerial vehicle along the path defined by the waypoints. Navigation System The navigation system determines the vehicle s position, velocity and orientation. In the various GNC architectures we examined and flew, the sensor used for position and velocity determination was GPS. The reason for using GPS is obvious: GPS is a ubiquitous utility in modern navigation which, depending on the type of implementation used, is capable of generating position estimates 15

Control Algorithms Vehicle Sensors Guidance Algorithms Navigation Algorithms Data Modem Operator Interface Avionics Suite (Navigation, Guidance & Flight Control) Data Link Ground Operating Station Figure 5.7: Block Diagram of MicroGear Navigation, Guidance and Control System. accurate to the meter- or centimeter-level. Other navigation systems (e.g., LORAN [20] or navigation grade Inertial Navigation Systems (INS) [21] without external aiding) cannot match this performance provided by GPS. In addition, GPS receivers are compact and have low power consumption. Thus, they are ideal for use in small aerial vehicles. GPS receivers capable of processing differential corrections and generating a navigation solution with accuracy on the order of a few meters are inexpensive and readily available. Orientation (or attitude) determination is considered part of the navigation system. In the systems developed for this project, attitude determination was accomplished by combining (or fusing) the information from an Inertial Measurement Unit (IMU) with the information from one or all of the following sensors: GPS, magnetometers (solid state compass) and air data (air speed) sensor. The IMU consists of a triad of accelerometers and rate gyros to measure vehicle acceleration and rotation rates, respectively. The aiding from the other sensors such as GPS or magnetometers is required because the attitude solution derived from an IMU will have drift errors. These drift errors can be unacceptably large (greater than 1 degree) unless highly accurate (and, thus, expensive) IMUs are used. However, IMUs that are capable of generating an attitude solution with accuracy on the order of half a degree are either too expensive (in excess of $60 K) or incompatible with miniature aerial vehicle operation. For example, the IMU used in the system described in [7] is one of the most accurate, high-end attitude determination system available today used in traffic management research similar to the application discussed here [21]. It weighs approximately 24 lbs, consumes 37.5 watts, costs approximately $100 K. Clearly, it is not suitable for use in inexpensive miniature aerial vehicle systems. This unavailability of compact, accurate and inexpensive attitude determination systems is one of the reasons why a single turn-key solution for GNC of miniature aerial vehicles is not available. The schemes used for integrating the IMU information with that of GPS or the other sensors makes assumptions on the dynamics (or the nature of the motion) of the vehicle. Thus, the performance reported in specification sheets for these attitude determination systems may not be achievable in all 16

operational scenarios. Control System The control system is what sometimes is referred to as the automatic pilot. It autonomously operates the aerial vehicle by moving the vehicle s control surfaces and throttle. To do this, it continuously compares the information from the navigation system (i.e., information about the current location, speed and attitude of the aerial vehicle) against the guidance system output (i.e., information about where the aerial vehicle should be based on the pre-defined flight trajectory and waypoints). If there is a mismatch between the navigation solution and the desired guidance path, the control system operates the control surfaces and the throttle of the aerial vehicle to make sure that it remains on or close to the pre-defined trajectory. Figure 5.8: Crossbow MicroNav GNC Hardware. The MicroGear Open-Source Software Used the MicroNav Sensors. The hardware behind MicroGear is the Crossbow MicroNav. The Crossbow MicroNav provides a system which contains an inertial sensor suite, an air data system and a GPS receiver integrated into a package with a small form factor. Figure 5.8 shows the MicroNav which is compact unit containing all the hardware required for navigation, guidance and control. The user supplies the algorithms. This latter feature an open source navigation, guidance and control algorithm suite enabled us to modify our algorithms to suite our specific needs. That is, the user is able to modify the GNC software and tailor it to satisfy specific needs. Since the operational characteristics of small aerial vehicles can be very different from one model to the next, GNC software has to be tuned for each specific vehicle. Similarly, different missions require differently tuned GNC algorithms. For these reasons, the MicroNav represents a very attractive platform for miniature UAS systems. The MicroNav system is currently allowing us to perform detailed analysis on risk associated with potential software malfunctions as part of another ongoing ITS Institute research project. Performance of the MicroGear is discussed later in the report. 17

5.3 Task 3: Ground Station Development The ground station is shown schematically in Figure 5.9. The major efforts under this task involved constructing the hardware required and developing the software that runs on the ground station. Each of these components (hardware and software) will be discussed separately in the paragraphs below. 900 Mhz 900 Mhz 2.4 GHz 72 Mhz Data Modem 1 Data Modem 2 Video modem Receiver Laptop 1 UAV Mission Control (Horizon Software) Laptop 2 Flight Trajectory Display (FlightGear Software) LCD Monitor display RC Transmitter Figure 5.9: Ground Station System Schematic. Ground Station Hardware The video and associated geo-referencing data transmitted from the aerial vehicle is received, processed and displayed by the ground station and its subsystems. At the ground station, data-link receivers with omni-directional antennas to receive the data from the aerial vehicle. The navigation, attitude and video data is processed by one of the ground station computer. The processing consists of assigning a GPS time tag and formatting the video and geo-referencing data for archiving. The processed video image is sent to the second ground station computer to be processed and then displayed to aid in real-time decision making. The video and geo-referencing data are both archived on the ground station computers. The archiving allows post-mission playback of the data. A photograph of the ground station that was depicted schematically in Figure 5.9 is shown in Figure 5.10. Ground Station Software The ground station software had three major functions. First, it archived video and GNC system data (aerial vehicle position, speed and attitude histories). Second, it provided a way to interact with the vehicle in real-time. That is, it allowed the operators on the ground to send commands such as waypoint changes to the aerial vehicle. Third, it presented data in real-time which enhanced situational awareness of the operators on the ground. The software used to interact with the aerial vehicle in real time was an off-the-shelf product supplied with the MicroPilot automatic pilot. Archiving of video data was similarly accomplished using an off-the-shelf software solution. The software used for the third function (enhanced situational awareness) was developed specifically for this project. The software is part of the MicroGear software suite. MicroGear not only allows visualizing data for enhanced situational awareness (as will be discussed below) but also provides a means by which to design, tune and validate GNC algorithms written for the MicroNav GNC system (as discussed 18

Figure 5.10: Ground Station Shown in Set-Up in a Laboratory. earlier). We will not discuss the latter functionality of MicroGear in this report as it is beyond the scope of this work. However, the use of the MicroGear software to enhance situational awareness is central to the project and will be discussed in some detail next. MicroGear Data Visualization Software MicroGear is adaptation of the open source flight simulator known as FlightGear. In a nutshell, the MicroGear software generates a synthetic display on one of the ground station computers. The display depicts the view that would be seen by a hypothetical observer on the miniature aerial vehicle. The view displayed is similar to but not identical to the view generated by the onboard video camera. The camera provides a view of the actual scene below the aerial vehicle at the time of flight. The view generated by MicroGear is based satellite imagery from a Geographic Information System (GIS) database. The GIS database information allows depicting roads, buildings, vegetation and other permanent or semi-permanent features. Given an accurate estimate of the vehicle s (and, thus, camera) position coordinates, attitude and a GIS database, a virtual reality (or synthetic) image of what the camera should be seeing can be constructed. Figures 5.11 and 5.12 show a comparison of the image from a video camera and a synthetic version of it generated by MicroGear. The photo is of the location where flight tests were conducted. It is known as Jensen Field and is located in Rosemount, Minnesota. Figure 5.11 is a snap-shot of a view captured by the onboard video camera. Figure 5.12 shows the synthetic image generated by MicroGear in flight over Jensen Field. Because of attitude and position errors as well as field of view differences between the camera and 19

Figure 5.11: Screen Capture of a Video from a Flight of the Unicorn over Jensen Field, Minnesota. the virtual image, the camera captured and the synthetically generated scene may not match exactly. In this case, matching the two images, however, is just a matter of stretching and shifting the synthetically generated image to match the image captured by the camera. Another source of a mismatch between the two images is due to the fact that the satellite imagery being displayed in MicroGear will be taken at a time prior to the flight. If the time when the GIS information was taken is closer to the time of flight, this mismatch between the MicroGear display will be less. Figure 5.16 shows a snap-shot comparison between a video image and MicroGear s depiction. Mismatches due to stretching/shifting have been corrected. However, a mismatch between the two (e.g., vegetation concentration) is due to the fact that the GIS database used pre-dates the date of the flight test. The fact that the display provided by MicroGear is based on satellite imagery and not real-time video data is the key to enhanced situational awareness because: 1. In scenarios where images from EO cameras is degraded by environmental factors such as fog, low-lighting (night time operations), smoke, haze, etc, the live video broadcast to the ground station from the aerial vehicle may not provide any useful information. However, since the MicroGear display is based on imagery not affected by these environmental factors, it will provide continued situational awareness such that the aerial vehicle is operated in a safe manner away from structures, buildings, etc. 2. Non-physical features can be incorporated easily into the display. For example, Figure 5.14 20