Real-Time Video Relay for UAV Traffic Surveillance Systems Through Available Communication Networks

Real-Time Video Relay for UAV Traffic Surveillance Systems Through Available Communication Networks Yu Ming Chen, Liang Dong Jun-Seok Oh Department of Electrical and Department of Civil and Computer Engineering Construction Engineering Western Michigan University Western Michigan University Kalamazoo, MI 49008 Kalamazoo, MI 49008 Abstract The unmanned aerial vehicle can be an efficient and economical solution to real-time surveillance of highway traffic. This paper describes a data link that connects the camera onboard an unmanned aerial vehicle to the monitoring terminals in the office of the Michigan Department of Transportation. A video signal captured by the surveillance camera can be either displayed on the terminals in real time, or stored on ground for future off-line analysis. The video signal is relayed via available mobile communication networks. An addition server is used in practice to guarantee the consistency of data flow and high throughput of the communication channel. U I. INTRODUCTION nmanned aerial vehicles (UAVs) have experienced an unprecedented growth over the past decade with applications in military and civilian. The researches on UAV have been active in many areas, for example, UAV navigation [1], UAV controller design [2], multi-uav position estimation [3], and cooperative air and ground surveillance [4]. The U.S. Department of Transportation started to realize the advantage of the UAV approach to traffic surveillance. Traffic surveillance cameras used to be mounted on microwave towers along the highways. However, the installation is costly and time-consuming. The UAV can be used as a high-altitude moving camera to cover a wide area. Aerial surveillance video is captured by cameras mounted on the UAV platform. The video signal is encoded and transmitted to the traffic office with low latency in order for the operator to monitor the roadway usage, respond quickly to traffic incidents, guide emergency vehicles, etc. Several research groups have been investigating the airborne traffic surveillance systems that use UAVs [5, 6]. The group at the Ohio State University provided results of a series of field experiments and argued for the benefits and barriers to deployment of such UAV traffic surveillance system [5]. There are implementation obstacles such as restrictive FAA regulations and real-time video data communication. The group at the University of Florida has implemented an airborne traffic surveillance system that encodes the video data of an aerial view, and transmits the multimedia video streams over a microwave IP network [6]. The ground communication was implemented by mounting receivers at two microwave towers at the two ends of the highway that is being monitored. Two computers at the two towers serve as video encoders. Another computer at the State Emergency Operation Center receives the two video signals and displays the stronger one. The data communication is conducted through an exclusive microwave network of the Florida Department of Transportation. In our implementation, radio transmission of the video data from the UAV to a ground control station is ready with the UAV package. The surveillance video collected at the ground control station has to be relayed to the office at the Michigan Department of Transportation in a timely manner. As a special feature of this project, the ground control station is usually located by the highway, and can get easy access to a roadside communication tower. We tested the use of the available mobile broadband network to relay the surveillance video. The wireless link quality has been experimented and security has been taken into account when using the public wireless network for UAV video relay. II. DATA COMMUNICATION SCHEMES The data link is designed based on a Mini-UAV package made by the MLB Company. The UAV package includes a radio frequency transmission system that connects the UAV with the ground control station. The video captured by the onboard camera is transmitted to a ground station laptop and further relayed to the end user computers. Our task of relaying video focuses on how to deliver the video signals from the UAV ground station in the field to the end user computers in the control office. Most of the end user computers have access to the Internet. In order to simplify the requirement of the end user computers, the public Internet is chosen as the communication backbone for video delivery. The UAV ground station, i.e. the control laptop, is positioned in the field such that the UAV has an effective flying radius that covers a segment of the Interstate Highway 94. The optimal location of the ground station may not have any wired Internet plug-in. Nevertheless, the ground station is most likely located close to the highway, where the base station towers for cellular telecommunications are available along the highway (Fig. 1). 1525-3511/07/$25.00 2007 IEEE

Figure 1. Communication Scheme 1: UAV ground control station sends video directly to end user computers via the mobile broadband network. Therefore, the nationwide Sprint mobile broadband network is exploited for the UAV ground station to transfer the video onto the Internet. As depicted in the figure, when the UAV captures the video signals of the traffic, it transmits the data towards the ground station. The ground station receives the radio signal and converts it into baseband digital video. The ground station laptop uses a wireless PC card to connect to the Sprint Network. The Sprint mobile broadband network is connected with the backbone Internet. Finally, the end user computers obtain the videos from the Internet. The Sprint mobile broadband network provides mobility; however, the wireless connection becomes the bottle neck of the data pipe. Therefore, when this video relay scheme is applied, it has limited bandwidth for data communication. A. Two Data Relay Schemes In order for the end user computers to retrieve the digital video captured by the UAV camera, in this project, we implemented two schemes to relay the digital video to the end users via the Internet. In the first scheme (Fig. 1), the UAV ground control laptop informs the end user of its IP address, and the end user computers can open a media player to retrieve the video stream from this IP address. We ought to mention that the Sprint mobile broadband service assigns a dynamic IP address upon every new request for connection. This means that the IP address is changed every time the ground control laptop connects with Sprint Network. Therefore, the ground crew needs to inform the end users of its recently acquired IP address of every connection. In the second scheme (Fig. 2), an additional server is used and served as a host of the data communication link. The server is operated with a fixed IP address which can be known a prior to the ground control laptop and the end user computers. The Figure 2. Communication Scheme 2: A server is used as the central point of the video relay. UAV ground control laptop streams the video to the server with this particular IP address, and the end user retrieves the video from the server via this particular IP address. In addition to real-time video streaming, the server can store the received video onto its hard drive for future off-line analysis. The server also hosts the project website. This website has a link to a page that displays the real-time video captured by the UAV onboard camera. In order to host the website and at the same time be ready for the incoming signals from the UAV ground control laptop, multiple network ports are opened at the server. The server may add additional processing delay and has certain concerns with the firewall. These issues will be discussed in detail in the following subsections. Compare Fig. 1 with Fig. 2, in the first scheme, multiple end users share the data pipe that has the data rate of the mobile broadband connection. Since the wireless link is the bottle neck for video signal transmission, the scheme can only support a limited number of end users with acceptable data rate. The sum of the maximum data rates that the end users can achieve is smaller than the data rate with which the UAV ground control laptop transmits to the server via mobile broadband connection. In the second scheme, UAV ground control laptop sends data to the server via the mobile broadband network. The server relays the data to the end user computers via the wired Internet connection. The maximum number of end users can watch the video simultaneously depends on the data rate of the wired Internet connection of the server, which is much higher than that of the wireless connection. Therefore, more end user computers can receive the video stream simultaneously with acceptable data rate. Here, the maximum data rate that each end user can have is smaller than the data rate with which the UAV ground control laptop transmits to the server via mobile broadband connection. Fig. 3 shows the block diagrams of

contents to a Server system which is located at Parkview Campus at Western Michigan University. The firewalls and communication ports are controlled and authorized by Networking Administration from the University. In order to receive any media content from outside the network, certain communication ports from WMU have to be opened. Table 1 shows the communication ports which have been issued and authorized from WMU for the Server System of the UAV project. Protocols Comm. Ports Purposes HTTP 80 Windows Media Service HTTPS 443 Hosting Website Figure 3. Block diagrams of UAV data communication links. UAV communication links of both Scheme 1 and Scheme 2. B. Stream Media Protocol A protocol is a set of rules and procedures on how data is communicated between devices or computer programs. Protocols enclose important information to ensure that data is delivered properly. In this UAV project, many protocols have used for video streaming in real time. Those protocols have been part of standards and well used in networks and the Internet. One protocol named Hypertext Transfer Protocol (HTTP) has mainly used at the distribution methods to transfer a video stream from the field. It is an application layer protocol that relies greatly on the transport capability of TCP for traffic control. Based on this protocol, the video stream can be delivered either to a server system or directly to a media player program behind a firewall in a private network system. C. Server Firewalls and Communication Ports A firewall is basically a piece of hardware or software that can be controlled by network administrators to decide which data packets can either enter or leave a protected network. In order to control the flow of data traffic, there are numbers of ports can be controlled for certain types of data packets in firewall. The firewall controls the data flow of traffic based on the destination port, the source IP address, and the target IP address. The information is related to its data packets. Some firewalls can also control the data flow further based on the protocols. For this UAV project, the allocating ports for Windows Media Streams are different and depend on ways of content distribution methods. In the PULL distribution method in Windows Media Encoder, the allocating port of firewall usually uses and sets at port 80 but is also changeable to other free allocating ports. Meanwhile, in the PUSH distribution method, the Windows Media Encode program sends media Table 1. Server communication ports. D. Wireless Data Link Sprint and Novatel Wireless have developed a new technology together for a high speed and nationwide network system. This technology is operated based on CDMA 1x Evolution Data Optimized (EvDO) and 1xRTT wireless network systems. It allows for maximum forward link capacity ranging from 500 Kbps up to 2.4 Mbps, and a maximum reverse link capacity of 153 Kbps [7]. EvDO networks are optimized for data transfer on the forward link through the use of packet based time division multiplexing and by scheduling packets based on the user channel quality, rather than utilizing traditional round-robin scheduling. In contrast, CDMA is used on the reverse link, but with adaptive rate control, which allows an increase in reverse link throughput that can be higher than 1xRTT networks. A revision for EvDO standard is designed to reduce network latency and raise network bandwidth further than the current EvDO maximum, with theoretical maximum forward link and reverse link capacities of 3.1 Mbps and 1.8 Mbps, respectively. Based on these wireless networks, a real-time video stream captured by the UAV is able to be broadcasted from a ground station in field to a place that has Internet access. Sprint has provided a way for users to be able to browse on the Internet using a Sprint Mobile Broadband Sprint PCS Connection Card (Merlin S620). This broadband PC Card, made by Novatel Wireless, is used at the ground control laptop in order for it to connect to one of cellular towers of Sprint Networks by the highway. According to Sprint, the PCS Connection Card can operate at average download speed ranging from 400 to 700 Kbps and up to 2 Mbps with peak rates in Mobile Broadband areas, and at average speeds of 40-70 Kbps and with peak rates up to 144 Kbps at its Nationwide PCS Network cover areas. Figure 4 shows a coverage map of Sprint mobile broadband service around Kalamazoo, Michigan. III. MEDIA PROCESSING PROGRAM The most important part of communication is to find out how a real-time video stream can be delivered from the ground station in the filed back to a control center at the Michigan Department of Transportation miles away. Due to the source

Figure 5. Media encoder test run with videos capture by the UAV camera. Figure 4. Coverage of Sprint Mobile Broadband services in West Michigan. material of UAV project, i.e. it is a digital video (DV) tape from Sony deck system, the encoder program may need to convert it from the DV format to another digital format (such as Windows Media Video). After the video stream is converted, it may be distributed over the Sprint network and further onto the Internet. The conversion process is performed by connecting a DV tape from Sony i.link ports to an IEEE 1394 port installed in a computer. According to the specification of the project, the real-time video stream is recorded as an analog format on a DV 8 tape in the Sony video deck system. Any media content has transferred as a digital format once exporting out through the USB port from the deck system. Therefore, a suitable image coding program called an encoder is needed for furthering processing. The main task of encoder is to compress the media content so that it can be delivered on the Internet. Compression is necessary because video in its original state is too big to deliver within available network bandwidth. The Windows Encoder is one of encoding programs that can convert and compress media content into a traversable network media format. The Encoder Series has applied in UAV project with the following functions: 1) Capturing a real-time video stream in simultaneously when Sony DV video deck device is recording the video stream from UAV plane. 2) Compressing the captured video stream into Windows Media Format in a pre-set coding bit rate. 3) Encoding and delivering the real-time video stream to a server system operating with Windows Media Services for broadcasting as a live streaming event on the Internet. 4) Using other routing to encoding and deliver the real-time video stream directly to end-users without using a server system for broadcasting on the Internet. 5) Customizing in a proper coding bit rate to compromise for a highly variation of transmission rate of Sprint Merlin S620 nationwide PC Card. 6) Converting files from one format or bit rate into Windows Media Format. The programmed media encoder was tested using a number of aerial surveillance videos. Fig. 5 is a screen shot of the video encoding process. On the left hand side, the input was a video from the Sony Video deck system. The Sony video deck system receives a video stream from the UAV and converts the video from analog to digital format. On the right hand side, the output was the result after encoding, with a size, resolution, and frame that had be pre-setup to a proper format and then the output was broadcasted on the Internet through the Sprint PC card. Windows media service is one of applications which have built into the Windows Server 2003 operating system. Based on functions provided from windows media service with a server system, the administrator can manage the media content to be published in real time to other clients on the Internet. The Real-time live broadcast videos will mainly be used during every UAV flying mission. The Windows Media Service has applied in UAV project with the following functions: 1) Setup the bandwidth limits for the publishing during the content distribution. 2) Control the number of clients accessing audio and video files simultaneously. 3) Manage the overall bandwidth demands of the windows media services functions; 4) Minimize the excessive demands of media service distribution over the network by properly configuration and optimizing media services functions. 5) Decide a Real-time live broadcasting to certain clients with authorized IP address; 6) Provide functions to multiple media files to a single broadcast or in a single directory for selective broadcasting. Windows media service provides two delivered methods: unicast and multicast distribution scheme. The unicast delivery provides a one-to-one video stream between the media server and each client system, whereas the multicast delivery option sends a single video stream that can be accessed by multiple users simultaneously [8]. IV. FIELD EXPERIMENTS At the beginning of the field experiments, a video stream was directly delivered from an encoder to an end user running a media player program. The video image was choppy and not received as continuous and stable. The image of the video stream was either frozen or buffered when the end users were watching the video. Some causes have found for the unreliable reception and used to solve for the video instability problem. First of all, the bandwidth promised by Sprint cellular

network is not as stable as a high speed cable network. The maximum upload speed is claimed to be between 144K and 152K bps. However, according to experiments, the actual transmission speed in the field was found to be in a lower range between 80K to 135K bps around the Kalamazoo area. Therefore, it is necessary to downgrade the quality of video images and provide buffering in a server as an intermedium during the distributions. Due to drawbacks of wireless communication in field tests, it is necessary to find out what resolution bit rate can be to provide an optimal image quality in real time distributions. An experiment was conducted and the results are shown in Table 2. It is a comparison between two transmission schemes in different image resolutions in real time distributions. According to the results, the transmission with no server system scheme could only provide a video stream in continuous and smooth image in a bit rate of 45K bps and below. Even though the resolution could increase up to 58K bps without any image buffering, the video stream would still freeze up a few minutes after the file started being received by end users. Therefore, an optimal resolution for transmission without a server is setup around 45K bps. A rate of resolution below 45K bps may not meet the requirement of the UAV project. An improvement of communication that includes a server system is needed. The server is operated as a host in order to increase the rate of resolution as well as speed of distribution. According to the experiment results, the transmission with a server system scheme could provide a video stream with a continuous and smooth image at a bit rate of 109K bps. The resolution can be setup up a screen size of 160x120 and a rate of 15 frames per second. Based on these results, the server system can actually improve the drawback of bandwidth limit from Sprint network. Also, during the late field tests, a server system actually provided the same rate of resolution to a few end users watching live media content simultaneously. Therefore, an optimal resolution for transmission with a server system can be setup at about 100K bps. One feature of the encoder program is that the frame rate can be set by the customer at a specific value of 15 or 30 frames per second. A higher frame rate can provide a smoother image for the end user s playback. However, a high frame rate may exceed the capacity of the bandwidth, and 30 fps are more images than the network can handle. Therefore, a better frame rate is found to be 15 fps for the UAV project. The screen size is another issue of image quality. Usually, a bigger screen size may provide a clearer picture of video stream. However, the bandwidth of the cellular network has limited possibilities of high quality video streams. From the experiment in the field, the image size of 240x160 is a tolerable setup. V. SUMMARY We proposed a video relay scheme for the UAV traffic surveillance systems through existing public mobile broadband networks. Although security is a concern, this communication scheme is easy to implement given the common setup of the UAV package and the availability of the wireless network along the highway. Experimental results with real aerial surveillance videos show that the proposed scheme can deliver quality videos to the control office in real time. At a field demonstration to the Michigan Department of Transportation, using the scheme that includes a server, we successfully relayed a video signal with an image size of 320x280, resolution 112 K bps, and 15 frames per second. With Server Without Server Bit Rate (Kbps) Frame Rate (fps) Screen Size Field Experimental Results Buffering Picture Frozen Picture Smooth 1128 29.97 320x240 Yes Yes No picture 764 29.97 320x240 Yes Yes No picture 548 29.97 320x240 Yes Yes No picture 340 29.97 320x240 Yes Yes No picture 282 29.97 320x240 Yes Yes No picture 148 15 240x180 No Yes Yes 109 15 160x120 No No Yes 58 15 160x120 No No Yes 43 15 160x120 No No Yes 28 15 160x120 No No Yes 1128 29.97 320x240 Yes Yes No picture 764 29.97 320x240 Yes Yes No picture 548 29.97 320x240 Yes Yes No picture 340 29.97 320x240 Yes Yes No picture 282 29.97 320x240 Yes Yes No picture 148 15 240x180 Yes Yes No picture 109 15 160x120 Yes Yes No 58 15 160x120 No Yes No 43 15 160x120 No No Yes 28 15 160x120 No No Yes Table 2. Video relay setup and results of field experiments. REFERENCES [1] D. H. Shim, H. Chung, and S. S. Sastry, Conflict-free navigation in unknown urban environments: Autonomous exploration for unmanned aerial vehicles, IEEE Robotics & Automation. Mag., pp. 27 33, Sept, 2006. [2] J. Cho, J. C. Principe, D. Erdogmus, M. A. Motter, Modeling and inverse controller design for an unmanned aerial vehicle based on the self-organizing map, IEEE Trans. Neural Networks, vol. 17, no. 2, pp. 445 460, Mar, 2006. [3] L. Merino, et. al., Vision-based multi-uav position estimation: Localization based on Blob feathers for exploration missions, IEEE Robotics & Automation Mag., pp. 53 62, Sept 2006. [4] B. Grocholsky, J. Keller, V. Kumar, and G. Pappas, Cooperative air and ground surveillance: A scalable approach to the detection and localization of targets by a network of UAVs and UGVs, IEEE Robotics & Automation Mag., pp. 16 26, Sept 2006. [5] B. Coifman, M. McCord, M. Mishalani, and K. Redmill, Surface Transportation Surveillance from Unmanned Aerial Vehicles Proc. of the 83rd Annual Meeting of the Transportation Research Board, 2004. [6] S. Srinivasan, H. Latchman, J. Shea, T. Wong, and J. McNair, Airborne traffic surveillance systems Video surveillance of highway traffic, ACM Int. Workshop on Video Surveillance & Sensor Networks (VSSN), Oct 2004, pp. 131 135. [7] M. Claypool, R. Kinicki, W. Lee, M. Li, G. Ratner, Characterization by Measurement of a CDMA 1x EVDO Network, Worcester Polytechnic Institute, Aug 25, 2006 [8] R. Morimoto, M. Noel, A. Lewis, Microsoft Windows Server 2003_Unleashed, Sams, 2006, pp. 1210-1235.