How To Compress Video For Real Time Transmission

Transcription

1 University of Edinburgh College of Science and Engineering School of Informatics Informatics Research Proposal supervised by Dr. Sethu Vijayakumar Optimized bandwidth usage for real-time remote surveillance system Evangelos Valtos Edinburgh, 18 March 2005

2 1. Introduction An interesting method of video compression for real time video streams in order to reduce the size of transmission is priority encoding of selected regions for each visual frame. Such an approach that encodes with priority certain interesting regions takes advantage of the neurobiological anatomy of the human eye. Photoreceptors on the retina have a highly nonuniform distribution which has an immediate effect on the area of a visual scene that is captured at high resolution. Only a region of 2 5 of visual angle around the point of gaze is highly processed, while the resolution decays logarithmically with eccentricity. Consequently it would be unnecessary to encode redundant information of a visual scene by using a uniform quality compression algorithm. A human observer would be able to efficiently process and perceive only a small part of an attended scene depending on his current point of fixation. The purpose of this project is to develop a real time surveillance system, where a camera could be guided remotely, by the point of gaze of a human observer, while a smart compression algorithm will be applied to the attended video stream to address the available transmission bandwidth. The algorithm that will be used will apply priority encoding of the attended regions of a visual scene, based on the captured point of gaze, to achieve a tradeoff between the overall compressed size of the video stream and the degradation of the video quality. A sort description of the proposed system architecture follows. A human observer will attend a location on a monitor fed by a remote camera. By capturing the human s point of gaze using an eye tracking device, the remote camera could be manipulated in order to turn towards the attended location of the scene. In addition to that a clever compression algorithm will be used on the transmitted video stream so that most of the communication bandwidth will be allocated to high fidelity transmission of a small region around the viewer s current point of regard, while peripheral image regions will be highly degraded and transmitted over little remaining bandwidth. The rest of the report is organized as follows. In section 2 the existing background is covered, where some of the related models are briefly described. Section 3 refers to the architecture and methodologies that will be used for the implementation of the proposed system. The evaluation process of the system is addressed in section 4, while the desired outcome of the project is discussed in section 5. Finally in section 6 a work plan, including the required tasks for the project completion, is made. 1

3 2. Background Diepen et al proposed a new technique [DGR94] that allows graphical masks or windows of arbitrary form, size and content to be moved quickly over a complex graphical stimulus. This moving overlay technique makes use of the ATVista Graphics Adapter, a board with the ability to mix an internally stored and an externally generated image into one composite image. By programming the internal image to be moveable and partly transparent, a highperformance moving mask or window is created. The technique is implemented on a standard PC interfaced with an eyetracker, thus bringing mask (window) movement under on line eye movement control. Kortum and Geisler have developed a preliminary version of a foveated imaging system [KG96], implemented on a general purpose computer, which greatly reduces the transmission bandwidth of images. Eye movements are recorded so that the high resolution region of the image can be kept aligned with the high resolution region of the human visual system. Their system has demonstrated that significant reductions in bandwidth can be achieved while still maintaining access to high detail at any point in an image. The work of Lee, Bovik and Kim [LBK99] introduces a prototype for foveated visual communications as one of future human interactive multimedia applications, and demonstrate the benefit of the foveation over fading statistics in the downtown area of Austin, Texas. In order to compare the performance with regular video, they use spatial/temporal resolution and source transmission delay as the evaluation criteria. A formal model [RS99] which describes consistently levels of detail and regions of interest for the progressive transmission of raster images is introduced by Rauschenbach and Schumann. This model serves as the foundation of a wavelet based image communication framework which supports the progressive, redundancy free transmission of multiple, possibly overlapping regions of interest in an image. To support demand driven transmission, a method for interleaving control commands with the embedded bit stream is introduced. Reingold and Loschky [RL02] report three experiments that document a slowing of peripheral target acquisition associated with the presence of a gaze contingent window. This window effect was shown for displays using either moving video or still images. The window effect was similar across a resolution defined window condition and a luminance defined window condition, suggesting that peripheral image degradation is not a prerequisite of this effect. The window effect was also unaffected by the type of window boundary used (sharp or blended). These results are interpreted in terms of an attentional bias resulting in a reduced saliency of peripheral targets due to increased competition from items within the window. 2

4 The theoretical benefits, implementational issues, and behavioral consequences of variable resolution displays are reviewed by Pakhurst and Niebur [PN02]. A mathematical analysis of computational efficiency for a two region variable resolution display is conducted. The results are discussed in relation to applications that are limited by computational resources, such as virtual reality, and applications that are limited by bandwidth, such as internet image transmission. 3. System architecture and methodology A steerable camera will be used to record in real time the scene of a distant area under surveillance. The camera will be connected to a pc responsible for the transmission of the recorded data and for the manipulation of the camera movement. The footage captured by the camera will be encoded using a compression algorithm dependent on the current point of gaze of a human observer. The compressed video stream will be transmitted to a distant pc situated in the same physical location as that of the human observer. A person will be able the view the area under surveillance on a monitor of a pc receiving the video stream from the remote camera. The observer will be free to look at any point of the monitor based on his personal intuition of what is interesting in the attended scene. An eye tracking device will be used to capture the point of the human observer and feed the coordinates to the serial port of the pc. These coordinates will determine the center of the attended region, which will define the area of the transmitted frames that will receive higher encoding. The extracted coordinates will then be sent back to the remote pc as input to the compression algorithm, in order to encode the compressed video stream according to the attended location. Thus, a better usage of the available bandwidth will be achieved by transmitting in high quality only the attended and hence the important part of a scene. In addition to the compression algorithm, the recorded coordinates form the eye tracking device will be used to turn the camera towards a specific direction so that the human observer can have a better view of the desired region of the remote location. So for example when the recorded coordinates indicate that the attended region is near in one of the four edges of the scene, the camera will turn so that more details around that point can be viewed. Consequently the resulting video stream will always be dependent on the real time point of attention of a human observer, in terms of the region of the area the camera will be turned at, and the allocation of the compression quality for each region of the transmitted video frames. Since the proposed system should be a real time application special consideration for the accuracy of the eye tracking systems should be taken, so that the outcome of the compression algorithm and the movement of the camera will reflect the current point of gaze as good as possible. 3

5 The programming language for the project will be C and the machines that will be used will run Linux operation system. 4. System evaluation process The success of the proposed system will be judged upon its ability to accomplish the desired course of its design, which is to provide a hardware and software model to allow the remote surveillance of a distant location. In order to evaluate the performance of the system existing constant quality compression algorithms will be used to be compared against the implemented spatial prioritization algorithm. The system will be tested by human observers and the final assessment will be made by evaluating the encoded video stream based on the quality degradation the observer can detect when the implemented algorithm is applied versus the ones used for the test. 5. Desirable outcome This project will attempt to apply existing approaches in interactive gazecontingent foveation for video transmission to develop a real time surveillance system. We hope that the proposed model will provide a useful scheme for real time video transmission over limited bandwidth lines, e.g. an internet connection. The model will provide a prototype which will include a sample hardware setup for the proposed surveillance system and the required software to facilitate the data exchange between the two sites. The goal is to develop a biologically motivated algorithm which apart from its apparent scientific value would result in a useful application with possible commercial use. The resulting algorithm could eventually become a successor of the current compression algorithms and it would be more efficient and suitable for use in video transmissions in the future where the need for real time and interactive applications is growing. Besides from the proposed application, possible future uses could include entertainment and medical applications, or even the facilitation of the communications between the Earth and distant space crafts in space missions, a domain ruled by the limitation in the available transmission bandwidth. As far as the project s success is concerned, the hot issue that needs to be addressed is the real time nature of the system. In case that this feature could not be accomplished or not accomplished to the point needed, we could, as a back up plan, work with the transmission of static images rather than moving video or just lower the requirement of the response time of the system. 6. Work plan The development phase of the project will be divided in smaller subtasks, which are described below, while a rough estimation of the needed time is made. 4

6 The first milestone in the system development will be the implementation of a simple user interface application using the eye tracking device. At first the simple task of moving the mouse pointer will be attempted, without paying much attention to precision. Such a task will be helpful to get familiarized with the data acquisition and manipulation from the eye tracking device. (1 2 weeks approximately) Next we will focus on the precision of the eye tracking process, since that would be a crucial part for the success of the whole project. In order to reflect the real time character of the system we should be able to capture the eye movement with the best precision possible. For that reason the algorithm should process the acquired coordination points and assure that they correspond to the attended location. (1 2 weeks approximately) After having accomplished the first simple task of manipulating the data fed by the eye tracking device, the task of moving the surveillance camera will be addressed. The user will be able to control the orientation of the camera with simple commands (move up, down, left, right) which will be initiated by the position of the point of attention of the user. For example when the viewed regions of a certain video frame are near the edge of this frame a movement command will be initiated to turn the camera in order to include the regions surrounding that point. (2 weeks approximately) The final key point on the development of the system is the implementation of the spatial prioritization encoding algorithm. Based on the coordinates of the attended location the algorithm will apply a compression scheme that will encode the area around the attended location with high quality while the rest of the video frame will be encoded in decreasing quality as the distance increases from that point. (3 weeks approximately) The compression algorithm and the camera movement will be combined to form the final system (2 weeks approximately). Finally the evaluation and testing phase of the system will occur (2 weeks approximately). The writing of the final report is estimated to last approximately 8 weeks and it will begin at some early point before the completion of the project. In Figure 1 the timeline of the project with the various subtasks is presented (the starting and ending dates are subject to changes during or before the beginning of the project). May June July August Start Precision Movement Compression Report Combine Test 5

7 7. References Figure 1: The project timeline. [DGR94] van Diepen, P. M. J., De Graef, P., & Van Rensbergen, J. (1994). On-line control of moving masks and windows on a complex background using the ATVista videographics adapter. Behavior Research Methods, Instruments, & Computers, 26, [KG96] Kortum P. T. and Geisler W. S Implementation of a foveated image-coding system for bandwidth reduction of video images. SPIE Proceedings: Human Vision and Electronic Imaging, 2657, [LBK99] Sanghoon Lee, Alan C. Bovik, Young Yong Kim: Low Delay Foveated Visual Communications over Wireless Channels. ICIP (3) 1999: [RS99] Uwe Rauschenbach and Heidrun Schumann, "Demand-driven image transmission with levels of detail and regions of interest", Computer & Graphics 23, pp , 1999 [RL02] Reingold E.M.; Loschky L.C., Saliency of peripheral targets in gazecontingent multiresolutional displays, Behavior Research Methods, Instruments, & Computers, 1 November 2002, vol. 34, no. 4, pp (9) [PN02] Parkhurst, D. and Niebur, E. (2002). Variable-Resolution Displays: A theoretical, practical and behavioral evaluation.human Factors, 44(4),