Cloud-Empowered Multimedia Service: An Automatic Video Storytelling Tool

Transcription

1 Cloud-Empowered Multimedia Service: An Automatic Video Storytelling Tool Joseph C. Tsai Foundation of Computer Science Lab. The University of Aizu Fukushima-ken, Japan Abstract Video storytelling has become a popular technology to let users design and plan their films. As the development of multimedia devices and network, more and more people will share their productions on the Internet. This approach is not only used in movie industry, but e-learning, digital archive and art. With the development of cloud computing, it can provide various computing and storage services over the Internet. Instead of using 3-D model generation system, we proposed a system composed of several video technologies to let teachers plan materials with existing avatars and scene from the cloud data base in this paper. Users can design the background by cloud panorama generation at first. Then, drug a trajectory line on the scene. The authoring tool will select an avatar from the database and make it with several kinds of behaviors. Users can select several behaviors during the motion trajectory; our tool will search the most suitable avatar and insert it into the panorama. This tool equips with special functions to integrate different motion tracks for the generation of video narratives is also presented in this paper. Keywords- Motion interpolation; Video Planning; Panorama Generation; Layer Segmentation; Cloud Computing I. INTRODUCTION (HEADING 1) As the interactive media has become pervasive and popular in our life, more and more researchers focus on this research field. Follow to this situation, author-ing tools are widely applied in various applications. For instance, users can em-ploy video editing software to process video retouching for generating a new film. Therefore, there are many researcher combine video authoring tool with education to generate varied teaching materials. How to attract students eyes and make courses interesting is the most important goal of teachers. Although there are many learning systems proposed, the operation of most of tools is not easy to use. Easy-to-use is the most important to every authoring tool. According to statistics, there are more and more teachers will choose the tools to generate their courses. Indeed, to apply accessorial teaching materials, such as videos, photos or games, will help to improve the effect of teaching. Cloud computing has become the most popular research field in recent years. As its development, people can store every kind of data on the cloud through the Internet. The cloud server provides hardware devices and software services as the data-center. By this way, users just need to prepare a PC with the network that the cloud services can be used. Therefore, it s another new research field to combine multimedia services and cloud computing, called cloud media. To provide an ef-fective, flexible and scalable data processing is the main goal of the ` Neil Y. Yen School of Computer Science and Engineerinig The University of Aizu Fukushima-ken, Japan neilyyen@u-aizu.ac.jp cloud multi-media. It also solves today s users requirements in high quality multimedia and applications. In other words, as the development of network and ubiquitous devices, such as smart phone, people attach importance to the effect of the multimedia. Through providing multimedia services by cloud, users can enjoy the services anywhere without expensive devices. Therefore, we would like to propose a storytelling tool via cloud computing and storage for users generating a new film in this paper. Users just need to choose a panoramic figure or update a video for generating a background, then decide a trajectory on the frame. Several behaviors are provided to generate avatars with special motions, the system will search the best fit avatars from the cloud database and insert into the panoramic figure. Interactive storytelling [1] can use tracking method to extract an avatar from the real video and combine it with a virtual reality model. In this paper [1], authors proposed an artificial intelligence system to let user interact with the virtual avatar. However, to extract a precise object from a video is a critical step to make the generating video much real. In [2], Khan et al. proposed a novel object tracking algorithm for processing complex cases in different scenarios. The algorithm employs particle filters and multi-mode anisotropic mean shift. It improves previous methods to make the tracker can detect the target object more accurately. Even if the target is occluded with some other objects, the tracking results still maintains a tracking box on the target object precisely. The method uses an online learning method to make the result of tracking not influenced by noisy shape changes. To process the multimedia service through the internet needs to consider the performance and quality. In [8], Zhu et al. proposed mul-timedia-aware cloud which can show how a cloud performs distributed multimedia processing and storage. Quality of service is another key problem has to consider in the cloud technology. A media-edge cloud (MEC) architecture is employed to solve the performance problem in cloud. In this paper, the main contribution is the generation of a multilayer video planning system based on cloud computing. It includes a new object tracking and extracting mechanism, a panorama generation and a spatiotemporal placement for deciding the avatar s location in the scene. A video clip is a spatiotemporal continuous list of frames. Each frame is com-posed by some regions. These regions can be treated as video layers. If video lay-ers can be manipulated properly, special effects of video (or forgery video) can be produced. It is a complicated procedure involves a series of challenge problems to produce video narratives. We summarize Copyright c 2010 Future Technology Research Association International 13

2 Volume 4, Number 3, September 2013 the procedure in several steps here with detailed solutions discussed in sections 2 and 3. Panorama Generation (Video scene): We use a frame referencing technique to find out the most similar patches from other frames to replace the area of fore-ground objects. After all frames are restored, the motion estimation will be used to combine all frames into a panorama scene. For this panorama, users can control it on the system to design a narrative film. Behavior Clip Preprocessing: To decide a new video or storytelling, users have to plan the motion trajectory of object or avatar. We define several behaviors to let users select. For each behavior, the system will search on the cloud database for the best bit motion clips. A technique of object size ad-justment is adopted here for regulating size of objects at different layer/depth. Furthermore, in order to generate a video according to the video planning, an algorithm of motion interpolation/extrapolation of avatars [3] is used to control the motion speed of the avatar and maintain the video length of the avatar clips the same with the panorama scene video. Video Narrative Generation: The spatiotemporal placement of avatars (in-cluding Z distance) is calculated, with possible conflicts identified and resolved by the user. Video frames and layers are calibrated with the selected video scene in a video schematic. A layer merging mechanism is applied to generate the final video narrative. Cloud Data Management and Computation: Cloud database is suitable to store multimedia data. Since it is scalable and flexible, users can update an amount of metadata onto the database. Users can apply to not only their data but also others information. We use cloud computing into object extracting, panorama generation and video narrative generation. Figure 1. The system framework, users have to upload the videos or photos to the cloud server, and then use the system to design the video. Figure 2. The video planning example, there are two avatars selected and their trajectories are designed on the panorama frame. In Fig. 1, we illustrate the system framework. Users have to update their videos or avatars on the cloud database, all of the preprocessing such as avatar extraction, panorama generation and avatar motion generation, will be computed and processed through the cloud computing. A video planning framework is showed in Fig. 2. We let users design the avatars motion trajectories and the cloud computing system will compute the location of the avatar to adjust the size. In this figure, there are two avatars and orange and blue lines are their trajectories. The red frame with dotted line means the video is playing at that time. The system will refer to the trajectory and the length of the panorama to compute the correct avatar s location and its size. The first step processes the background video into a panorama. It will be discussed in section 2. We describe the remaining steps in section 3. Finally, the results and conclusion will be showed in section 4 and 5. The Fig. 3 shows the flowchart of our system. If users update a video with several 14 Copyright c 2010 Future Technology Research Association International

3 foregrounds, the object extracting method will be applied to generate avatars and update to the database in the step2. After this step, the video inpainting algorithm can be used to restore the damage area of the video. The inpainting results will be combined into a panorama. In step 4, an interface is provided to let users design the new film. It can load metadata from the cloud database and generate new film by cloud computing. We will explain all of the steps in detail in the following sections. Figure 3. The system flowchart. II. PANORAMA GENERATION The first challenge of this system is the background video generation. In order to let users operate the system conveniently, we propose a panorama generation algorithm to create a background figure for users designing the new video. To make sure the generating background video without any avatar, we use video inpainting [4] to remove all of the objects in the video. Before processing the inpainting algorithm, we have to mark or extract the existing foregrounds from the videos. A. Object Tracking and Extracting To extract the objects from video, we proposed an object tracking and extracting algorithm to segment the foreground from the background. The extracting foreground clips can be used as the new avatars, the processing of the avatar will be discussed in the next section. In our algorithm, the motion estimation, grab-cut and mean shift are utilized to track and segment the selected object from the video. Although there are many object tracking algorithms proposed, most of them are hard to map to cloud computing. Therefore, we would like to use several methods in this system At first, users have to select and mark the target object in the first frame of the video clips. To make sure the extracted avatar without any pixel of background, users need to mark the target object accurately. We adopt CDHS algorithm [5] to estimate the motion vector of each block on the contour of the target object. According to the motion vector of each block, we can compute a rough position of the target object in the next frame. A rough tracking window to the next frame be computed, the grab-cut [6] algorithm is employed to segment the foreground from the rough tracking result. At the beginning of this algorithm, we collect the information of the target object, includes color and structure. Therefore, the detail information can be checked and make the foreground label and background label much clear. In order to make a double check for the extracted result, the mean-shift algorithm [7] is proposed to process color segmentation or clustering, we use this method to recognize the foreground and background. The color information of masked object by users will be processed in mean-shift filter, we will get the 3D color matrix of the object. The color of result from grab-cut can be converted into CIELuv and compare with the result of the matrix. It will eliminate the pixels of background and a more accurate extracted result will be generated. B. Object Removal and Panorama Generation In this step, we combine video inpainting and motion estimation to finish the panorama generation. To restore the damage part of the video, the video inpainting algorithm [4] is employed to inpaint the video. It can maintain the time continuity of the video, therefore the removal part will be smoother after inpainting. The panorama generation is based on the results of video inpainting, we use the inpainting results as the reference frames to generate the panorama figure. The motion estimation algorithm [5] is used to compute the motion map between each frame. The information of motion map is a pattern to decide the additional spatial range from next frame with the current frame. Therefore, we can combine all of the additional range of each frame into a panorama mosaic. III. VIDEO PLANNING According to our system, users have to select the avatars and background scene from the cloud data base. The motion trajectory of the selected avatar on the panorama mosaic is also decided in this step. There are also two motion tracks need to be chose by users. In order to produce a realistic result, several techniques such as object size adjustment and motion clip Copyright c 2010 Future Technology Research Association International 15

4 Volume 4, Number 3, September 2013 processing includes motion interpolation/extrapolation should be considered. Object size adjustment is used to regulate the size of object at different video layer/depth. Motion interpolation and extrapolation is used to produce a suitable video length of each motion clip. A. Avatar Size Adjustment and Motion Interpolation/ Extrapolation Our cloud system can let users update every kind of avatar with different resolution. However, it will be difficult to compute the camera parameters which are the references to adjust the size of avatar or the depth of the background video. Therefore, we assume each object s height is 170 cm and we can modify all of the avatars into the same size. The size can be simulated as the quarter of the height of the panorama figure if the avatar is set in the middle of the video. According to this method, we can adjust all of the avatars from the data base into a suitable size and make the generating video realistic. For generating a meaningful video narrative, an algorithm of motion interpolation/extrapolation of avatars is used in this paper for producing a suitable video length of each motion clip. In an earlier work [3], authors propose an algorithm which can extract an object from video and analyze object motion in the whole video sequence. The work [3] is summarized as follows. Since the detail motion information of an object is obtained, the procedure of motion interpolation can be used to generate a new posture of object from the original motion clip. B. Motion Track and Video Planning The basic element to tell a story in a video narrative is called motion clip. A motion clip can be a target object tracked from a video. This video is not neces-sary obtained from the same camera as the background video. Essentially, a mo-tion clip relies on the user to choose a motion track in the story. The challenge is-sue is to provide a spatiotemporal placement of motion clips as well as to define how motion tracks can be merged. The algorithm for story generation is based on the panorama as a time line. In our current implementation, we have implemented two types of motion tracks: Regular Motion Track: We can let users to choose the speed of the avatar s movement. It can be modified by above step. Extrapolated Motion Track: The cycle of the avatar s motion can be repeated until the end of the background video. Special Behavior Track: The above two types of motion tracks can be merged to create new motion tracks. The spatiotemporal placement algorithm takes several steps: 1. The user has to select a motion clip, decide its motion track starting loca-tion, and give extra parameters such as speed and length of extrapolation. 2. For each motion track 2.1. Compute time slots of appearance 2.2. Compute relative position of the object on frame 3. The user has to select a playback speed. The playback tool decides posi-tion of frames in the panorama and combines frames into a video. IV. EXPERIMENT RESULTS We demonstrate video planning in this section. Each example combines different layers from at least two video clips. The tone and size of all foreground objects are adjusted. In our experiments, the block size used in the object tracking is 7-by-7 pixels. For the performance, we choose 7- by-7 pixels as the tracking size. The patch size used in the proposed motion interpolation procedure is 3-by-3-by-3 pixels. Some examples are illustrated in Fig. 4. The Fig. 4 (a) shows the steps of users design. Users can select avatars from the database and design the lines of the avatars in the generating panorama. From N1 to N4 are four sets of video narratives. N1 illustrates three original persons removed and a flip person inserted. N2 illustrates special effects such as avatars walking in different speed and spark motion. N3 and N4 demonstrate two kinds of motion combined and interpolated then inserted in different layers and video scenes. Table 1 summarizes the computation time in different steps, with respect to the examples shown in Fig. 4. We subdivide the video processing procedure into the following steps, as shown in Fig Object Tracking and Extracting: This algorithm is described in section 2. The target object can be extracted and the background can be restored by inpainting technology. 2. Panorama Generation: In section 2, b, we explain how to generate a pano-rama from the inpainting background clips. 3. Avatar Size Adjustment: We have to consider about the size of avatar to make the result can be normalized. 4. Motion Interpolation: This step includes several portions, including motion analysis, patch assertion, and motion completion via inpainting. Mainly, the algo-rithms discussed in Section 3.A are evaluated. 5. Video Planning: In this step, we have to make the avatars combine with the background. TABLE I. THE PERFORMANCE OF EACH STEP IN THIS SYSTEM. Video Sequences N1 75* frames 1. Object Tracking and Extracting 2. Panorama Generation 3. Avatar Size Adjustment 4. Motion Interpolation 5. Video Planning Total 16 Copyright c 2010 Future Technology Research Association International

5 N2 N3 N4 102*181, 153*140, 140* * * frames frames frames (a) User design steps N1: N1: N1: N2: N2: Copyright c 2010 Future Technology Research Association International 17

6 Volume 1, Number 1, December 2013 N2: N3: N3: N3: N4: N4: N4: Figure 4. Experimental s V. CONCLUSION We proposed a series of mechanisms to generate video narratives from existing video clips. We use patch referencing and frame blending to generate a video sce-ne as a base for the user to plan for video tracks in video narrative generation. Object s tone and size adjustment are used to make the result more realistic. We allow video layers in different videos to be combined by adjusting the saturation, the intensity, and the spatiotemporal placement of layers. We demonstrate the feasibility of using our mechanisms for special effect production in digital movies. Applications of our mechanism include video forgery and special effect production. There still exist few limitations in our proposed algorithm. We consider the following issues as our future work: Non-repeated motions are not able to extrapolate. We are investigating a sophisticated interpolation methods based on 3-D reconstruction techniques. 18 Copyright c 2010 Future Technology Research Association International

7 Realistic shadows cannot be produced by our algorithm. The selected video which is used to generate a video scene should at least contain a foreground object for regulating the size of object correctly. REFERENCES [1] F. Charles, M. Cavazza, S. J. Mead, O. Martin, Alok Nandi, Xavier Marichal, Compelling experiences in mixed reality interactive storytelling, Proceedings of the 2004 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, [2] Z. H. Khan, I.Y. Gu and A. Backhouse, "Robust Visual Object Tracking using Multi-Mode Anisotropic Mean Shift and Particle Filters," IEEE Transactions on Circuits and Systems for Video Technology, Vol. 21 (1) pp , [3] J. C. Tsai, H.-H. Hsu, S.-M. Chang, Y.-H. Wang, C. C. Chao and T. K. Shih, Motion Extraction via Human Motion Analysis, Journal of Mobile Multimedia, Vol. 6(1), 63-72, [4] T. K. Shih, N. C. Tang and J. -N. Hwang, "Exemplar-based Video Inpainting without Ghost Shadow Artifacts by Maintaining Temporal Continuity," IEEE Transactions on Circuits and Systems for Video Technology, Vol. 19, Issue 3, March 2009 pp.: [5] C. -H. Cheung, L. -M. Po, Novel cross-diamond-hexagonal search algorithms for fast block motion estimation, IEEE Trans. on Multimedia, Volume 7, Issue 1, pp , Feb [6] C Rother, V Kolmogorov and A Blake, ""GrabCut": interactive foreground extraction using iterated graph cuts," ACM Trans. Graph., vol. 23, pp , [7] D. Comaniciu and P. Meer, Mean Shift: A Robust Approach toward Feature Space Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, NO. 5, May [8] W. Zhu, C. Luo, J. Wang, and S. Li, Multimedia Cloud Computing, IEEE Signal Processing Magazine, Vol. 28 (3), pp , BIOGRAPHIES Joseph C. Tsai is a Ph. D and and a postdoctoral fellow in the Foundation of Computer Science Lab. at The University of Aizu, Japan. He received the BS and MS and Ph. D degrees from Tamkang University, Taiwan in 2006, 2008 and 2013, respectively. His research interests include computer vision, image and video processing, cloud computing and their applications. Neil Y. Yen received doctorates in Human Sciences at Waseda University, Japan, and in Engineering at Tamkang University, Taiwan in His doctorate in Waseda University was funded by the JSPS (Japan Society for the Promotion of Science) under RONPAKU program. He joins The University of Aizu, Japan as an associate professor since April Dr. Yen has been engaged extensively in an interdisciplinary field of research, where the themes are in the scope of Big Data Science, Computational Intelligence, and Human-centered Computing. Dr. Yen has been actively involved in the research community by serving as guest editors, associate editor and reviewer for international referred journals, and as organizer/chair of ACM/IEEE-sponsored conferences, workshops and special sessions. He is now a member of IEEE Computer Society, IEEE System, Man, and Cybernetics Society, and technical committee of awareness computing (IEEE SMCS). Copyright c 2010 Future Technology Research Association International 19