SIP EXPRESS MEDIA SERVER (SEMS) WITH MPEG4 SUPPORT Pradnya Gaonkar Kodanda Rami Reddy V Sandeep Mamilla Shubhra Pahwa Siva Mullati pradnya.gaonkar@iiitb.org kodandaramireddy.v@iiitb.org sandeep.mamilla@iiitb.org shubhra.pahwa@iiitb.org siva.mullati@iiitb.org International Institute of Information Technology Bangalore www.iiitb.ac.in Technical Report IIITB-OS-2011-11B APRIL 2011
Abstract The aim of the project is to add MPEG-4 video support to SIP Express Media Server (SEMS). It is a free and extensible media application server presently supporting various audio applications only. The server architecture does not support video of any format. Hence the aim of this project is to enhance the capabilities of the SEMS to support MPEG-4 video. The video_playback application that is developed is very similar to the existing audio announcement application in the SEMS, but in the former the server is configured to play MPEG-4 video for the client unlike the latter that plays audio in wave format. Project source can be downloaded from the URL: http://sourceforge.net/projects/semswithmpeg4 2011 Pradnya Gaonkar, Kodanda Rami Reddy V, Sandeep Mamilla, Shubhra Pahwa, Siva Mullati. This material is available under the Creative Commons Attribution-Noncommercial-Share Alike License. See http://creativecommons.org/licenses/by-nc-sa/3.0/ for details.
Acknowledgement We take this opportunity to thank everyone who has been a source of help and support in the course of this project. We also express special gratitude towards our advisors, Prof. Shrisha Rao and Dr. Madan Pande for their encouragement and valuable guidance that helped us to overcome hurdles and keep ourselves motivated.
Contents 1 Introduction 1 2 System Architecture and Components 2 3 Implementation Details 6 3.1 Goal 1: Setting up the Network between a Client and the SEMS 6 3.1.1 SEMS and X-Lite Client Setup for Announcement Application.......................... 7 3.1.2 SEMS and UCTIMS Client Setup............ 7 3.2 Goal 2: Video Application Development for the SEMS.... 7 3.2.1 Modifications in the SEMS Core............. 7 3.2.2 Development of the video_playback Application.... 9 3.2.3 Development of the Mp4 Application (Implementation of MPEG4 Codec).................... 9 4 Challenges Faced 9 4.1 Setting Up the Environment................... 10 4.2 Challenges Faced in the Application Development....... 10 5 Results 11 6 Conclusion 13 References 14
List of Figures 1 SER as SIP stack (Source: [1]).................. 3 2 SEMS layers with the plug-ins (Source: [1]).......... 4 3 Session objects for (a) request and (b) termination (Source: [1]) 5 4 Media flow in SEMS (Source: [1])................ 5 5 Connection established between SEMS and the X-Lite client. 11 6 Wireshark Trace of the established connection between SEMS and the X-lite client....................... 12 7 Confirmation that the video_playback application was successfully loaded.......................... 13
1 Introduction In the present scenario of ever developing technologies, audio-video streaming on the web has become very popular. The core of the SEMS architecture allows basic call, audio processing and loads plug-ins such as Voicemail, ISDN Gateway, Conferencing, Announcement and Echo, using codecs such as G711.u, G711.a, GSM 06.10, ilbc and Speex, which are all audio based codecs and thus support only audio streaming. The capabilities of the server have not yet been extended to support any kind of video. The SEMS is licensed under the GPLv2. The latest version is the SEMS 1.4.0. The server has some special features like MP3 file writer, Text-tospeech, Interactive voice response (IVR) etc. Standard PC architectures can be used for the configuration of the server. Application development in SEMS is very simple. A plug-in can be developed using the server s API in C++. A few changes in the source would also have to be made for the server to identify the newly built application. Problem Statement SIP Express Media Server (SEMS) is a free media server used for various audio applications and is extensible through plug-ins. The applications can be developed and made to pickup the existing codecs from the SEMS core. Applications like audio conference, voicemail, announcement mailbox etc. are available in SEMS. But the server architecture has not been provided with video handling functionality. The goal of the project is to add MPEG-4 video support to the SEMS, by making necessary modifications in its core and available applications. Literature Study A lot of research has been going on in the field of video streaming and media servers. But the SEMS has not been explored much yet. It has been proposed in [4], a functional architecture which enables interworking between MPEG-4 DMIF and SIP for heterogeneous IP video conferencing. DMIF-SIP IWF (Delivery Multimedia Integration Framework DMIF Session Initiation Protocol - SIP Inter working Function) is a server used for signaling translation. SIP User Agent Client makes a call to the MPEG-4 1
DMIF Terminal. The DMIF to SIP signaling makes MPEG-4 DMIF Terminal to call SIP Client. In [3] LIVE 555 Server [2] is described and methodologies to integrate it with baseline H.264 streamer encoder running on TI s embedded platform are presented. This streams live video over the network and displays it on remote systems. The same can then be analyzed and optimized. The media server is modified to make it compatible with the other hardware. The video is encoded and is streamed to the Live555 media server for multicast over IP network. The remote stations can display the video using the open source VLC media player, which contains the relevant H.264 codec. As far as the SEMS is concerned, it supports the most commonly used patent free codecs. Thus codecs like g729 are not supported. The audio codecs shipped with the SEMS are: 1) G711.u and G711.a: G.711 is an ITU-T standard for audio companding. It is primarily used in telephony. The standard was released for usage in 1972. Its formal name is Pulse code modulation (PCM) of voice frequencies. 2) GSM 06.10: GSM 06.10 was the first digital speech coding standard used in the GSM digital mobile phone system. Also called Full Rate (FR) codec. 3) ilbc (Development Version): Internet Low Bitrate Codec (ilbc) is a royalty-free narrowband speech codec, developed by Global IP Solutions (GIPS) formerly Global IP Sound. 4) Speex (Development Version): Speex is an Open Source/Free Software patent-free audio compression format designed for speech. The MPEG-4 technology needs to be implemented with the knowledge of the coding and decoding algorithms available for the same. The present standard for it is the object-based or content-based coding, where a video can be handled as a set of objects. 2 System Architecture and Components The Session Initiation Protocol (SIP) is a signaling protocol. The SEMS supports many applications. It has a slim core as a framework and is extendable due to its capability to support plug-ins. The following are the components that describe the SEMS [1]. 2
SIP Express Router (SER) SER sockets are used to reply to the incoming requests when routed to the SER. It also makes requests when needed. The Real-time Transport Protocol (RTP) is used for delivering media packets over the IP. The SER can work as a proxy server and can be configured for special purposes like the applications in the SEMS. The SER has very good functionality, high performance, flexibility and interoperability. It is also found to support databases. Also, it has been improved upon by the SIP tools. It also works across different platforms. It is written in C and is ported to other operating systems like Linux. It can support IPV4 and IPV6. Figure 1: SER as SIP stack (Source: [1]) As shown in the figure the calling device makes an SIP request, it is routed to the SEMS via the SER through its socket. The SEMS then processes the request and replies to the calling device using the RTP. 3
Control/Signaling and Media/RTP layers Figure 2: SEMS layers with the plug-ins (Source: [1]) The Control/Signaling and Media/RTP layers are shown in the figure. The SEMS is flexible enough to allow users to build their own applications. As shown in the figure, the applications are in the control layer where the signaling also takes place. The Media layer processes media or RTP and holds media plug-ins. Core and Plug-ins As shown in the Figure 2, the vertical partitioning of the SEMS structure consists of the core and the plug-ins. The slim core (framework) of the SEMS carries out the basic processing and the plug-ins support the application. These plug-ins are of two types: Media plug-ins (audio/image) for media processing. The application plug ins for application logic. Session Objects A session object is created by the application for every session. These session objects reside in the session container which in turn manages their lifecycle. 4
As shown in the figure, when the SEMS gets the INVITE request, it asks the application to create session objects. The application creates the objects and returns them to the container where the objects reside throughout the session. Figure 3: Session objects for (a) request and (b) termination (Source: [1]) When the termination request (BYE) is made, the container returns the objects and closes the session. The objects cease to exist beyond the session. RTP and Media Processing Figure 4: Media flow in SEMS (Source: [1]) The RTP packets are received by the receiver from the sockets. The RTP object of a session sends this request to the session as an input. The output 5
is generated after the session processes the request which in turn would be sent through the RTP object as the RTP reply. Threads and Call Groups Each session creates a thread for signaling to implement the application logic. Python/C++ Applications Interactive voice response (IVR) plug-in helps the server to run the Python scripts for an application using the embedded python interpreter. But Python is not widely used and is replaced by C++. The SEMS supports many applications like conferencing, mailbox, voice mail, announcements etc. due to the above components. In this project, the existing capabilities have to be extended to support MPEG-4 video. MPEG4 is a standard for building and delivering multimedia over the internet for audio and video. It was defined by MPEG (Moving Picture Experts Group) committee within the ISO. MPEG-4 specifies a very efficient video codec using algorithms for high quality video over the internet. 3 Implementation Details The project goal has been split into two broad sections: Goal 1: Setting up the network between a client and the SEMS. Goal 2: Video application development for the SEMS. 3.1 Goal 1: Setting up the Network between a Client and the SEMS The SEMS was configured with a few compatible clients for testing the applications that are shipped along with it. The SEMS was installed on Ubuntu 10.04 and Fedora for testing purposes. The X-Lite, UCTIMS and EyeBeam clients were configured with the server. 6
3.1.1 SEMS and X-Lite Client Setup for Announcement Application X-lite is a SIP-based softphone used for audio applications. A few modifications in the SEMS configuration files, reg_agent.conf and sems.conf had to be made. The server client setup was completed by changing system network and proxy settings.the server response was observed during the call setup and the answer phase. The same was useful to find the entry points in the SEMS for the codec addition. Thus, the X-Lite Client was successfully configured with the SEMS server on localhost. 3.1.2 SEMS and UCTIMS Client Setup As adding video support to the server is the goal of this project, a video client was required in order to test the newly added video application. Thus, UCTIMS Client was downloaded and installed on Ubuntu 10.04. Again the configuration settings of the server were modified as per the client and it was configured so as to work with the server. 3.2 Goal 2: Video Application Development for the SEMS The entire application development process was divided into the following modules for simplicity and ease of development. The process included the SEMS core modification and application wrapper creation for adding video support to the media server. The applications are developed by building application wrappers and then plugging them into the system by bringing about the required changes in the SEMS core. The implementation phase for the application was broken down into two sub-phases: Extending the SEMS capability to extract just the MPEG-4 audio from the MPEG-4 video. Completing the application development process by extending the above to run the complete MPEG-4 video. 3.2.1 Modifications in the SEMS Core The SIP Express Media Server supports various applications to provide audio services to clients. But the same has not been extended to support video. 7
The SEMS core and the existing source code is vast and the aim of this module was to identify the parts of the source code where the modifications have to be made to extend it for video support. A detailed Solution Study of previous attempts made in adding video support to SEMS was done and the following approach was adopted: Adding support for multiple media streams per session. Overhauling the Offer/Answer code (AmSDP.cpp and in AmSession.cpp). Adding support for pass-through video media. Enhancing announcement code to accept video with video mixing. Considering the existing announcement application as a reference, the video_playback application was built and placed in the application folder. This application (like the audio announcement application) was designed to announce a video to the SIP client. The client would request for the video playback from the SEMS which would instead fetch the MPEG-4 video file from the SEMS library where it is located and would play the same to the client. The ways in which this video_palyback application differs from the audio announcement application in SEMS is that, the existing audio application plays the wav audio file that is in the SEMS library and does not need to create any multisession or do multistreaming, which is a necessity in the case of the MPEG-4 video announcement. The video application thus built needs a new codec to run successfully and cannot be run with help of the audio codecs that are used for the existing audio application in SEMS. Libavcodec was identified as the best suited video codec for the application and a corresponding wrapper for MP4 codec was included in the list of codecs in codec.h file. Apart from the codec.h file, the following files in the SEMS core had to be modified for recognizing the newly built video_playback application: AmSession.cpp: For adding the Multisession support (audio + video streams in MPEG-4 video) AmConfig.cpp: For changing the basic server configuration to detect the video AmSdp.cpp: For SDP definition of the video amci.h: For codec definition along with codec.h 8
3.2.2 Development of the video_playback Application The application video_playback was created using the SEMS API in C++ and placed in the apps folder of the SEMS tree. This application has been built similar in functionality to the Annoncement application in SEMS. The video_playback folder contains the following files: video_playback.h: Header file video_playback.cpp: C++ Source file Makefile: The Makefile Readme: Documentation 3.2.3 Development of the Mp4 Application (Implementation of MPEG4 Codec) The existing MP3 application in the SEMS source was studied and an application called MP4 was built. This was placed in the apps folder in the SEMS source. For this, the Makefile and the CMakefile were created. The goal was to extract and read the audio part of an mp4 file. The code to do the same was written with the help of the available opensource which is cross-platform solution to record, convert and stream audio and video. It includes the libavcodec an audio/video codec library. The following are the files inside the mp4 application folder: mp4.c: C Source file. Makefile: The Makefile. CmakeLists.txt: For cross platform compilation. 4 Challenges Faced The following challenges were faced during the development process of the project: 9
4.1 Setting Up the Environment Port Conflict: The default port for SIP proxy is 5060, but at times it did not work, throwing an error which said that the port was already in use. If the audio port is busy, it threw an error that the following port was busy. Thus, the client had to be restarted in order to get it back to a working condition. 4.2 Challenges Faced in the Application Development SEMS source code compilation and successful running of the server was the initial challenge faced by the team. But the same was tackled smoothly and the server was installed and was configured with the client. The wrapper application development was done and it had to be recompiled to plug it in the SEMS structure. The testing phase was dependent on the network setup between SEMS and a video SIP client. Not much of testing could be done due to the dependencies. 10
5 Results The following are the snapshots of the testing process carried out separately on the network setup and the video_playback application as per the goals mentioned in the previous sections. The figure below shows SEMS and Xlite client configured on the local host. The server is running the announcement application for the client having the address 127.0.0.1@mydomain.net. This application plays the default.wav file in the SEMS core. The details regarding how the call is established through the INVITE message, the used protocols, client and server IDs etc. can also be seen in the call establishment report. Figure 5: Connection established between SEMS and the X-Lite client 11
Figure 6: Wireshark Trace of the established connection between SEMS and the X-lite client The Wireshark traces of successful call establishment and termination phases through the INVITE and BYE messages is shown in the figure above. The SIP was filtered out from the rest of the log to get only the SIP related information. It shows that 474 bytes were captured with localhost acting as both transmitter and reciever. The source and destination ports are 5081 and 5060 respectively. 12
Figure 7: Confirmation that the video_playback application was successfully loaded The figure above shows the successful loading of the video_playback application in the SEMS. The highlighted portion shows the log message regarding the generation of the video_playback.so file for the application and also the acceptance of the application by the server as a plug-in. 6 Conclusion The aim of the project of adding video support to the SIP Express Media Server was partially fulfilled. The application development for the same has been completed and the mp4 codec has been plugged into the apps folder of the SEMS. The SEMS successfully loads this video_playback application built for MPEG4 video announcement. The modified development plan was to extract the MPEG4 audio out of the video. Application for the same has been built and integrated with the core. The SEMS is now partly ready to support video, as the application built can extract the audio from the MPEG4 clip under proper codec configuration conditions. The basic network configuration of SEMS with SIP clients has been tested. Also the software 13
support for MPEG4 video added to the SEMS has been tested partly under certain compatibility constraints. The project can be taken further by enabling the SEMS to successfully read the video clip in its library (both audio and video portions) by appropriate selection and installation of the FFMPEG library packages. Also the multistreaming capability can be added to the SEMS core which would help to stream simultaneous audio and video portions of the MPEG video clip to the client. The adding of video support to the SEMS can further trigger the introduction of new video based applications in the server that can share the SEMS core along with the presently available audio applications. References [1] S. Sayer, Sems design overview, Inside the Media Server, [Online] available: http://www.iptel.org/sems, [Oct, 2006]. [2] LIVE555, [Online] available: http://en.wikipedia.org/wiki/live555, [Aug, 2006]. [3] N. Vun, Implementation of an Embedded H.264 Live Video Streaming System, in 14th International Symposium on Consumer Electronics, Singapore, [2010]. [4] T. Ahmed, A. Mehaoua, and R. Boutaba, Interworking between SIP and MPEG-4 DMIF for Heterogeneous IP Video Conferencing, in The IEEE International Conference Communications (ICC), vol. 4, [May, 2002], pp. 2469 2473. 14