Internet Voice, Video and Telepresence Harvard University, CSCI E-139. Lecture #12

Transcription

1 Internet Voice, Video and Telepresence Harvard University, CSCI E-139 Lecture #12 Instructor: Len Evenchik IAD AESK Lecture Agenda Welcome Course Logistics Final Exam due next Monday May 13 th Telepresence Systems and Protocols IETF WG CLUE QoS and Telepresence WebRTC What do we do next? One Minute Wrap-Up (please complete online) Morton Heilig's Sensorama, 1961 Please note: Neither Harvard nor the instructor are endorsing any specific vendor product or service. Len Evenchik Page 1

2 Telepresence: Stereoscopy and Sensorama Normal Rockwell, 1922 Morton Heilig's Sensorama, 1961 L. Evenchik 2012, SK2012AE-2 Course Logistics L. Evenchik 2012, SK2012AE-2 Len Evenchik Page 2

3 Q&A and Topics from Last Week Dynamic Adaptive Streaming over HTTP (DASH) Dynamic adaptive streaming allows the client to specify the quality of the video that it receives from the server. Forms of adaptive streaming were being used prior to HTTP based video streaming. Server has multiple versions of the same video file encoded at different bit rates, and a manifest is created with information on all the various options. Client and HTTP server maintain a two-way control channel, separate from the media. The client tells the server the quality of the video file it wants at any time. Allows for both VOD and Live Streaming. The HTTP streaming protocol, and the dynamic bandwidth adaption are distinct from the codec and the wrapper Len Evenchik (evenchik@fas.harvard.edu) Page 3

4 Telepresence Systems and Protocols Some History of Telepresence (1) Punch Magazine talked about Telepresence almost 100 years ago. Marvin Minsky wrote about Telepresence in Omni magazine in June In the article, he said, The biggest challenge to developing telepresence is achieving that sense of being there. Please see My introduction to telepresence, in the form we talk about it today, was from a student in the networking course I was teaching in about The company was called TeleSuite and it was founded in (Wikipedia) Len Evenchik (evenchik@fas.harvard.edu) Page 4

5 Some History of Telepresence NSF, DARPA, and the Internet 2 and other research communities have been actively working on Telepresence and related areas for over a dozen years. Internet2 video services» Interactive displays» Gigapixel cameras» There are many many definitions of telepresence today, and a large number of vendors are working on components, systems and environments. Telepresence Systems: the name has been used for many years. Room System circa 1994 CLI Standalone System 1997 TeleSuite, circa 2002 Len Evenchik (evenchik@fas.harvard.edu) Page 5

6 Telepresence Current Definition and Requirements Telepresence tries to create the reality or experience of actually being there via a video conference. System requires multiple HD displays and cameras. The displays should present the far-side participants in a natural way System requires high quality and natural sounding audio for both the primary speaker (at that moment in time) and the other participants. This allows for a natural conversation. System attempts to convey the natural eye contact that you typically have in a meeting, and also attempts to convey the natural gestures that typically occur in a meeting. System requires seamless content sharing between sites, which is as difficult as the video problems. Note that the above are not defined as technical requirements. Telepresence System, the Current Generation Cisco 3000 Lifesize 200 Polcyom RPX Note that there are a dozen vendors in this market today. Len Evenchik (evenchik@fas.harvard.edu) Page 6

7 Telepresence: Basic Technical Requirements (1) Telepresence tries to create the reality or experience of really being there via a video conference. Requires multiple HD displays and cameras. Today, this means multiple independent codecs that are coordinated by a system controller. Camera location and stability (no vibration), and image overlap (if present) must be very precise. Natural sounding audio requires multiple microphones, excellent ambient noise control and echo cancellation, fullduplex audio (of course) and speaker locatization. L. Evenchik 2012, SK2012AE-2 Far-End Display Polycom RPX, 2010 Len Evenchik (evenchik@fas.harvard.edu) Page 7

8 Telepresence Room Layout Polycom RPX, 2010 Camera Placement for Eye Contact Polycom RPX, 2010 Cisco Telepresence Camera is above screens Len Evenchik Page 8

9 Where are the Microphones? Cisco TX9000 Polycom RPX Telepresence Room Layout Cisco CTS3000 Polycom RPX Len Evenchik Page 9

10 Telepresence: Technical Requirements (2) Multiple independent H.264 video streams, each at 5Mbps for 1080p display. This bit rate was typical last year, but I have seen 2 Mbps specified in some current systems. Multiple audio streams and multiple content streams. Different number of audio versus video streams. Session management is done (today) via H.323, TIP or SIP. All systems use some type of proprietary protocol extension for overall system control. Users have just a simple GUIbased control panel (simpler than the one in this classroom.) An MCU or managed bridge service is required. Typically a vendor managed network service with a tight SLA for bandwidth, delay, jitter and error rate is required. Telepresence Building Blocks Consumer Internet Private IP with QoS and overlay networks Video conferencing systems will be a mix of telepresence and the other endpoints we have talked about Len Evenchik (evenchik@fas.harvard.edu) Multipoint conference unit with support for proprietary, H.323 and SIP end points Telepresence System Manager Page 10

11 Multipoint Bridging (1) A Multipoint Conference Unit (MCU) or Bridge is the video equivalent of a teleconference bridge. It is a device that joins multiple video conferences together so that three or more users can be on the same call. (MCU = MP + MC) A small bridge that supports three of four users can be part of a client (PC or laptop), and a larger bridge is typically a standalone device. We use a bridge system for this course. Management and Gateway functions are typically included in standalone systems. This includes support for multiple protocols (SIP, H.323, Cisco TIP), access to Skype and vendor-specific VoIP systems, and support for POTS. All Telepresence system assume the use of a bridge service along with other high-end support functions (such as realtime call monitoring and debugging.) Multipoint Bridging (2) Large bridges are expensive, and there are about a dozen vendors in the market today. Large organizations can purchase and manage their own bridges. When part of a private network, it is more straightforward to ensure security and QoS. Bridge services are also provided by carriers, bridge service providers, and other organizations (such as Internet 2.) Bridging services are different than web-based video conference services, but they do include common functions. The end user can see a number of different layouts: speaker preference, continuous presence or Hollywood Squares, or a combination of each. Len Evenchik (evenchik@fas.harvard.edu) Page 11

12 Current Systems use a Combination of Speaker Preference and Hollywood Squares IETF CLUE WG ControLling multiple streams for telepresence (clue) Len Evenchik (evenchik@fas.harvard.edu) Page 12

13 IETF CLUE WG ControLling multiple streams for telepresence (clue) IETF CLUE IETF WG for ControLling multiple streams for telepresence Current telepresence systems are based on standards such as RTP, SIP, H.264, and H.323 but they do not interoperate without operator assistance and additional equipment Most basic need is to describe and negotiate the use of the multiple streams of audio and video used by the systems The goal of the WG is to create specifications for SIP-based conferencing systems to identify and coordinate the use of multiple media streams. Note that this is SIP based. About a dozen IDs related to CLUE as of May Source is CLUE Charter Len Evenchik (evenchik@fas.harvard.edu) Page 13

14 How do you identify the screens? Cisco 3000 Lifesize 200 Polcyom RPX IETF CLUE (2) As described in the CLUE charter, interoperability means the entities need to specify and understand the following information about media streams. Spatial relationships of cameras, displays, microphones, and loudspeakers - relative to each other and to likely positions of participants Viewpoint or field of view/capture for camera/microphone/display/ loudspeaker - transmitter needs to understand how best to compose the video streams for receivers, and the receiver needs to know the characteristics of its received streams Aspect ratio of cameras and displays Usage or type of stream. For example, is the stream a person, a presentation, or document camera output? Which source (s) does a receiver want to receiver: center camera, left camera or voice activated. Work is also being done to define the interaction between CLUE and SDP messages Source is CLUE Charter and IDs Len Evenchik (evenchik@fas.harvard.edu) Page 14

15 Proposed use of SDP for Multiple Streams (1) Parameters for an individual encoding Source is ID draft-romanow-clue-sdp-usage-02 Proposed use of SDP for Multiple Streams (2) An Encoding Group is defined as a set an individual encodings Source is ID draft-romanow-clue-sdp-usage-02 Len Evenchik (evenchik@fas.harvard.edu) Page 15

16 QoS, NAT and Security for Telepresemce NAT for Telepresence Streams The same NAT and Security issues we talked about earlier apply to Telepresence systems. Debugging is actually more difficult...abridged SDP. Offerer -> Answerer o=alice IN IP4 tels1.atlanta.com m=audio RTP/AVP 0 97 a=rtpmap:0 PCMU/8000 a=rtpmap:97 ilbc/8000 m=video RTP/AVP c=in IP4 telsv1.atlanta.com a=rtpmap:98 H264/90000 a=fmtp:98 profile-level-id=42a01e; packetization-mode=0; etc. m=video RTP/AVP c=in IP4 telsv2.atlanta.com a=rtpmap:98 H264/90000 etc.. m=video RTP/AVP c=in IP4 telsv3.atlanta.com a=rtpmap:98 H264/90000 etc.. Len Evenchik (evenchik@fas.harvard.edu) Page 16

17 Bandwidth with/without QoS for 5Mbps Telepresence video stream Consistent throughput when QoS marked Traffic w/o QoS Packet Loss with/without QoS for 5Mbps Telepresence video stream Traffic w/o QoS Showed Packet Loss Traffic with QoS, no loss The traffic sent without QoS showed consistent packet loss, while the traffic with QoS marked showed almost no loss. Len Evenchik Page 17

18 Jitter with/without QoS for 5Mbps Telepresence video stream There is increased jitter when QoS is not marked. Jitter w/o QoS Jitter with QoS RTCWeb / WebRTC Real-Time Communication in WEB-browsers (IETF) Web Real-Time Communications (W3C) The IETF and W3C are collaborating on the development of these standards and protocols Len Evenchik (evenchik@fas.harvard.edu) Page 18

19 RTCWeb: IETF Working Group and WebRTC: W3C Working Group WebRTC Use Case Browser to Browser Real Time Video Without Plug-ins Web Server Browser JavaScript Browser User Client HTTP Signaling, Media Capabilities, etc Voice and Video (SRTP and DTLS) Browser JavaScript Browser User Client Len Evenchik Page 19

20 WebRTC Video Chat Client ( IETF and W3C Collaboration The IETF and W3C are collaborating on the development of standards, protocols and APIs to allow for real-time communication (voice, video and data) between web browsers without the use of additional plug-ins. This is part of the trend to integrate functionality directly into web browsers. From the WebRTC charter (W3C): The mission of the Web Real-Time Communications Working Group. is to define client-side APIs to enable Real-Time Communications in Web browsers. and The definition of the network protocols used to establish the connections between peers is out of scope for this group; in general, it is expected that protocol considerations will be handled in the IETF. From the RTCWeb charter (IETF): The IETF WG will produce architecture and requirements for selection and profiling of the on the wire protocols. The architecture needs to be coordinated with W3C. The IETF WG work will identify state information and events that need to be exposed in the APIs as input to W3C. The W3C will be responsible for defining APIs to ensure that application developers can control the components. Len Evenchik Page 20

21 WebRTC Block Diagram (source is webrtc.org) WebRTC APIs The W3C defines 3 APIs for WebRTC (note that current browser implementations might name them differently) getusermedia this interface represents sending and receiving a stream of audio and/or video from local and remote devices. RTCPeerConnection - connecting to remote peers, browser to browser, using NAT-traversal technologies such as ICE, STUN, and TURN. Note that the W3C spec says that the actual communications are coordinated via a signaling channel which is provided by unspecified means. RTCDataChannel - represents a bi-directional data channel between two peers. Note that the W3C spec says that the actual wire protocol between the peers is out of the scope for this specification. Source is WebRTC 1.0, working draft August 21, Len Evenchik (evenchik@fas.harvard.edu) Page 21

22 Direct Browser-to-Browser Communication What procedures and functionality are required for direct browser-to-browser real time video communication The browsers must identify each other in some way (IP address, user address and/or user name) and then initiate point-to-point communication. This communication must be able to work with NATs and firewalls. The browsers must coordinate and agree on the codecs that they will use and how they will display the video. The browsers must send and receive real time audio and video, and should also support QoS. The browsers and clients must report errors and status, and be able to gracefully close the connection. RTCWeb Building Blocks As you study each of these building blocks it is very helpful to think about the protocol framework that we have used for real-time communication. In other words, signaling, negotiation of capabilities, and sending media Browsers that support the HTML5 Video tag Support for the WebRTC APIs from W3C Audio and video codecs and related wrappers WebSockets enables usage of SIP with HTTP & web browsers STUN, TURN and ICE JSEP (Javascript Session Establishment Protocol) DTLS / SRTP Len Evenchik (evenchik@fas.harvard.edu) Page 22

23 WebRTC Interop Testing From SIPit 30 held in February 2013 WebRTC Video Chat Client ( Len Evenchik Page 23

24 WebRTC Video Chat Demo ( WebRTC Use Case Browser to Browser Real Time Video Without Plug-ins Web Server Browser JavaScript Browser User Client HTTP WebSockets SIP, JSEP, etc Voice and Video (SRTP and DTLS) Browser JavaScript Browser User Client Len Evenchik Page 24

25 WebSocket Protocol as a Transport for SIP (1) WebSocket Protocol as a Transport for SIP (2) Len Evenchik (evenchik@fas.harvard.edu) Page 25

26 WebSocket Protocol as a Transport for SIP (3) Web Sockets Browser Support (May 2013) Source Jitter with QoS Len Evenchik (evenchik@fas.harvard.edu) Page 26

27 Javascript Session Establishment Protocol WebRTC to SIP-based System Web Server WebRTC to SIP Gateway HTTP, WebSockets SIP, JSEP, etc SIP SIP Proxy Browser JavaScript Voice & Video DTLS & SRTP UDP & RTP SIP Browser User Client SIP Client Len Evenchik Page 27

28 WebRTC to SIP Client (Source IETF) WebRTC Client to SIP Network source ttp://code.google.com/p/webrtc2sip/ Len Evenchik Page 28

29 WebRTC to SIP Demo ( WebRTC Links of Note Test page for WebRTC browser support Basic WebRTC video conference implementation Javascript SIP Library WebSockets and other browser support tables Browser interoperability notes for WebRTC HTML5 JavaScript SIP client Examples of RTCPeerConnections Len Evenchik Page 29

30 What do we do next? Thank you! Please remember to submit a One-Minute Wrap-up! It was a pleasure to meet you and work with you this term! Please keep in touch. Len Evenchik (evenchik@fas.harvard.edu) Page 30