Introduction to WebRTC. Luis López

Transcription

1 Introduction to WebRTC Luis López

2 Real-time Communications (RTC) 2

3 WebRTC: a definition WebRTC: a new player Framework, protocol and API that provide real-time voice, video and data in web browsers and other applications (by Salvatore Loreto) WebRTC as a framework Technological capabilities enabling RTC on web browsers: Codecs, NAT traversal, security, transports, etc. Basing on standards: RTCWeb Working Group of the IETF for protocols WebRTC as an API Capabilities are exposed to web developers in an abstract manner and adapting to HTML5 philosophy Basing on standards: WebRTC Working Group of the W3C for JavaScript APIs 3

4 Why WebRTC Other WWW 4

5 Why WebRTC Plug-in WebRTC 5

6 Who is who in WebRTC 6

7 What s WebRTC: browser architecture *This image has been borrowed from: 7

8 WebRTC: P2P communications *This image has been borrowed from: 8

9 WebRTC: NATs *This image has been borrowed from: 9

10 Developing WebRTC apps rtc/basics/ 10

11 Example

12 Why Kurento? WWW RTC developer experience Before WebRTC After WebRTC Next natural step Developing the client side Begin End Begin End Unified APIs Standards FOSS Multiplatform Unified APIs Standards FOSS Multiplatform Developing the infrastructure side Begin Unified APIs Standards FOSS Multiplatform End 12

13 Kurento media framework components Low level media capabilities Transport, encode/decode, transcode, CV, mix, etc C/C++ C++ Server + Gstreamer plugins Kurento Server Kurento Application Server Signaling and control capabilities SIP, HTTP, RESFul, etc. Java Spring + JBoss + Mobicents SIP Servlet Kurento Android SDK Kurento WWW SDK RTC library for Android Java and C Multicodec support HTML5 multimedia development Video tag, WebRTC Javascript 13

14 Kurento Application Server: extending the WWW development model HTTP request: I want this resource Signaling request: I want this media Process WWW request - DDBB access - Transactions - Security tools - Etc. HTTP response: The resource Process media request - API - DDBB access - Transactions - Security tools - Etc. Signaling response: The media is here Intuition behind traditional WWW Applications (Servlets, ASP, PHP, Rails, etc.) Intuition behind Kurento development APIs: Multimedia RTC is just another feature of your application 14

15 SRC API: media elements and media pipelines Element Provides a specific media functionality Building block Send/receive media Process media Transform media The API provides a toolbox of media elements ready to be used. New media elements can be added pipeline Chain of media elements implementing the desired media functionality. The API provides the capability of creating media pipelines by joining media elements of the toolbox Element PlayerEndPoint Filter WebRtcEndPoint SRC SRC 15

16 Application architecture Other systems and databases Signaling and WWW traffic Signaling and WWW traffic HTTP Servlet Web SIP service DD.BB. Servlet s Java EE compatible container Specific application logic API JMS Decode Video Augmented Reality Video Playing and Recording Computer Vision Encode Video Kurento Server Infrastructure 16

17 Possible use cases: just integrate with Java EE and GStreamer Verticals E-Health Kurento + HAPI ( P2D video conferences as Electronic Health Records Smart cities Kurento + NGSI + OpenCV + Google Maps City crowds movement tracking Traffic density tracking Telco infrastructures Kurento + Mobicents IMS application server B2B & B2C WWW RTC Kurento + CRM APIs Enriched video conferencing with customer personal data Kurento + ESB Billing, video event processing, physical security, etc. 17

18 Application example: requirements Kurento Server Infrastructure 18

19 SRC Application example: = "MyWebRtcService", path = "/pathtoservice") public class MyWebRtcService implements WebRtcHandler { public void onrequest(webrtcrequest request) { //I can authenticate using any of the Java EE available mechanisms PipelineFactory mpf = request.getpipelinefactory(); Pipeline mp = mpf.createpipeline(); //I could decide the type of processing connecting to a DDBB JackVaderFilter filter = mp.newfilter().withtype(jackvaderfilter.class).build(); RecorderEndpoint recorder = mp.newrecorderendpoint().withuri("file:///myfile.webm"); filter.connect(recorder); HttpEndpoint httpendpoint = mp.newhttpendpoint().build(); filter.connect(filter); //I could connect only audio or video separately } request.start(filter, filter); To file RecorderEndpoint To Network Video tag or CDN HttpEndpoint JackVaderFilter Source WebRtcEndpoint From network 19

20 Application example: result 20

21 element toolbox Transport WebRtcEndpoint RtpEndpoint HttpEndpoint Repository PlayerEndPoint RecorderEndPoint Group communications MainMixer GridMixer RoundRobinMixer Filters FaceRecognitionFilter (events) JackVaderFilter QR/Barcode detector PlateRecognitionFilter (events) ColorTrackingFilter (events) And growing Available as part of 21

22 elements: WebRtcEndpoint Full implementation of the RTCWeb protocol stack SRTP ICE DTLS Allow sending and receiving WebRTC flows at the media server infrastructure WebRTC traffic Source WebRTC traffic WebRtcEndpoint 22

23 elements: HttpEndpoint downloading compatible with the HTML5 video tag WebM (Chrome, Firefox) MP4 (Chrome, Firefox, IE, Safari) uploading compatible with HTML file input tag Multipart support HTTP WebM/MP4 HttpEndpoint 23

24 elements: RtpEndpoint Full-duplex RTP multimedia exchange H.264 H.263 VP8 Many different audio codecs supported RTP traffic Source RTP traffic RtpEndpoint 24

25 elements: UriEndpoints PayerEndpoint Play media from file or URL Support for most popular formats RecorderEndpoint Record media to file or URL WebM MP4 RecorderEndpoint Source PlayerEndpoint 25

26 elements: Mixers (in progress) Make possible group communications ForwardingMixer One-to-many replication of flows A source can be assigned to any of the sinks Multiple sources supported MainMixer Mixes media A source can be assigned to a combination of sinks Multiple sources supported Source Source Mixer Source 26

27 SRC Filters Seamless integration into OpenCV Face recognition Augmented reality Subtitle adding Color manipulation QR detection People counter Plate recognition Etc. Filter 27

28 SRC Filter with events Filters can provide events to the application Events are generated at the media server Events can be propagated to the client app Code example: Pipeline mp = mpf.create(); PlayerEndPoint playerendpoint = mp.newplayerendpoint( " ZBarFilter filter = mp.newzbarfilter().build(); playerendpoint.connect(filter); filter.addcodefounddatalistener(new EventListener<CodeFoundEvent>() public void onevent(codefoundevent event) { session.publishevent(new ContentEvent(event.getType(), event.getvalue()));... Events Source PlayerEndpoint ZBarFilter 28

29 SRC SRC The magic of pipelines: Transparent media adaptation Agnostic media adaptor Acts every time a source is connected to a sink Adapts media formats as required by the involved media elements 100% transparent for the application developer Element An agnostic media adaptor is hidden behind every connection making compatible element formats transparently Element VP8 H

30 SRC Complex examples: Heterogeneous group communications Source WebRtcEndpoint RecorderEndpoint Source WebRtcEndpoint Mixer Source Filter RtpEndpoint 30

31 Complex examples: WebRTC to HTTP Source WebRtcEndpoint RecorderEndpoint Mixer HttpEndpoint Mixer HttpEndpoint HttpEndpoint 31

32 Complex examples: WebRTC Playing Source WebRtcEndpoint RecorderEndpoint Mixer Filter HttpEndpoint SRC Source PlayerEndpoint 32

33 WebRTC CV game SRC SRC SRC SRC KkRY ChromaFilter FaceOverlayFilter WebRTC full duplex client video application WebRtcEndpoint SRC MirrorFilter PointerDetectorFilter Pipeline Upload to Youtube 33

34 Collaborations welcome Thank you very much for your attention Complains, suggestions and comments can be sent to: Luis López 34