WebRTC.... GWT & in-browser computation. Alberto Mancini, Francesca Tosi JooinK.com

Transcription

1 WebRTC... GWT & in-browser computation Alberto Mancini, Francesca Tosi JooinK.com

2 WebRTC Plug-in free realtime communication WebRTC is a free, open project that enables web browsers with Real-Time Communications (RTC) capabilities via simple Javascript APIs. source: webrtc.org

3 WebRTC Just another JS API or WebRTC is a new front in the long war for an open and unencumbered web Brendan Eich (Mozilla CTO and inventor of JavaScript)?

4 Browser Piercing a Javascript Application cannot access the filesystem and then FileAPI open f/d socket connections and then WebSockets use graphics accelerator and then WebGL span multiple threads and then WebWorkers

5 Browser Piercing a Javascript Application cannot acquire audio and video and then WebRTC communicate P2P and then WebRTC

6 WebRTC across platforms Chrome (mobile too) Firefox (mobile too) Opera Native Java and Obj-C

7 WebRTC JS-API Acquiring video and audio navigator.getusermedia(constraints, successcallback, errorcallback);

8 WebRTC JS-API Acquiring video and audio navigator.getusermedia(constraints, successcallback, errorcallback);

9 WebRTC JS-API Constraints Controls the contents of the MediaStream Media type, resolution, frame rate var constraints = {video: true}; video: { mandatory: {... }, optional [{... }] }

10 WebRTC JS-API Constraints Controls the contents of the MediaStream Media type, resolution, frame rate var constraints = {video: true}; video: { mandatory: { chromemediasource: screen }, optional [{... }] }

11 WebRTC JS-API Acquiring video and audio navigator.getusermedia(constraints, successcallback, errorcallback);

12 WebRTC JS-API errorcallback function errorcallback(error) { console.log("error: ", error); } ;-)

13 WebRTC JS-API Acquiring video and audio navigator.getusermedia(constraints, successcallback, errorcallback);

14 WebRTC JS-API successcallback function successcallback(stream) {... } stream: MediaStream; a flux of audio- or video-related data.

15 WebRTC JS-API successcallback function successcallback(stream) { var video =... video.src = window.url.createobjecturl(stream); } (W3C File API)

16 WebRTC JS-API full sample var constraints = {video: true}; function successcallback(stream) { var video = document.queryselector("video"); video.src = window.url.createobjecturl(stream); } function errorcallback(error) { console.log("navigator.getusermedia error: ", error); } navigator.getusermedia(constraints, successcallback, errorcallback);

17 WebRTC JS-API RTCPeerConnection getusermedia + RTCPeerConnection

18 WebRTC JS-API RTCDataChannel Bidirectional communication of arbitrary data low latency API similar to WebSockets

19 RTCPeerConnection sample pc = new RTCPeerConnection(null); pc.onaddstream = gotremotestream; pc.addstream(localstream); pc.createoffer(gotoffer); function gotoffer(desc) { pc.setlocaldescription(desc); sendoffer(desc); } function gotanswer(desc) { pc.setremotedescription(desc); } function gotremotestream(e) { attachmediastream(remotevideo, e.stream); }

20 RTCPeerConnection

21 RTCPeerConnection

22 RTCPeerConnection

23 STUN

24 TURN

25 WebRTC made easy... Video chat: SimpleWebRTC easyrtc webrtc.io Peer-to-peer data: PeerJS Sharefest

26 Our target getusermedia: gives us real time stream from the cam we can draw the <video/> into a <canvas/> from the canvas we can extract pixel data getimagedata() and then elaborate.

27 Computing in the browser goals - Almost native performance - Manage complex algorithms and applications on the browser our approach - javascript as a target language (GWT) - hand written optimized pieces of code

28 GWT GWT Web Toolkit - java to javascript (optimizing) compiler - emulated jre - widget set - develop in java and execute in javascript

29 JooinK s WebRTC sample Sample Demo

30 PicShare Tecnologies WebGL for 3D graphics WebRTC for MediaSteams ImgUR for pictures store AppEngine to store indices GWT as a glue

31 FOSS gwt-nyartoolkit.googlecode.com gwt-webgl.googlecode.com The strategy: compile a java-lib with GWT into javascript

32 What s next Challanges real-time on mobile apps real world recognition: computer vision (BoofCV)

33 What s next Our approach use TypedArrays everywhere offload computation to the graphic accelerator through WebGL help the JIT compiler/optimizer with asm.js

34 References (where we got inspiration and most of the pictures) (spec)

35 That s all folks!!! Francesca Tosi [email protected] Alberto Mancini [email protected]

36 Markers - use at picshare.jooink. com