WebRTC... GWT & in-browser computation Alberto Mancini, Francesca Tosi JooinK.com
WebRTC Plug-in free realtime communication WebRTC is a free, open project that enables web browsers with Real-Time Communications (RTC) capabilities via simple Javascript APIs. source: webrtc.org
WebRTC Just another JS API or WebRTC is a new front in the long war for an open and unencumbered web Brendan Eich (Mozilla CTO and inventor of JavaScript)?
Browser Piercing a Javascript Application cannot access the filesystem and then FileAPI open f/d socket connections and then WebSockets use graphics accelerator and then WebGL span multiple threads and then WebWorkers
Browser Piercing a Javascript Application cannot acquire audio and video and then WebRTC communicate P2P and then WebRTC
WebRTC across platforms Chrome (mobile too) Firefox (mobile too) Opera Native Java and Obj-C
WebRTC JS-API Acquiring video and audio navigator.getusermedia(constraints, successcallback, errorcallback);
WebRTC JS-API Acquiring video and audio navigator.getusermedia(constraints, successcallback, errorcallback);
WebRTC JS-API Constraints Controls the contents of the MediaStream Media type, resolution, frame rate var constraints = {video: true}; video: { mandatory: {... }, optional [{... }] }
WebRTC JS-API Constraints Controls the contents of the MediaStream Media type, resolution, frame rate var constraints = {video: true}; video: { mandatory: { chromemediasource: screen }, optional [{... }] }
WebRTC JS-API Acquiring video and audio navigator.getusermedia(constraints, successcallback, errorcallback);
WebRTC JS-API errorcallback function errorcallback(error) { console.log("error: ", error); } ;-)
WebRTC JS-API Acquiring video and audio navigator.getusermedia(constraints, successcallback, errorcallback);
WebRTC JS-API successcallback function successcallback(stream) {... } stream: MediaStream; a flux of audio- or video-related data.
WebRTC JS-API successcallback function successcallback(stream) { var video =... video.src = window.url.createobjecturl(stream); } (W3C File API)
WebRTC JS-API full sample var constraints = {video: true}; function successcallback(stream) { var video = document.queryselector("video"); video.src = window.url.createobjecturl(stream); } function errorcallback(error) { console.log("navigator.getusermedia error: ", error); } navigator.getusermedia(constraints, successcallback, errorcallback);
WebRTC JS-API RTCPeerConnection getusermedia + RTCPeerConnection
WebRTC JS-API RTCDataChannel Bidirectional communication of arbitrary data low latency API similar to WebSockets
RTCPeerConnection sample pc = new RTCPeerConnection(null); pc.onaddstream = gotremotestream; pc.addstream(localstream); pc.createoffer(gotoffer); function gotoffer(desc) { pc.setlocaldescription(desc); sendoffer(desc); } function gotanswer(desc) { pc.setremotedescription(desc); } function gotremotestream(e) { attachmediastream(remotevideo, e.stream); }
RTCPeerConnection
RTCPeerConnection
RTCPeerConnection
STUN
TURN
WebRTC made easy... Video chat: SimpleWebRTC easyrtc webrtc.io Peer-to-peer data: PeerJS Sharefest
Our target getusermedia: gives us real time stream from the cam we can draw the <video/> into a <canvas/> from the canvas we can extract pixel data getimagedata() and then elaborate.
Computing in the browser goals - Almost native performance - Manage complex algorithms and applications on the browser our approach - javascript as a target language (GWT) - hand written optimized pieces of code
GWT GWT Web Toolkit - java to javascript (optimizing) compiler - emulated jre - widget set - develop in java and execute in javascript
JooinK s WebRTC sample Sample http://picshare.jooink.com Demo
PicShare Tecnologies WebGL for 3D graphics WebRTC for MediaSteams ImgUR for pictures store AppEngine to store indices GWT as a glue
FOSS gwt-nyartoolkit.googlecode.com gwt-webgl.googlecode.com The strategy: compile a java-lib with GWT into javascript
What s next Challanges real-time on mobile apps real world recognition: computer vision (BoofCV)
What s next Our approach use TypedArrays everywhere offload computation to the graphic accelerator through WebGL help the JIT compiler/optimizer with asm.js
References http://io13webrtc.appspot.com/#1 (where we got inspiration and most of the pictures) http://www.youtube.com/watch?v=p2hzzkd2a40 http://www.webrtc.org (spec)
That s all folks!!! Francesca Tosi francesca@jooink.com Alberto Mancini alberto@jooink.com
Markers - use at picshare.jooink. com