collaborative development of interactive software systems software architecture for immersipresence

Transcription

1 collaborative development of interactive software systems software architecture for immersipresence alexandre r.j. françois visiting assistant professor of computer science tufts university research assistant professor of computer science viterbi school of engineering, university of southern california

2 Software Architecture Design, analysis and implementation of software systems Improve the flexibility and comprehensibility of software systems Address modularization as a design issue (higher level than language level) Explicit system structure Technical basis for design Provable properties Blue-prints for implementation Tools for analysis Project management Separation of concerns Planning: cost estimation, resource allocation

3 Immersipresence Vision of the Integrated Media System Center Combine real world with virtual world NSF ERC in Multimedia, est Experience immersion, presence Interact naturally Collaborate through shared virtual/augmented space Build systems capable of: Handling video, sound, haptics, etc. Real-time analysis/synthesis (immersion) Low latency (interaction)

4 Requirements for Immersipresence Interoperability Efficiency On-line, real-time, low latency Scalability Combine research from different fields/teams Performance evaluation and prediction General model for distributed asynchronous concurrent processing of data streams

5 SAI Principles Time Volatile vs. persistent data Asynchronous concurrent processing Architectural style [ICSE2004] High level abstractions Hybrid model or more general model?

6 related work: concurrent computation process calculi: csp, ccs, acp, pi-calculus synchronous message passing abstract time actor model asynchronous message passing concurrent processing ptolemy heterogenous mixtures of models of computation hard real-time systems time is an external constraint (add-on)

7 related work: visual programming Khoros/Cantata Pd OpenMusic Max/MSP LabView

8 related work: alds and uml Visual Paradigm for UML architecture description languages (adls) syntax (graphical) + semantics + tools domain/style specific: rapide, c2, sadl, wright, etc. unification efforts: acme, alfa dynamic systems? unified modeling language (uml) object-oriented concepts collection of loosely related standards structure? scalability? ArchStudio4 (C2)

9 primitives & organizing principles cell processing unit (no state) repository stream repository cell shared persistent data passive pulse: persistent data stream active pulse: volatile data flow of volatile data process dependency process trigger

10 Architectural Middleware Support architectural abstractions Pulse, source, cell, etc. Direct mapping from logical specification to code! Modular Flow Scheduling Middleware (MFSM) Open source project: mfsm.sourceforge.net C++, cross-platform (GNU compiler) Base library, functional modules, documentation, tutorials

11 MFSM Diagram

12 VisualSAI

13 CAMSHIFT Tracking Tracking algorithm: CAMSHIFT Continuous Adaptive Mean SHIFT (Bradski, 1998) Mean shift: iteratively find the mode in a probability density distribution (Comaniciu & Meer, 1997) Perceptual User Interface Real-time video processing

14 camshift graph Video input +input image CAMSHIFT processing CAMSHIFT parameters +composite image Image display

15 architectural refinement: camshift Video input +input image CAMSHIFT tracker CAMSHIFT tracker node +new position Rendering +composite image Image display

16 architectural refinement: camshift Video input Video input +input image CAMSHIFT tracker CAMSHIFT tracker node +input image Step I +new position Rendering +composite image Image display Color conversion +HSV image Step II Histogram node Step III Last known bounding box Back-projection +back-projection image CAMSHIFT +new bounding box, size and orientation Rendering +composite image Constrains Image display

17 architectural refinement: camshift Video input +input image CAMSHIFT tracker CAMSHIFT tracker node Step I +new position Rendering +composite image Image display Step II Histogram node Step III Last known bounding box Video input Video input +input image +input image Color conversion Color conversion +HSV image +HSV image Back-projection +back-projection image CAMSHIFT +new bounding box, size and orientation Rendering +composite image Constrains Image display Back-projection Histogram node +back-projection image Box update Mean shift +new position Last known bounding box Moments +new bounding box, size and orientation Rendering +composite image Image display

18 loops and time pulsar rate iterative process => feedback while((eps>min)&&(n<max)){ kp=f(k); eps=g(kp-k); k=kp; n++; } update k f k +f(k)

19 composition: stevi Stereo input and pre-processing concurrent processing +input left +input right tracking pattern feedback robust adaptive Video input Video input interface left interface right Color transform right Color transform left barrier synchronization Photo courtesy of ETRI Stereo video input +HSV right +HSV left Color data left Color data right +Color data right +Color data left Barrier Stereo multi-target tracking Barrier synchronization Predict targets Update targets Tracking repository +Predicted targets visualization Tracked targets Track targets +Tracked targets +Residual image left +Residual image right feedback to developer Acquire targets modular architecture +New targets Visualization patterns facilitate integration Render targets +Composite image left +Composite image right Image display left Image display right

20 loops and time pulsar rate iterative process => feedback while((eps>min)&&(n<max)){ kp=f(k); eps=g(kp-k); k=kp; n++; } update k 3d engine event loop => concurrent streams while(1){ process_user_inputs(); process_nw_inputs(); process_physics(); process_ai(); render(); } f k +f(k)

21 musa.rt architecture Pulsar MIDI input + event Compute tonal structures Pulsar Pulsar Poll device Spiral array camera and other process parameters + command Render Control camera Process event

22 esp architecture Pulsar Pulsar Poll device + command Compute position Pulsar MIDI event buffer current position and velocity parameters for rendering and other processes Pulsar Render graphics Generate MIDI events Update velocity + events MIDI output synthesizer

23 crosswinds architecture Networking N/w receive N/w receive interface Keyboard control Rendering Gamepad control + buffer Filter buffers Keyboard input Gamepad input Decode key Decode gamepad Pulsar + filtered buffer Decode buffer Update + event interaction state Finish turn Update game state Game state Interaction state Chat data Rendering parameters Update + event interaction state Compose message Receive message Update game state Update + event interaction state Render + event Finish turn Finish turn + event Encode buffer Encode buffer + buffer N/w send N/w send Update game state + event Encode buffer + buffer N/w send N/w send interface + buffer N/w send

24 mimi architecture MIDI Input Visualization MIDI input MIDI input interface Process event Pulsar Pulsar + event Sample Current state + character Improvize Add character Oracle Render oracle + character and other process parameters Improvisation Engine Push Push Play Tracks Play + character Performance Engine mixer synthesizer Current state Generate MIDI events + events MIDI output MIDI output interface MIDI Output Render tracks

25 Summary SAI: Software Architecture for Immersipresence Design and analysis of complex software systems MFSM: Architectural middleware Patterns for synchronization Applications: Interactive music systems, Computer vision and graphics systems, Distributed Interactive games, etc. For more information:

26 SAI Properties (1) Model time explicitly in data and processing Model modularity Separation of concerns Scalability Model concurrent execution (asynchronous) Decouple throughput and latency Model distributed computing

27 SAI Properties (2) Facilitate system design Intuitive architectural style, based on data streams Unified processing model and unified data model Design patterns Facilitate system analysis Safety, liveness, etc. Facilitate distributed development Fast integration Code reusability Facilitate system maintenance, modification and evolution Change in algorithm and in function

28 Acknowledgments and Disclaimer The research presented here was funded in part by: The Integrated Media Systems Center, an NSF Engineering Research Center, Cooperative agreement No. EEC The Advanced Research and Development Activity of the U.S. Government under contract No. MDA C-0036 Any opinions, findings and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect those of the funding agencies.