Design in contesti tecnologici innovativi: Tangible Computing. Le tecnologie considerate

Transcription

1 Interaction Design Design in contesti tecnologici innovativi: Tangible Computing 1 Le tecnologie considerate Che cosa sono + Esempi RFID Mobile devices Computer Vision Touch screen e smart boards Wearable computing Tangible computing Web-based Cooperative Interaction Ambient Intelligence 2 1

2 Tangible technology key reference: The review report of NESTA FUTURE LAB (UK) 11_and12/12_01.htm (A review of tangible computing, mainly focused on its use for learning) 3 From GUI to TUI Tangible User Interface (TUI) as the evolution of GUI (Graphical User Interface) 4 2

3 GUI (Graphical User Interface) Desktop metaphor Digital spaces are manipulated with simple input devices (e.g., keyboard and mouse), which are used to control and manipulate visual representations displayed on output devices such as monitors, whiteboards or head-mounted displays. GUI COGNITIVE MISMATCH: The mapping between the manipulation of the physical input device (eg the point and click of the mouse) and the resulting digital representation on the output device (the screen) is relatively indirect and loosely coupled E.g., on the PC desktop, I have to move the mouse in 2D on a metaphorically horizontal surface ( the desktop ), yet the output appears on a vertical 2D plane (the screen) 5 Tangible User Interfaces (TUI)_1 Attempt to remove the input/output distinction of GUI: the distinction between input and output becomes increasingly blurred An attempt to overcome the difference between the ways we input and control information and the ways this information is represented 6 3

4 Tangible User Interfaces (TUI)_2 Emphasis on touch and physicality in both input and output. Control device often closely coupled to the physical representation of actual objects represented by the digital ones E.g., buildings in an urban planning application are manipulated using tangible bricks in a children s block building task two levels of direct manipulation: Manipulation of the control device Manipulation (via the control device) of the digital objects 7 Some examples_1 ABACUS - The oldest tangible computing system (much before ICT!) beads, rodes, and frame provide (physical) representations of numerical values can be manipulated to control the representations themselves (no distinction between control device and output device) Below, a today s tangible: the I/O Brush at MIT Media Lab ( 8 4

5 Some examples_2 MAGICPLANET: to help kids understand planetary motion The physical table showing orbital paths aroudn the sun is augmented with overlaid 3D animations. The children have to place cards depicting each of the planets in its correct position. As they do, they can see overlaid on the the cards a 3D animation of the planet spinning on its own axis and orbiting around the sun 9 Tangibles technologies and Ubiquitous Computing Tangible technologies are the technologies underlying TUI Yet another part of ubiquitous computing, in which technology is so embedded in the world that it disappears Once again, an implementation of Mark Weiser (1991, Xerox PARC) vision, where the digital world blends into the physical world and becomes so much part of the background of our consciousness that it disappears Ubiquitous computing is analogue to print technology in the 15 th : now text is a form of symbolic representation that is completely ubiquitous and pervasive in our physical environment 10 5

6 Tangible technology Based on physical objects directly or indirectly augumented with computing capabilities, and able to interact/communicate/control other devices Main application domain: learning (especially with children) 11 Tangible technologies characterized by the peculiar behavior of control devices and resulting digital effects Conventional, generic control devices (e.g., mouse, pen): the behavioural meaning of user control is input device independent: the effects are not related to the perceived properties of the input device Control devices based on tangible technoloy: a close correspondence in behavioural meaning between input and output: the effects of interaction are related to the perceived properties of the control device eg the effects of using a stylus to draw a line directly on a tablet or touchscreen are related to the (apparent) physical nature of the stylus, e.g. its size 12 6

7 Classification dimensions Tangible technologies may differ for several interdependent aspects: The metaphor The behavioral mapping The embodiment 13 Classification dimensions: The metaphor Two aspects: 1) The degree of analogy between the physical (visual, tactile, auditory) appearance of the control device and its corresponding apparance in the real world Two extremes Control device resembles its digital counterpart (e.g. physical models of the building are used to move buildings around a digital map on the display) Abstract symbolic objects (with limited or no analogy at all) 2) The extent to which the user s action with the control device are analogous to real world effects of similar actions 14 7

8 Classification dimensions: The behavioural mapping The behavioral mapping control device digital output : two extremes One-to-many (the same input device determines multiple effects) 1-to-1 (one effect for each input device) 15 Classification dimensions: Embodiment Embodiment: two related characterizations The degree of cognitive distance between input/ control device and output Example of high embodiment: Interactive skin - artificial skin that can react to being touched or stroked by changing lights on its surface, vibrating, etc. The degree of attention payed by users to the object being manipulated rather than the toolbeingusedtomanipulate the object Total embodiment when input and out devices are the same (see interactive skin above), 16 8

9 Alcune tipologie di tangible tecnologies digitally augmented paper and books physical objects as icons (phicons) digital manipulatives sensors/probes 17 Digitally augmented books_1 Examples: Interactive children s storybooks The system developed by Back et al (2001), which has a soundtrack triggered by moving one s hands over the book. The book has RFID tags embedded within it which sense which page is open, and additional sensors (capacitive field sensors) which measure an individual s position in relation to the book and adjust the sound accordingly LeapPad - a digitally augmented book with a pen that enables words and letters to be read out when a child touches the surface of each page. The book can also be placed in writing mode and children can write at their own pace

10 Digitally augmented book and immersive virtual reality Magic Book ( A MagicBook looks like a normal storybook with colorful pages and simple text. When readers turn the pages, the pictures pop off the page and come to life as three-dimensional animated virtual scenes, which they see images overlaid onto the page of the book (fig 1) When they wear a lightweight head mounted display (HMD), by touching a switch on the HMD, readers can fly into the virtual scene and freely explore the immersive 3D virtual reality environment (fig 2 and 3) (1) (2) (3) 19 Digitally augmented paper_1 Examples: Paper++ project ( Digital augmentation of paper with multimedia effects. Technology: real paper together with special transparent and conductive inks that could be detected with a special sensing device (a wand ). The whole surface of the paper is active, without involving complex technology embedded in the paper itself

11 Digitally augmented paper_2 KidStory project ( Using tangibles to allow kids create and explore interactive stories. Children s physical drawings on paper are RFID tagged, digitally reproduced, and inserted in the interactive stories thay are building. Kids can use their paper drawing, via a RFID reader, to physically navigate between episodes in their virtual story displayed on a large screen, using the KidPad storytelling software. (Kids can also use tagged toys, representing story character, for the same purpose) 21 Digitally augmented paper and ambient intelligence Examples: SHAPE project (see also slides in file design con tecnologie innovative-ambient intelligence ) a mixed reality experience to aid children (and adults) to discover, reason and reflect about historical places and events A (digitally augmented) paper-based history hunt around Nottingham Castle + a mixed reality experience in a Story tent During history hunt, kids search for clues about historical events which took place in the castle and make drawings or rubbings on paper at a variety of locations. The paper is electronically tagged and used to interact with a virtual reality projection on a display called the Storytent inside the castle museum. Inside the Storytent, a Radio Frequency ID (RFID) tag reader is positioned embedded in a turntable. When placed on the turntable, each paper clue reveals an historic 3D environment of the castle from the location at which the clue was found, thus explicitly linking their physical exploration of the castle (using the paper clues) with a virtual reality simulation of the castle as it was in medieval times. Child and adult in the Storytent with close-up of turntable (inset)

12 PHICONS: PHIsical objects as digital icons physical objects other than paper (such as toys, blocks etc.) used to trigger digital effects Examples: CROMARIUM A mixed reality activity space that uses tangibles to help children aged 5-7 years experiment and learn about colour mixing, using a variety of physical and digital tools. It allows children to combine colours using physical blocks, with different colours on each face. By placing two blocks together children could elicit the combined colour and digital animations on an adjacent screen. 23 DIGITAL MANIPULATIVES physical devices which have computational properties embedded in them In some case, computationally enhanced versions of toys enabling dynamics and systems to be explored 24 12

13 DIGITAL MANIPULATIVES: EXAMPLE_1 (Commercially available!) LEGO Mindstorm Born at MIT (1993) robotics construction kit by which children can create Logo (*) programs to control LEGO assemblies. The kit consists of motors and sensors and children use Logo programs to control the items they build, and Programmable Bricks (P-Bricks) that contain output ports for controlling motors and lights, and input ports for receiving information from sensors (e.g light, touch, temperature). Programs are written in Logo and downloaded to the P-Brick or to the robots, which then contains the program and are autonomous. (*) LOGO: a simple programming language for kids that gives them commands and control structures for operating a turtle, representing the on-the-screen cursor or a physical object, e.g., a robot 25 (A Funny example of LEGO Mindstorm use) One girl built a bird feeder with a sensor attached so that when the bird landed to feed, the sensor triggered a photo of the bird to be taken. The girl could then see all the birds that had visited the feeder while she was away

14 DIGITAL MANIPULATIVES: EXAMPLE_2 BitBalls (MIT Media Lab) A transparent rubbery ball that can be thrown up into the air and can store acceleration data which can then be viewed graphically on a screen used in scientific investigations with children in learning kinematics. 27 DIGITAL MANIPULATIVES: EXAMPLE 3 Ambient Wood project A Aim: to create a novel learning experience to support scientific enquiry about the biological processes taking place in a wood Technology A WIFI infrastructure + web server Wireless speakers PDA Pingers Probe tool Periscope Tangible interfaces Interactive board 28 14

15 The PDA AMBIENT WOOD The probe tool The Periscope The indoor board 29 DIGITAL MANIPULATIVES: Ambient Wood project (cont.) WIFI infrastructure created inside the wood, it can monitor the children's positions, tracking any data the children collect, and triggering location-based information Wireless speakers hidden in sections of the woods to provide a range of realistic sounds of animals in the habitat and abstract sounds that represented various plant processes. Periscope a standalone viewing tool to provide children with access to prerecorded videos about the habitat Pingers Short-range FM transmitters hidden in predetermined spot; can detect the presence of kids in a range of 3 meters, and react to their presence (interacting with the PDA) 30 15

16 DIGITAL MANIPULATIVES: Ambient Wood project (cont.) PDA and Probe tool Children can explore a woodland area in pair, each pair using two mobile devices: a PDA that presents digital information triggered by the immediate surrounding environment; e.g., it can show an image of a plant or animal together with a voice-over about an aspect of its habitat, whenever the children walk close to a pinger hidden in a predetermined spot, in order to draw their attention to a section of the woods at pertinent times, and to induce them to think and reflect upon it. Probe tool allows data to be captured by the children according to their exploration (humidity or light measures). Readings of the probes appear on the PDA display as dynamic visualizations. The probe tool also transmits and stores all the readings and the location at which they were collected in the woodland (using GPS) to enable the children to reflect about them later in the classroom 31 Ambient Wood project (cont.) Tangible interface: RFID tagged tokens made from the images they had received within the wood, in order to reflect on their findings RFID tagged objects pre-prepared by the teachers (e.g., a can of spiders ). Outdoor, in the woodland, children can use the tagged objects in conjunction with the periscope. Using for example, a can of spiders to interact with the periscope, the can view on the periscope screen the potential effects for introducing a new organism (a new specie of spiders) in the habitat that they explored in the environment, and to reflect in this problem. Indoor (e.g., in the school lab), an interactive tangible board allows the children to reconstruct (and receive feedback about) what they had seen, collected, or heard. Multimedia content is provided on the display whenever certain combinations of tagged tokens are detected by the board, e.g., concerning the habitat related to token images 32 16

17 DIGITAL PROBES Physical tools which act as sensors or probes of the environment (eg light, colour, moisture). Some applications in TANGIBLE STORY TELLING Example: SHAPE Project (see previous slides) Use of flashlights or torches to interact with the Storytent display, ie., to manipulate the objects in the 3D virtual world displayed on the tent. trigger audio clips on the wall of the underground caves in Nottingham Castle, enabling users to hear about the history of the caves as it happened at the very location where they shone the flashlight Interact with digital sand (Sandpit): As users collaboratively shone their flashlights on the sand it burned a virtual hole through the surface to reveal images beneath, telling the history of Nottingham Castle 33 17