TimeOut: Designing for Emotional and Cognitive Control. Master Thesis in Information Science

Transcription

1 TimeOut: Designing for Emotional and Cognitive Control Master Thesis in Information Science Author: Eivind Flobak Advisor: Frode Guribye May 2017

2

3 Acknowledgments I would like to express my gratitude to my advisor, Frode Guribye, for inviting me to join the INTROMAT research project for the completion of my master s degree. Thank you for your continued support, encouragement, and patience. I would also like to thank all collaborators and participants in the Intromat research project. Some of you in particular: Astri J. Lundervold, for your interest and insightful comments throughout my thesis research; Daniel André Jensen, for your continued support and cooperation throughout the design process presented in this thesis. Thank you Joakim and Daniel for tireless efforts to proofread this thesis. Thank you Hanne, Ole and Simen for the good times at 635. Thank you, Julia, for all your loving support. Lastly, I want to thank all participants of this study for their time and effort.

4

5 Contents List of Figures List of Tables List of Algorithms v vi vii 1 Introduction Research Question Research Aims Structure of Thesis Background Human-Computer Interaction Affective Computing Affective Interaction Design Principles for Affective Interaction Design Challenges for Affective Interaction Applications of Affective Computing Pervasive Affective Sensing Emotional and Cognitive Control Mental Stress ADHD A Short Overview Diagnostics Epidemiology Treatment Electrodermal Activity Characteristics of EDA Signals Related Work Designs for Supporting People with ADHD Design Principles Designing for ADHD Designs for Stress Monitoring Summary of Related Work Chapter Summary Methodology Design as a Science - An Introduction Research Through Design Problem Identification Evaluation of the Design Process iii

6 iv Contents Why RtD? Involving users: User-Centered Design Persona Scenario Prototyping Evaluation Field Trials Technology Deployment in the Wild Chapter Summary Development of Prototype First Iteration Design Workshops Conceptual Design Evaluation Second Iteration Establishing Requirements Further Design and Prototyping Feasibility Study of Biosensor Wristband Third Iteration Refining Requirements Designing for Interpretive Flexibility Intervening to Alleviate Stress Algorithm for Pervasive Affective Sensing Prototype Cognitive Walkthrough Evaluation Fourth Iteration Incorporating Feedback from Previous Iterations Designing Tasks for Reducing Emotional and Cognitive Activation Improving Algorithm for Pervasive Affective Sensing Implementing a Fully Functional Prototype Evaluation Summary of Development Chapter Evaluation Method Presentation of Results Analysis of User-Generated Data Accuracy of Affective Sensing Possible Causes of False Positives Analysis of User Experience Synchronous Visualizations of EDA Positive Interventions Interruptions from using TimeOut History of Previous Interventions Technical Issues Limitations Summary of Evaluation Chapter Discussion 63 iv

7 Contents v 6.1 Accuracy of Affective Sensing Algorithm Design Implications Potential for Improvements of TimeOut Synchronous Visualizations of EDA: The Affective Loop Design Implications Potential for Improvement of TimeOut Task-Based Intervention Retrospective of Past Interventions Designing Interventions for Supporting Emotional and Cognitive Control TimeOut s Effect on Emotional and Cognitive Control Summary of Design Implications for Supporting Emotional and Cognitive Control Summary of Discussion Chapter Conclusion and Future Work Future Work Appendix A Consent Form 81 Appendix B Trier Social Stress Test 82 Appendix C Early Design Workshop 83 Appendix D Pervasive Health 2018 Extended Abstract 85 v

8 List of Figures 2.1 Two sessions of EDA measurement presented by TimeOut showing tonic and phasic changes in EDA signal Persona illustrating the archetypal user of TimeOut Empatica E4 wristband device BPMN illustrating the components and flow of TimeOut Low-fidelity prototype of TimeOut Recorded EDA levels of participant #1, red line marks test start Suggestive visualizations of the user s EDA data A cue for intervention that is dismissible by avslå Second iteration of lo-fi prototype Assessment dialog for recording subjective experience prior to and during intervention EDA data and description prior to intervention, followed by EDA data while completing the task-based intervention Intervention design for supporting emotional and cognitive control Description of TimeOut for first-time users The different nuances of colors green, yellow and red visualize the user s current EDA level. (See subsection for further reference) A dialog presented to the user represents a cue for intervention One of the participant #1 s constructive interventions Participant #2 being intervened at a moment of frustration Participant #3 s decreased skin conductance while performing breathing exercise at day two of participation Vague presentation of skin conductance levels after stopping TimeOut mistakenly intervening a participant who experienced a moment of amusement vi

9 List of Tables 2.1 Overview of related studies Comparison of low-fidelity and high-fidelity prototyping (Rogers, Sharp, & Preece, 2015, p. 395) Overview of design evaluations presented in this thesis Quantitative results of field trial Categorization of each participants false positives vii

10 List of Algorithms 1 Basic algorithm for establishing EDA baseline Function to compute percentage change in new EDA in regards to baseline EDA Movement and temperature-filtering affective sensing algorithm Baseline by moving average viii

11 Chapter 1 Introduction At certain times we all experience difficulties with maintaining our attention and regulating impulsive thoughts and behaviors. The extent to which we experience such difficulties may, however, have a considerable impact on our lives. An excessive presence of such difficulties may lead to a diagnosis of attention deficit hyperactive disorder (ADHD) and is associated with negative consequences in the form of social difficulties, poor work performance, and low self-esteem (American Psychiatric Association, 2013; Biederman & Faraone, 2006). Currently, people with ADHD primarily receive pharmacotherapy (Dodson, 2005). Although these treatments have been shown to be effective in reducing difficulties related to symptoms of ADHD, undesired side-effects of stimulant medications such as methylphenidate have led to a need for non-pharmacological treatments (NCCMH, 2009). There is also a documented desire from patients for such alternatives (Solberg, Haavik, & Halmoy, 2015). Information and communication technologies have opened up for novel approaches to mediate treatment of mental illness with interactive technology (Donker et al., 2013; Mohr, Burns, Schueller, Clarke, & Klinkman, 2013). Recent advances in wearable technology offer possibilities for continuous monitoring of the physiological and contextual data of its users. Although early adopters and researchers of wearable technology are optimistic for the utility of pervasive monitoring for health care institutions, these novel technologies are yet to be implemented in clinical practice. The project INTROMAT (INTROducing Mental Health through Adaptive Technology) received funding as an IKTPLUSS Lighthouse project from The Norwegian Research Council in 2016, and is established as a five-year research project to study, design, test, and implement adaptive technology in mental health care (INTROMAT, 2017). The study presented here is part of INTROMAT and the case study focusing on developing digital health services for cognitive training in ADHD. This thesis explores some of the challenges related to providing people with ADHD with assistive technology to support their emotional and 1

12 2 Chapter 1. Introduction cognitive control. In the effort of designing a pervasive affective sensing system for people with ADHD, TimeOut, was developed to prompt its users with a Goal Management Training-inspired intervention by continuously monitoring and analyzing electrodermal activity. Recently, designing technologies for supporting people with ADHD has received attention from the human-computer interaction (HCI) community (Sonne, Marshall, Obel, Thomsen, & Grønbæk, 2016). This study is placed within the research field of HCI, and draws on inspiration from design exemplars of affective computing and assistive technology. HCI research has been described as the study of constructive, empirical and conceptual problems related to human factors in computing (Oulasvirta & Hornbæk, 2016). 1.1 Research Question The overarching research question for this thesis is: How can we design interventions supporting emotional and cognitive control in adults with ADHD? To answer this question, a literature review was conducted to survey previous work related to the research question. Through a research through design-process, a prototype, TimeOut, was constructed with the aim of supporting adults with ADHD to control their emotional and cognitive activation. An empirical study of six participant s use of TimeOut was conducted to study how interventions cued by TimeOut s monitoring of electrodermal activity was experienced in real-world contexts. 1.2 Research Aims The purpose of this study is twofold: Constructive: The development of design-rationales for, and iterative development of a pervasive affective sensing system for adults with ADHD. This includes selection of methods for constructing such a system, as well as establishing user requirements. The main contribution of this research is the produced knowledge about how to design for improved emotional and cognitive control in adults with ADHD. Empirical: Investigating and elaborating characteristics of use and experience from using the constructed prototype, TimeOut. By conducting a field trial in a real-world context, phenomena of human factors in using pervasive affective sensing systems for emotional and cognitive control are elicited and described. Furthermore, this thesis seeks to communicate the findings of this study in a way that is extensible to future research in the field of HCI. 2

13 Chapter 1. Introduction Structure of Thesis The following list presents the structure and outline of this thesis. Chapter 1 introduces the problem domain, the research question and aims of this study. Chapter 2 presents literature relevant to this study before discussing related work. Chapter 3 describes methods used to conduct the presented research. Chapter 4 details the design and development of TimeOut. Chapter 5 describes how TimeOut was evaluated and the results of said evaluation. Chapter 6 discusses the evaluation s results in respect to the research question and the research contribution of this thesis. Chapter 7 concludes this thesis with a summary of what was found and presents a proposition for future work to extend the presented research. 3

14 Chapter 2 Background This chapter presents the research fields within which this research is situated. Then, a definition of emotional and cognitive control is given, followed by an overview of ADHD and psychological interventions. Lastly, a literature search for related work is detailed and discussed. 2.1 Human-Computer Interaction Human-Computer Interaction (HCI) is a field of research situated at the intersection of computer science, behavioral sciences, and design. HCI as a field of study gained prominence in the 1980s when personal computing became a widespread phenomenon (Carroll, 2013). In its formative years in the 70s, HCI research was primarily concerned with issues relating to usability. With this effort, researchers and designers aimed to create efficient, intuitive and safe user interfaces for computers. Practitioners found that an early focus on users and tasks, followed by empirical measurement of the user interface s usability qualities, while repeated in an iterative process, would yield interfaces that met usability standards (Gould & Lewis, 1985). Further, Nielsen s techniques for heuristic evaluation of usability goals have been influential throughout the course of HCI research (J. Nielsen, 1993; J. Nielsen & Molich, 1990). The tradition of usability engineering, where the goal is to improve systems concerning efficiency and error-proneness, were met by a somewhat contrasting movement of user experience (UX). Designing for user experience means to involve the users emotional and sensory perception of the user interface (Buchenau & Suri, 2000). This goes beyond the concept of look and feel as defined by Houde and Hill (1997), in its emphasis on the concrete sensory, UX also includes aesthetic qualities, affections or experiential aspects 4

15 Chapter 2. Background 5 of using technology (Wright, Wallace, & McCarthy, 2008). HCI has transformed its approach in regards to methodologies throughout the years, defined metaphorically by Bødker (2006) as waves of HCI research approaches. In her description of HCI research s second wave, interaction in work settings and Computer Supported Collaborate Work (CSCW) garnered the attention of researchers. Bødker (2006) states that Rigid guidelines, formal methods, and systematic testing were mostly abandoned for proactive methods such as a variety of participatory design workshops, prototyping and contextual inquiries (p. 1). The focus in the third wave of HCI has been described to be of non-purposeful, non-work, non-rational by Bødker (2006), who argues that conceptually and theoretically, the third wave HCI focuses on the cultural level, e.g. through aesthetics, expansion of the cognitive to the emotional (Bødker, 2006, p. 1-2). Methodologies of the third wave have been found to move away from comprehensive user studies, such as ethnography, ethnomethodology and other prolonged studies of people. New methods, such as in-the-wild studies (Brown, Reeves, & Sherwood, 2011) and cultural probes for eliciting requirements of users in an art-focused manner (Gaver, Dunne, & Pacenti, 1999), provide researchers with methods that are less rigorous than methods that were prominent in the CSCW research community. HCI as Problem-Solving for Understanding Human Factors in Computing While HCI does share some of its problem domains and techniques with computer science, and other engineering sciences, HCI serves a distinct goal: to solve important problems in human use of computers (Oulasvirta & Hornbæk, 2016, p. 4957). In an essay on the meta-scientific account of HCI research, Oulasvirta and Hornbæk disambiguate HCI as problem-solving research of three paradigms: empirical, conceptual, and constructive (2016). In their typology, empirical research is defined as creating or elaborating descriptions of real-world phenomena related to human use of computing (Oulasvirta & Hornbæk, 2016, p. 4958). That is, exploring some phenomena novel to HCI research, discovering some fundamental factors of this phenomena, and in turn measuring and quantifying the effects on something of interest (e.g. the usability of utilizing this phenomenon in interaction) (Oulasvirta & Hornbæk, 2016). Conceptual research is defined as work that tackles the explanation of previously unconnected phenomena occurring in interaction (Oulasvirta & Hornbæk, 2016, p. 4958). Oulasvirta and Hornbæk argue that exploration of such problems are due to its limited attention and reach in HCI research (Oulasvirta & Hornbæk, 2016). It is worth noting that there are other venues, journals, and books where conceptual research receives more 5

16 6 Chapter 2. Background attention. Lastly, constructive research aims to produc[e] understanding about the construction of an interactive artifact for some purpose in human use of computing (Oulasvirta & Hornbæk, 2016, p. 4960). Constructive research in HCI communicates how a prototype was made. For example, a detailed description of the design process including design-rationale, methods such as participatory design, and other qualities of the design process. A common contribution of constructive research are design principles which contributes to the HCI communities knowledge of how to design for a given context or domain. The exposition of these paradigms leads us to a general definition of a research problem in the field of HCI: a stated lack of understanding about some phenomenon in human use of computing, or stated inability to construct interactive technology to address that phenomenon for desired ends (Oulasvirta & Hornbæk, 2016, p. 4960). These paradigms of problem-solving in HCI research are found to exist in tandem with each other, usually by combining two paradigms to explore some novel or established pattern of human use of computing. For example constructive-empirical studies that produce some novel interaction modality and contribute to the understanding of related phenomena. This thesis contribution is the construction (see chapter 4) of TimeOut as an assistive technology for adults with ADHD, and the empirical study of users experience of its design (see chapter 5). 2.2 Affective Computing Affective computing research encompasses a broad range of existing knowledge from fields such as psychology, cognitive, physiology and computer sciences (Tao & Tan, 2005). According to Tao and Tan (2005) the goal of affective computing is to assign computers the human-like capabilities of observation, interpretation and generation of affect features (p. 981). Originally coined by Picard (1997), she proposed that we give computers the ability to recognize, express, and in some cases, have emotions (p. 1). As a general definition, affect is the experience of emotion (Hogg, Abrams, & Martin, 2007). To understand human emotions in a veridical manner by the use of computers, a thorough understanding the phenomenology of emotion is required. Currently, there is no consensus in psychology on a shared definition of such phenomena, as more than a hundred definitions exist and new definitions are still emerging. Most scholars agree, however, that emotions are separable by two dimensions: valence (positive versus negative experience) and arousal (activated versus deactivated) (Hogg et al., 2007; Kleinginna & Kleinginna, 6

17 Chapter 2. Background ). (See section 2.3 for more on arousal and activation). Although Picard claims not to intend to propose any explanation of emotions as a phenomenon, Picard modeled the affective computing field to explore affect recognition, interpretation, and simulation of emotions (Picard, 1997; Picard et al., 2004). Affective computing research aim to fill the gaps of knowledge on emotion by utilizing an engineering approach through the use of computation (Picard, 2010). Picard calls for a joint effort by engineers and psychologists to create new ways for people to measure, share, analyze, and learn from objective emotional responses in situations that truly matter to people (2010, p. 250) Affective Interaction The field of affective computing have traditionally held that emotion is an internal, individual, and delineable phenomenon, which operates in concert with and in the context of traditional cognitive behavior (Boehner, DePaula, Dourish, & Sengers, 2005, p. 59). It has been agreed that affective computing takes a computer perspective [to emotional phenomena] (Hassenzahl & Tractinsky, 2006, p. 93). In other words, emotion is formalizable and as such should be treated as information. Critical of such an informational model of emotion in affective computing, Boehner et al. (2005) introduced an alternative model of emotion as interaction. Boehner et al. held the view that affective computing was headed in the wrong direction with its reductionist philosophy. In their opinion, the interactional approach should mediate the user s affective states in a way that encourages awareness of and reflection on emotions. Not by discrete, quantifiable measures, but as constructs of social and cultural experiences. Höök argues that bodily experiences are integral to how we interpret and make sense of the world (2009, p. 3585). The interactional approach to affective computing is driven by a socially situated perspective of emotion, although it is not limited to this point of view. The everyday, physical, bodily experiences of emotion processes should also be considered when designing for affective interaction (Höök, 2009). In an effort to include such corporeal, embodied experiences in affective computing, Sundström, Ståhl, and Höök introduced the concept of affective loop experiences. In their view, the communication of emotion is not a matter of transferring simple information to a user; it is a matter of physically and intellectually experiencing the whole range of emotions that make up a conversation (Höök, Ståhl, Sundström, & Laaksolaahti, 2008, p. 648). More specifically, affective loop experiences refer to human experiences of affective computing designs where one cannot distinguish the intellectual from sensual experience. It is a constant flow of affective communication between the user and the system, in which the 7

18 8 Chapter 2. Background user should be empowered by the loop experiences to learn from their affects and emotions. Further developments on the perspective of embodiment in affective interaction by Höök et al. argue somaesthetic design as an approach to deepening the experience of [users ] own felt bodily sensations and movements rather than external sensory interactions (Höök et al., 2015, p. 26). This perspective distances it from past explorations of affective interaction in that the designed feedback modalities should be minimally invasive to the user, empowering users with a truly noninvasive and holistic bodily experience Design Principles for Affective Interaction A set of design principles for the interactional approach to affective computing were given by Boehner et al. (2005), some of which are presented below: The interactional approach relies on and supports interpretive flexibility. The interactional approach avoids trying to formalize the unformalizable. The interactional approach focuses on people using systems to experience and understand emotions (p ). Interpretive flexibility refers to leaving the definition and interpretation of emotion to the users of affective computing systems. By leaving the definitions to the user, their understanding of their affective state appears in a situated way over the course of interaction. A continuation of interpretive flexibility exists in Sundström et al. (2007) s affective loops, as discussed above. Höök (2009) identifies affective loops as a distinct use quality in interactional designs of affective computing systems. Related to interpretive flexibility, as described by Boehner et al. (2005), Höök (2009) identifies designs that support affective loop experiences have all left space, or inscribable surfaces, open for users to fill with content (p. 3592). Consequentially, affective loops move the definition of emotion and affect to the user, by adding an interpretive layer to the affective data and as such supports interpretive flexibility. Affective loops can be experienced by immediate feedback, such as biofeedback applications, or in retrospect after prolonged monitoring or self-reporting of affective data. By avoiding trying to formalize the unformalizable, Boehner et al. (2005) seek to avoid the action of forcing formalized expressions of emotions upon users. As users cannot always articulate emotions and affects in a straightforward way, it can be invasive for the user if the system offers a seemingly veridical claim of the user s emotional state. 8

19 Chapter 2. Background Design Challenges for Affective Interaction There are some key design challenges when designing for affect as an interaction. Boehner et al. (2005) found that affect as interaction is not yet as well-understood as affect as information (p. 66). That is, as we are used to interpreting systems as bearers of discrete, quantifiable information, understanding affective interaction often involve vague data which the user must interpret critically. Meaningful experiences are found in the affective loop that the system incites. There is also a challenge in the necessity to develop substantially new evaluation strategies, since exisiting evaluation strategies are based on an informational model (Boehner et al., 2005, p ). Therefore, instead of merely quantifying the quality of information given by such affective interactions, we should evaluate the user experience of such systems. If the design facilitates the user s awareness of their emotional states, the design is a success Applications of Affective Computing Pervasive Affective Sensing Pervasive affective sensing systems is an application of affective computing concepts. Kanjo, Al-Husain, and Chamberlain (2015) describe these systems as unobtrusive in that they gather affective data from users of smartphones, wearable sensors, digital cameras, or other sensors noninvasively. These data can be continuously analyzed for affect recognition and visualization. The application of such systems includes, but are not limited to, promotion of health and well-being for individual and collective users (Kanjo et al., 2015). In this section, a short overview of the sensing, analysis and application qualities of pervasive affective sensing applications are given. To sense a user s emotional state some input data is necessary for the pervasive affective sensing system. Such data can take the form of self-reported accounts and measurements of affect, physiological monitoring (e.g. by a wearable wristband), digital cameras for recording facial expressions, or monitoring speech by recording audio. We can differentiate these inputs by their obtrusiveness. Self-reporting requires an action by the user, besides relying on the user s subjective account of their emotion or affect. Using physiological monitoring or digital cameras are less obtrusive as the sensors record data synchronously. Currently, sensors have a lightweight and ubiquitous design (Kanjo et al., 2015). Physiological data available through wearable sensors include electrodermal activity (EDA, see section 2.5), electrocardiogram in which heart rate variability and blood pressure can be derived, and electroencephalography that measures brain activity. To make sense of the input, some data analysis is required. In general, this involves recog- 9

20 10 Chapter 2. Background nition of a distinct emotion and visualization of input for a user interface. Kanjo et al. describe sensing emotional states as [following] a statistical, probabilistic, and machine learning approach, where a huge amount of data is collected for training and testing a classifier (Kanjo et al., 2015, p. 1202). Often, these sensor data are analyzed in tandem with other contextual information such as self-reporting or contextual sensor data such as movement through an accelerometer sensor (Kanjo et al., 2015). 2.3 Emotional and Cognitive Control An aim of this study was partly to create technology that can support adults with ADHD s emotional and cognitive control. This section gives a definition of how this concept is understood in the context of this study, drawing on the concepts of inhibitory control and executive functioning. Following that, an account of mental stress is given, which is a phenomenon that may influence one s emotional and cognitive control. Inhibitory control is thought to be a cognitive process that allow us to stop a planned or ongoing thought and action (Williams, Ponesse, Schachar, Logan, & Tannock, 1999, p. 205). This includes the ability to restrict impulses and behavioral responses to stimuli. Inhibitory control is vital to be able to immediately adapt a more appropriate behavior to cope with the stimuli (Williams et al., 1999). For example, it could be said that someone who reacts with anger and temper tantrums when encountering problems in using a computer program, instead of readjusting the problem-solving approach, displays poor inhibitory control. Executive functioning is defined as the mental processes that allow us to concentrate and maintain our attention (Diamond, 2014). More broadly, executive functioning helps us select and successfully monitor our own behavior in regards to the achievement of chosen goals (Malenka, Nestler, & Hyman, 2009). For this thesis, emotional and cognitive control is defined as a person s self-regulation of emotion and cognition, which is dependent on inhibitory control and other aspects of executive functioning Mental Stress The term stress is a word that is familiar in our everyday discourse. In itself, stress is quite vague so for this section I will give a short overview of how mental stress is defined in psychology. In psychology stress is conceived of as an ongoing interaction between an organism and 10

21 Chapter 2. Background 11 its environment (Passer et al., 2008, p. 519). A more formal definition of this concept of stress is given: [Stress is] a pattern of cognitive appraisals, physiological responses, and behavioral tendencies that occurs in response to a perceived imbalance between situational demands and the resources needed to cope with them (Passer et al., 2008, p. 520). When using the term stimulus or situational demands in regards to stress we are usually concerned with stressors. These are events external to us that pose some challenge for us to cope with. A stressor can be of minor or major significance. Minor stressors can be everyday hassles e.g. the daily commute and difficult co-workers, whereas major stressors can involve sudden changes in life situation, loss of a loved one or career failure. These stressors are immediately subject to some cognitive appraisal as they are perceived by the organism, or rather, person. Stressors are evaluated in regards to possible consequences: positive, neutral or threatening to ones well-being. Further, ones resources available to cope with the stressor are evaluated. Does the person have what it takes in terms of knowledge, cognitive abilities, technical skills or social network to overcome the potential threat? (Passer et al., 2008) Physiological responses, such as racing heart beat or sweaty hands, are experienced as the stressors are appraised. It is interesting to note that the physiological responses and cognitive appraisals will affect one another. For example, as you prepares to enter into an oral examination or job interview you might find yourself trembling and your heart pounding as you sit down [...], you may appraise the situation as even more threatening than you did initially (Passer et al., 2008, p. 521). Studies have shown that people with ADHD have an increased vulnerability to everyday stressors (Lackschewitz, Hüther, & Kröner-Herwig, 2008). Furthermore, it has been established that reduced executive functioning and reduced inhibitory control characterizes people with ADHD (Barkley, 1997; Marije Boonstra, Oosterlaan, Sergeant, & Buitelaar, 2005). 2.4 ADHD A Short Overview Attention deficit hyperactive disorder (ADHD) is a mental disorder described by American Psychiatric Association (2013) in the American Diagnostic and Statistical Manual of Mental Disorders (DSM-5) and as Hyperkinetic disorder by the International Statistical Classification of Diseases and Related Health Problems (ICD-10) by the World Health Organization (1993). ADHD is characterized by its three core features: problems of maintaining attention, excessive activity and impulsivity (Helsedirektoratet, 2014). In this section I will give 11

22 12 Chapter 2. Background a short overview of the diagnostics, epidemiology, causes and treatment of ADHD Diagnostics In the medical classification list DSM-V, issued by the American Psychiatric Association (2013), ADHD is described in the following paragraph: a neurodevelopmental disorder defined by impairing levels of inattention, disorganization, and/or hyperactivity-impulsivity. Inattention and disorganization entail inability to stay on task, seeming not to listen, and losing materials, at levels that are inconsistent with age or developmental level. Hyperactivityimpulsivity entails overactivity, fidgeting, inability to stay seated, intruding into other people s activities, and inability to wait symptoms that are excessive for age or developmental level (American Psychiatric Association, 2013, p. 32). To provide a basic understanding of what characterizes ADHD this section gives a short introduction to each of ADHD s core features. Inattention: Described as difficulty with sustaining attention on a single task over a period of time, and difficulties with organizing tasks. It is common, however, for people with ADHD to show high levels of focus on tasks they are highly interested and/or skilled in (Helsedirektoratet, 2014). Hyperactivity: Characterized by fidgeting or tapping with hands and feet while being seated in a chair. Being on the go and seemingly driven by a motor are common descriptions of people suffering from hyperactivity (Helsedirektoratet, 2014). Impulsivity: Actions that are performed without any premeditation or regard for their consequences; talking excessively, blurting out the answer before the other party has completed the question, inability to wait for your turn (e.g. in conversations, or while waiting in line) (American Psychiatric Association, 2013; Helsedirektoratet, 2014) Epidemiology The prevalence of ADHD in the population is dependent on whether we are measuring for Hyperkinetic disorder by ICD-10 standards or ADHD by DSM-5 criteria. The WHO estimates 1-3 % of all children and youth would fulfill the criteria for a diagnosis of Hyperkinetic disorder, while APA estimates that 5 % would fulfill the criteria for ADHD as specified in the DSM-5 (American Psychiatric Association, 2013; World Health Organization, 1993). 12

23 Chapter 2. Background 13 A consensus study found that most people with ADHD have prevailing symptoms of their mental disorder in adult life. Kooij, Bejerot, Blackwell, et al. (2010) found that about two thirds of adults diagnosed with ADHD in childhood struggles with ADHD in adulthood. Adults with ADHD struggle with task management, keeping deadlines, are easy to distract and often bored. Their symptoms of hyperactivity are experienced as restlessness, inner discomfort, tension and being overly talkative. Impulsivity leads the adults to hastiness and sometimes unfounded decisions (Helsedirektoratet, 2014). ADHD in adulthood has been shown to be associated with significant risk of drug abuse (Groenman, Oosterlaan, Rommelse, et al., 2013; Harstad & Levy, 2014) Treatment Currently, treatment of ADHD is primarily based on pharmacotherapy (Dodson, 2005). While these treatments have been shown to be effective, undesired side-effects of stimulant medications such as methylphenidate have led to a need for non-pharmacological treatments (NCCMH, 2009). There is also a documented desire from patients for such alternatives (Solberg et al., 2015). Non-pharmacological psychological intervention treatments of ADHD have been developed, such as cognitive behavior therapy for ADHD, psychoeducation and supportive coaching, although further research is needed to provide evidence of its effectiveness on treating people with ADHD in clinical practice (Kooij et al., 2010). Psychological intervention have, surprisingly, not been explicitly defined in literature (Hodges et al., 2011). Oxford English Dictionary (2017) gives a general definition of intervention: The action of intervening, stepping in, or interfering in any affair, so as to affect its course or issue. For this thesis, intervention is understood as: An activity, cognitive or physical, which a user is either asked to perform or self-initiates for the purpose of modifying an undesired internal state or ongoing behavior. Goal Management Training Goal Management Training (GMT) is a non-pharmacological psychological intervention treatment found to be effective in treating patients with reduced inhibitory control and executive functions (Levine et al., 2011). A pilot study of GMT as an intervention for adults with ADHD have shown significant improvement of inhibitory control for patients that received GMT in combination with psychoeducation and counseling compared to those who received only psychoeducation 13

24 14 Chapter 2. Background and counseling (In de Braek, Dijkstra, Ponds, & Jolles, 2012). The effectiveness of GMT treatment for adults with ADHD is currently under further research at the Department of Biological and Medical Psychology at the University of Bergen. 2.5 Electrodermal Activity Electrodermal activity (EDA) is the property that causes continuous variation in electrical characteristics of skin. The electrical activity output in human skin was first observed by German researchers in Studies of psychological factors related to EDA is attributed to the 1880s when French researchers discovered a parallel between the amount of anesthesia given to hysterical patients and their subsequent drop in EDA (Neumann & Blanton, 1970). Apparently, there is a relation between the intensity of EDA output and psychological arousal in subjects. The ease of obtaining EDA measures and the simple relation between signal amplitude and arousal has led to electrodermal recording to become one of the most frequently used biosignals in studies of psychophysiology (Boucsein, 2012). Even though the relation between EDA and psychological arousal has been researched since the 1880 s, theorists are still uncertain of their exact causal relation to emotion: At present, psychophysiological patterns for different emotions [...] do not seem easily attainable (Boucsein, 2012, p. 381) Characteristics of EDA Signals Measurement of EDA is traditionally characterized by two types of changes in physiological signal: Tonic skin conductance level: the smooth underlying slowly-changing levels (Empatica, 2016) (See Figure 2.1a). Phasic skin conductance response: the rapidly changing peaks (Empatica, 2016), which are EDA responses to stimuli (See Figure 2.1b). 14

25 Chapter 2. Background 15 (a) Tonic changes in skin conductance, characterized by a slowly climbing EDA signal. (b) In addition to tonic changes in skin conductance, some phasic changes are apparent as rapidly changing peaks in EDA signal. Figure 2.1: Two sessions of EDA measurement presented by TimeOut showing tonic and phasic changes in EDA signal. 2.6 Related Work A search for literature was carried out to present a review of existing technologies and designs to alleviate the symptoms experienced by adults with ADHD. Some criteria were set up for guiding inclusion in the review: 1. Novel designs tackling problems related to ADHD. 2. Designs utilizing physiological sensors. Mainly, the ACM digital library was used as ACM hosts the SIGCHI. In addition, Springer- Link and IEEE were used for further literature search. Research on the development of technologies to support people with ADHD has received limited attention in the HCI community (Sonne, Marshall, et al., 2016). As we can see from Table 2.1 on page 19, research projects focusing on ADHD have gained some popularity in the last few years. Still, research on technology to support adults with ADHD is scarce Designs for Supporting People with ADHD MOBERO is an example of assistive technology for families with children with ADHD (Sonne, Müller, Marshall, Obel, & Grønbæk, 2016). By giving children and parents a list of tasks to do, MOBERO helps to establish morning and bedtime routines. When completing tasks, the child achieves tangible stars for their collection. The user study of MOBERO found that using the assistive technology was beneficial for both parents and children. ADHD is highly heritable, and therefore parents may have some of the same challenges as their children in executive functioning (Helsedirektoratet, 2014; Sonne, Müller, et al., 2016). 15

26 16 Chapter 2. Background FOQUS by Dibia (2016) implements the Pomodoro time management technique, a tool for guided meditation, and positive message priming in a smart-watch for adults with ADHD. Interaction with FOQUS is done intentionally by the user, although heart rate is monitored during guided meditation for the user to assess the effect of meditation on heart rate. Further designs using biofeedback for inciting calmness in people with ADHD have been developed by Sonne and Jensen (2016). In their design ChillFish, children control the movement of a computer animated fish by controlling their breathing. If they master controlled breathing as instructed by the design, the animated fish will collect stars. In this way, ChillFish gamifies controlled breathing as a means for calming children with ADHD before bedtime Design Principles Designing for ADHD Recently, some work has been done on formulating design principles for HCI researchers when designing for ADHD users. Focusing on assistive technology for supporting executive functioning, Weisberg et al. (2014) provide5 the following design principles for their work on TangiPlan: (1) Facilitate organization, time management and planning. (2) Involve caregivers in the process, as they are the main agents of change, but strive to reduce conflict. (3) Implement intervention techniques suggested by experts. (4) Avoid distraction by mobile phones. (5) Avoid intrusion. (Weisberg et al., 2014, p. 294) This set of principles was formulated on the basis of interviews performed with potential users who were diagnosed with ADHD and experts in the treatment of children with ADHD. The experts, notably psychiatrists and psychologists, also (...) stressed the importance of caregiver involvement caregivers are known to play a crucial role in children s motivation and ability to overcome challenges (Weisberg et al., 2014, p. 294). Sonne, Marshall, et al. (2016) argue for some of the same design principles as Weisberg et al. (2014). In Sonne, Marshall, et al. s design strategies, some principles are outlined: Provide Structure to Facilitate Activities: Structure is beneficial for people with ADHD, as they are more likely to succeed in completing tasks if they occur in a predictable pattern (Sonne, Marshall, et al., 2016, p. 67). Minimize Distractions: (...) it is beneficial to limit external distractions in order to prevent people with ADHD from losing attention (Sonne, Marshall, et al., 2016, p. 67). Encourage Praise and Rewards: Praising and rewarding a child or a teenager with 16

27 Chapter 2. Background 17 ADHD is a core element in parent training as this promotes desired behaviors (Sonne, Marshall, et al., 2016, p. 68). A limitation to the presented design principles in regards to the study presented in this thesis is its scope. The work of Weisberg et al. (2014) and Sonne, Marshall, et al. (2016) are aimed at designing for children with ADHD, rather than adults. It is reasonable to assume that designing for adults should take some slightly different steps than for designing for children. Research on designing for adults with ADHD were not found when searching literature for this review. Nonetheless, both children and adults diagnosed with ADHD experience the same symptoms of the ADHD disorder and these principles therefore should be suitable for aiding in the design for adults with ADHD Designs for Stress Monitoring Some explorations in designs of systems dedicated to stress management have been done as well. Sanches et al. (2010) argue that designing systems which automatically detect and warn users of their stress levels are flawed in their inherent reductionist nature. The experience of stress, they argue, is dependent on the individuals resources for coping with stressful events (see subsection 2.2.1). Therefore, their design features an interface that invites the user to reflect on their stressful events throughout the day, and determine themselves whether they have successfully coped in a positive way. Sanches et al. argue that By finding patterns in their own behavior, users can start figuring out both what stresses them and how to cope (Sanches et al., 2010, p. 48). Sanches et al. (2010) found some qualities to be important for achieving such an affective loop for the user in a stress management application: an interactive history of prior bodily states, a sense of aliveness in the interface, fluent transitions between states for all variables measured, and most importantly, an ambiguous and open but still consistent design allowing for users own interpretation and reflections (Sanches et al., 2010, p. 55). See subsection for a further discussion on affective loops. MoodWings by MacLean, Roseway, and Czerwinski (2013) takes a wearable approach to interacting with its users. In their field study of MoodWings, a butterfly is placed in the user s car window that reflects stress levels by angling its wings as it is deduced from the wearers EDA output. In this context, the participants found the butterfly to amplify their experience of stress. McHugh, Dawson, Scrafton, and Asen (2010) present a biofeedback device using heart rate monitoring to trigger sound alerts when school children at risk of expulsion become 17

28 18 Chapter 2. Background stressed. Its purpose is to alert the user and their immediate surroundings of stressors that may lead to violent behavior, so that a systemic intervention, such as anger management, can be applied to the user Summary of Related Work As stated at the start of this section on related work, Table 2.1 on page 19 gives an overview of design research exemplars related to this study. Each design included in the table is detailed by a short description, category for interaction and technology, and a description of its specified user group. The literature search uncovered a few existing designs for supporting people with ADHD; current research on designing for ADHD is scarce. The literature search showed that when designing for ADHD one should minimize distractions, provide structure and implement intervention techniques in cooperation with clinical experts (Sonne, Marshall, et al., 2016; Weisberg et al., 2014). 2.7 Chapter Summary This chapter presented an account of HCI and different kinds of HCI research contributions. Then, the field of affective computing and its counterpart of affective interaction. Pervasive affective sensing systems were presented as an application of HCI and affective computing for monitoring and analyzing affective user data continuously. A short overview of ADHD and a definition of emotional and cognitive control was presented to give a theoretical background to the research question and the constructive design effort of developing TimeOut. Lastly, a literature search was presented in a section on related work. Some designs for ADHD were presented that gave some design principles to take into account in the design of TimeOut. This chapter represent the theoretical grounding of this research project. 18

29 Chapter 2. Background 19 Table 2.1: Overview of related studies. Study Short description Category Target group ChillFish: A Respiration Incite calmness in children Biofeedback, Children with Game for Children with before bedtime by Tangible ADHD. ADHD Sonne and Jensen (2016) following breathing exercise. interaction (TI), Assistive technology (AT) Changing Family Practices Assist families to establish AT, TI Children with with Assistive Technology Sonne, Müller, et al. (2016) effective morning and bedtime routines. ADHD. TangiPlan: designing [...] to enhance executive functioning among children with adhd Weisberg et al. (2014) A Brain-Computer Interface Based Attention Training Program [...] Lim, Lee, Guan, and Fung (2012) In Situ Cues for ADHD Parenting Strategies Using Mobile Technology Pina et al. (2014) MoodWings: A Wearable Biofeedback Device for Real-Time Stress Intervention MacLean et al. (2013) Mind the Body! Designing a Mobile Stress Management Application Encouraging Personal Reflection Sanches et al. (2010) Tracking Mental Engagement: A Tool for Young People with ADD and ADHD Beaton, Merkel, Prathipati, Weckstein, and McCrickard (2014) FOQUS: A Smartwatch Application for Individuals with ADHD [...] Dibia (2016) Hearts on their sleeves : [...] systemic biofeedback in school settings McHugh et al. (2010) Uses tangible objects to help a child complete its morning routines. 3D game where the child controls an avatar by concentrating. Detects stress by EDA wristband, which cues parents to intervene with a well-defined strategy. Wearable butterfly object that flaps its wings when EDA metrics are rising. Mobile service for reflection of both negative and positive behavior. Pomodoro time mangement application for adults with ADHD. Analyzing heart rate data and provides self-calming techniques to prevent emotional outbursts. AT, TI Children with ADHD Neurofeedback, Children BCI, EEG, ADHD. training AT, Biofeedback, EDA Biofeedback, EDA, Wearable Creates a map of locations and events where the user have been mentally engaged. Biofeedback, Wearable, Reflective mobile application, EDA Reflective mobile application, EEG Wear- AT, able Biofeedback, HR Parents children ADHD. with of with Adults who wish to control stress. Adults who wish to contemplate feelings of stress. Young adults with ADHD. Adults ADHD. with Children at risk of expulsion from school. 19

30 Chapter 3 Methodology For this study the following research question were outlined: How can we design interventions supporting emotional and cognitive control in adults with ADHD? In this chapter, an overview of the methods and techniques used to design a prototype as an answer to the stated research question is presented. The focus of this chapter is to give a justification of choices made in regards to research design, methods and techniques used for designing TimeOut. 20

31 Chapter 3. Methodology Design as a Science - An Introduction The scientific method is a pattern of problem-solving behaviour employed in finding out the nature of what exists, whereas the design method is a pattern of behaviour employed in inventing things of value which do not yet exist. Science is analytic; design is constructive. (Gregory, 1966, p. 6) The component of design in research has been a much discussed topic (Cross, 1982; Simon, 1969; Storni, 2015). As technological advances have been made and prompted studies into the construction of new interactive systems, there has been a need for formalizing the practice of design within the realms of the scientific community. Early developments of design science do not consider the artifact as a legitimate or important source of study, whereas contemporary HCI research employs an exploratory approach in which the hypothesis is subject to continuous re-framing (Zimmerman, Forlizzi, & Evenson, 2007). Cross argues in Designerly Ways of Knowing that one does not have to understand mechanics, nor metallurgy, nor the molecular structure of timber, to know that an axe offers (or explains ) a very effective way of splitting wood (Cross, 1982, p. 26). With this quote, he is pointing out that knowledge resides in objects, and we should therefore regard scientifically crafted artifacts (called prototypes in this thesis) as sources of scientific knowledge. In the field of HCI a framework for producing knowledge from the design of prototypes, research through design (RtD), has been widely adopted (Zimmerman et al., 2007). This study uses RtD as a framework for guiding the research process to scientifically construct a prototype and produce knowledge from said prototype. 3.2 Research Through Design Zimmerman et al. (2007) sought to differentiate what encompasses design practice from design research. In their proposed model of how to conduct HCI research, they emphasize how interaction designer work to create the right thing : a product that transforms the world from its current state to a preferred state (Zimmerman et al., 2007, p. 493), whereas industry practitioners of design focus on making commercially successful products. 21

32 22 Chapter 3. Methodology Problem Identification Ill-defined or wicked problems are central to RtD. These are problems that are complex or vague in ways that traditional engineering approaches do not apply. Originally coined by Horst Rittel in the context of organizational science, wicked problems are a class of social system problems, which are ill-formulated; where the information is confusing; where there are many clients and decision makers with conflicting values; and where the ramifications in the whole system are thoroughly confusing (Churchman, 1967, p. 141). To address such wicked problems in design research, the RtD framework postulate that interaction designers should Integrate the true knowledge (the models and theories from the behavioral scientist) with the how knowledge (the technical opportunities demonstrated by engineers). Design researchers ground their explorations in real knowledge produced by anthropologists and by design researchers performing the upfront research for a design project. (Zimmerman et al., 2007, p. 497) In the case of this study the wicked problem is to support emotional and cognitive control for adult ADHD patients by designing adaptive technology. To define a problem space for this challenge workshops were set up in cooperation with the INTROMAT research project. In the course of three workshops HCI researchers, psychology researchers, design professionals and some expert users met to discuss and share ideas that could cumulate in adaptive technology for addressing the aforesaid problem. Further details on this workshop can be found in subsection Evaluation of the Design Process In any research paradigm there is the need for criteria to evaluate what is sound research. Zimmerman et al. (2007) set out some guiding principles for evaluating prototypes in regards to providing the scientific community with knowledge that can be built upon: Process: There is no expectation for interaction design research to be reproducible. That is, as artifacts of design research are unique, reproducing a research project s process will not result in the same artifact. Therefore, judgment of the quality should be whether the research has been applied with rigor, and the rationale for choosing methods and decisions on design choices (Zimmerman et al., 2007). Invention Further, the novelty of a design research project is critical for it to be considered a contribution to the research community. An artifact created through research must demonstrate significant invention. Therefore, a thorough literature 22

33 Chapter 3. Methodology 23 review is needed to demonstrate how the contribution advances the research community (Zimmerman et al., 2007). Relevance: As discussed above, there can be no expectation that two designers given the same problem will come up with identical artifacts. Instead of applying this criteria of validity that is central to behavioral sciences, design research should argue for its relevance. That is, how the process is framed in respect to the real world, and why it is an important problem to solve by design research (Zimmerman et al., 2007). Extensibility: Finally, the ability to build on the resulting outcomes of the interaction design research: either employing the process in a future design problem, or understanding and leveraging the knowledge created by the resulting artifacts (Zimmerman et al., 2007, p. 500). For example, a research through design study could result in design considerations or design implication, which future research could build upon. Concluding this thesis, a thorough discussion of what was found in the construction and evaluation of the proposed design is presented in chapter 6. From this discussion some design principles for further work in this field is proposed for other design researchers to apply when tackling similar challenges Why RtD? This research is framed within the framework of RtD because it acknowledges the prototype as a source of knowledge and as a contribution to research. Further, RtD represents a framework that is widely adopted by papers presented in HCI conferences such as Conference on Human Factors in Computing Systems (CHI) which this thesis generally cites throughout. 3.3 Involving users: User-Centered Design As described in section 2.1, HCI made a shift in the 1980 s towards involving users in early stages of interaction design processes to ensure usability standards were successfully achieved. In todays discourse for interaction design the involvement of users is a matter of course. For this research project the principles and common techniques of User-Centered Design (UCD) was employed to guide and evaluate the design process. According to Rogers et al. (2015), what makes out UCD today is: 23

34 24 Chapter 3. Methodology Early focus on users and tasks. Empirical measurement. Iterative design. In the following subsections some techniques that were used for establishing requirements for this design process are described Persona Personas are not real people, but they represent them throughout the design process. They are hypothetical archetypes of actual users. Although they are imaginary, they are defined with significant rigor and precision. Actually, we don t so much make up our personas as discover them as a byproduct of the investigation process. We do, however, make up their names and personal details. (Cooper, 1999, p. 85) Persona as a method for emphasizing the user in design is a controversial topic. Usually attributed to Cooper (1999), a persona avoids problems of involving users by simply excluding them (Blomquist & Arvola, 2002, p. 197). Pruitt and Grudin (2003) hold that personas allow for understanding of the values, fears, needs of users, while Bødker, Christiansen, Nyvang, and Zander (2012) found that personas cannot make absent users have an active role in design (p. 99). Cooper argues that details of personas are not made up, however, there are no formal agreed-upon method for eliciting personas from data gathered about real users. In a study of the use of personas in Denmark, L. Nielsen and Storgaard Hansen (2014) found that out of 12 interviewed companies the majority used qualitative data to define their personas. They found that personas provided the companies with a common language that enabled them to discuss the needs of their users critically. For this study, personas were used primarily for communicating an archetype of a possible user. Personas was elicited from workshops held with potential users Scenario A scenario is an idealized but detailed description of a specific instance of human-computer interaction (Young et al., 1987). The scenarios help designers understand and analyze problems they set out to design by evoking work-oriented communication among stakeholders and afford multiple views of an interaction (Rosson & Carroll, 2002). 24

35 Chapter 3. Methodology 25 In a survey of the use of scenarios in different fields of research, Go and Carroll found that some key characteristics make out the use of scenarios in HCI: Analyzing user tasks Envisioning future work Mock up and prototyping Evaluating the constructed system Deriving learning materials Developing design rationale (Go & Carroll, 2004, p. 47). As we see from their list of characteristics, scenarios can be used throughout the design process. Bødker (2000) performed a study of three Danish companies different take on the use of scenarios for designing systems and products. One example of use was the provocation of thoughts and ideas through plus and minus scenarios (Bødker, 2000, p. 69). These scenarios address the positive and negative aspects in a caricatured manner. Bødker argues that by emphasizing the extreme consequences of a design, as opposed to giving a nuanced middle ground, it is easier to discover all sorts of demands for a future application. For this study text-based scenarios were created to develop a design rationale in an early stage, and for communicating the purpose of the design. 3.4 Prototyping A prototype is what Zimmerman et al. (2007) refer to as the embodiment of the right thing, as detailed in section 3.2, when constructed by design researchers. Prototypes are manifestation of a design that allows stakeholders to interact with it and to explore its suitability; it is limited in that a prototype will usually emphasize one set of product characteristics and de-emphasize others (Rogers et al., 2015, p. 386). The design community distinguishes prototyping in low-fidelity and high-fidelity modalities. Low-fidelity prototyping refers to visualization of design ideas (Sefelin, Tscheligi, & Giller, 2003, p. 778), specified by Rudd, Stern, and Isensee (1996) as [depiction of] concepts, design alternatives, and screen layouts. Prototypes of this modality are limited in function and interaction. Mockups, wireframes and Wizard of Oz-demonstrations are concepts that fall into this category. Common for these are the necessity of a facilitator for the demonstration or testing of the prototype (Rudd et al., 1996). High-fidelity prototyping, on the other hand, should have the complete functionality and interactivity of the intended final design. Prototyping in high-fidelity is critical for a research through design-project as this demonstrates what the envisioned right thing constitutes. For a comparison of low- and high-fidelity modalities of prototyping see Table

36 26 Chapter 3. Methodology Table 3.1: Comparison of low-fidelity and high-fidelity prototyping (Rogers et al., 2015, p. 395). Modality Advantages Disadvantages Low-fidelity Lower development cost Evaluates multiple design concepts Useful communication device Facilitator-driven Limited usefulness for usability tests Poor detailed specification to code to High-fidelity Complete functionality Fully interactive Look and feel of final product Use for exploration and test More resource-intensive to develop Time-consuming to create Not effective for requirements gathering 3.5 Evaluation Iterative design processes are dependent on repeated evaluation to gather data on how proposed design choices are perceived by potential users and domain experts. These data form the basis for a rationale on how we define the next iteration s design choices. For the design process that resulted in TimeOut, presented in this thesis, each iteration s contribution was evaluated to determine further iterate the design process. Cognitive walkthrough evaluation was conducted to give a group of domain experts, clinicians and patients, a sense of where TimeOut was headed. Their continued feedback shaped the design process. An overview of the conducted evaluations throughout the design process is displayed in Table 3.2. Table 3.2: Overview of design evaluations presented in this thesis Design phase Evaluation type Participants Page Conceptual design Domain expert evaluation 1 33 Biosensor feasibility study Controlled experiment 3 37 First iteration of lo-fi prototype Cognitive walkthrough 5 36 Second iteration of lo-fi prototype Cognitive walkthrough 6 43 High fidelity prototype Field trial

37 Chapter 3. Methodology Field Trials Technology Deployment in the Wild As pervasive affective sensing systems concern human emotions and related complex phenomena, it is difficult to examine the actual use patterns and personal experience of using such designs. Field trials offer researchers the opportunity to deploy technology in a user s natural setting and thus study the use in a real context. In the wild field trials would, therefore, be an appropriate evaluation method for TimeOut. After introducing the benefits and challenges of performing in the wild field trials, a semi-formalized method technology probing will be briefly presented. These methods make up the technique used for evaluating the last iteration of TimeOut for this research, which is presented in chapter 5. Early methods in HCI research of testing interactive prototypes usually involved cognitive experiments in controlled environments. Beginning with the shift to ethnographic research in the second wave of HCI, field trials have gained prominence and are characterized by its turn to the wild (Bødker, 2006; Crabtree et al., 2013; Olson & Kellogg, 2014). By performing field trials in the wild, researchers are evaluating technology in their actual use contexts such as the participant s personal homes or workplaces. Unlike traditional user-centered design evaluation methodologies such as lab trials, heuristic usability evaluations, and ethnographic approaches, in the wild field trials evaluate new technologies and experiences in situ (Rogers, 2011). This allows the researchers to observe and record how prototypes are used and integrated within people s lives (Chamberlain, Crabtree, Rodden, Jones, & Rogers, 2012). Brown et al. (2011) argue that in the wild field studies not only accept but welcome unanticipated use of technology. Brown et al. (2011) call for more transparent and honest communication of HCI researcher s field trials in regards to method and results. Abandoning any notion of deterministic relationships between technology and use, thus rejecting controlled experiments for rich user studies, Brown et al. accept that no field trial is free of bias or other fatal flaws as it may be interpreted with a positivistic perspective on research validity. Some distinct features of in the wild field trials were found and presented by Brown et al.: Demand characteristics: One source of behavior for trial participants is their interpretations of what would be seen as the right behavior for those running the trial (Brown et al., 2011, p. 1662). Brown et al. argue that the desire to produce valuable data for researchers are an inherent quality in participants of field trials as participation demands the generosity of investing time and energy over an extended period of time. 27

38 28 Chapter 3. Methodology Lead participants: Participants that engage with the technology and reflect on its use by themselves and others in a particularly insightful way, or [...] encourage involvement by others who are involved in the trial (Brown et al., 2011, p. 1663). Brown et al. view this feature of lead participants as a major benefit of in the wild field trials as these participants may discover uses of technology that otherwise could remain unknown. Trial design: The relationships between the participants, and our framing of the system at the start of the trial, led to very different types of use between the different participants (Brown et al., 2011, p. 1664). I.e., there is no expectation of reproducibility of results found in in the wild field trials. Furthermore, no subjects are alike just as any social setting contains far too much variability to be exactly reproduced. As detailed above, in the wild field trials have some inherent issues. Brown et al. argue that researchers should embrace these issues and be honest about their effects on results, both positive and negative. Further, Brown et al. encourage researchers to be honest about the failures of their designs, while at the same time abandon the notion that a failed design equates to failed research. In the next section, technology probe as a method for performing in the wild field trials, will be presented. Technology Probe Technology probes are a particular type of probe that combine the social science goal of collecting information about the use and the users of the technology in a real-world setting, the engineering goal of field-testing the technology, and the design goal of inspiring users and designers to think of new kinds of technology to support their needs and desires (Hutchinson et al., 2003, p. 18). Drawing on inspiration from Gaver et al. (1999) s cultural probe for eliciting user requirements, Hutchinson et al. developed the technology probe as a means of evaluating early designs by field-trial, but perhaps more interestingly, as a means of discovering unexpected use of technology. Hutchinson et al. holds that technology probes should be used in early phases of design processes to inspire future iterations. Therefore, technology probes should be designed to: Provide simple functionality. Offer flexibility and encourage users to reinterpret the design. Provoke users to think critically and inspire creative thought (Hutchinson et al., 2003, p. 19). Following these guidelines for designing a technology probe, the probes may motivate par- 28

39 Chapter 3. Methodology 29 ticipants to inspired engagement in interviews and workshops. For the design and development of TimeOut, technology probes were used as a method for conducting a field trial after the fourth, and presently, last design iteration. Although it is recommended by Hutchinson et al. (2003) to use technology probes in the early stages of design, it was necessary to first develop TimeOut as a functional prototype before probing it to participants. 3.6 Chapter Summary This chapter detailed the research design of the study presented in this thesis. To answer the research question of this study, a research through design-framework was applied to find answers that are extensible to other research projects. Techniques and methods for involving users in the design process was described, and evaluation methods for researching empirical evidence of the use of the constructed prototype were detailed. 29

40 Chapter 4 Development of Prototype Following a research through design approach involves finding the true knowledge of a domain and utilizing the how knowledge of various fields of engineering to create the right thing, as detailed in chapter 3. This chapter details the iterative process of designing and developing TimeOut in a research through design-based exploration into pervasive affective sensing for the purpose of supporting users self-regulation of emotional and cognitive experiences. A total of four design iterations were completed to produce the current state of TimeOut. While the project started out as an application for detecting and alleviating mental stress, it evolved to consider a more general sense of stress; activation. The effort to design TimeOut included colleagues from the INTROMAT research project. Especially for the third and fourth iteration where the team in their roles as possible future users, domain experts, and HCI/design experts, greatly influenced the design of TimeOut. As such, TimeOut reflects a team effort to design a pervasive affective sensing system for adults with ADHD. Before detailing each iteration, a summary of each iteration is presented: First iteration: Eliciting requirements by performing workshops with users that resulted in a persona and scenario. A conceptual design was articulated and evaluated by a domain expert. Second iteration: Presents mock-ups of interactional flow and lo-fi prototype. Biosensor wearable acquired and studied in a small-scale controlled experiment. Third iteration: Design of task-based intervention and implementation of simple algorithms for detection of affective states. Further design and evaluation of lo-fi prototype. Fourth iteration: Improvement of algorithms, further design explorations, and an implementation of a fully functional affective sensing system. 30

41 Chapter 4. Development of Prototype First Iteration The design exploration began with several meetings held by the INTROMAT research group. As these were the very first meetings between peers of different professions and academic disciplines it was necessary to establish a shared understanding of the user group for this project. As the INTROMAT research project is a user-centered initiative, three adults diagnosed with ADHD were invited to our workshops as expert users. Their involvement later evolved to becoming participants of the design team, although this is not further documented in this thesis as their involvement does not directly include the TimeOut design process. In this section, I will give an overview of the workshops held with our expert users and detail the persona and scenario elicited from these meetings. Drawing on these items, a conceptual model of a future prototype is given and evaluated by an expert on neuropsychology Design Workshops A total of four workshops were held, three of which involved adults diagnosed with ADHD. The goals of these workshops were to: establish a shared sense of what the experience of having ADHD may be, brainstorm some possible applications for assisting adults with ADHD, and ideate these possible applications into plans for future prototypes. Methods used included general verbal discussion, drawing sketches, and categorizing ideas by organizing sticky notes. These workshops were documented by the participants personal notes and photographs of sticky notes, as can be seen in Appendix C. For the first workshop, academics in the field of neuropsychology, HCI, master students of HCI and design professionals from the consultancy industry met to discuss opportunities and possibilities of designing assistive technology for the support of adults with ADHD. For the second and third workshop, three adults who struggle with difficulties associated with their ADHD diagnosis in their daily lives joined us as participants. A summary of the most significant experiences shared by the expert users that are relevant to this thesis project are given: The participants described many different needs and experiences they have had concerning their condition. One of particular interest to this study was the need for them to stop and recollect their thoughts every once in a while. This intervention was said to be something that usually had to be initiated by someone external to themselves. Typically, this person 31

42 32 Chapter 4. Development of Prototype intervening to help stop and make sense of the situation was a friend, significant other, or family member. Examples of situations where the need to stop emerged were given, also addressed metaphorically as a red stop-button. One participant shared a story of how the participant in a work context had the habit of working for hours in deep concentration on cognitively demanding tasks. The participant had a manager that was conscious of his or hers dispositions, so the manager would actively compel the participant to take breaks. In the absence of such a manager in other jobs, the user would experience job burnout. An important aspect to note here is the inability to properly self-regulate their thoughts in these situations and the emotions they may elicit. Some attention were given to their habits on the use of ICT. It was mentioned that smartphones and tablets were actively used, and found instrumental in organizing their day-today activities. When asked what they would want if they could have any imaginable object, one user expressed the need for a system that gives the necessary control to achieve effective actions. While that may sound elusive, it was a powerful ethos to help establish the design process that resulted in TimeOut. To easily, and efficiently, grasp the essence of these experiences in terms of common methods for communicating design ideas and processes, a persona and scenario were created to elaborate on an archetype of the typical users and their situations. Persona Figure 4.1 presents the persona that guided this and future iterations of TimeOut. It was inspired by literature in section 2.4 and the workshops documented above. Figure 4.1: Persona illustrating the archetypal user of TimeOut. 32

43 Chapter 4. Development of Prototype 33 Scenario The scenario that accompanies the above persona was created to emulate a sense of the potential user s everyday challenges and needs associated with their condition of ADHD: Dave has recently recovered from months of sick leave from his work as an editorial writer for a financial magazine. He suffered from severe occupational burnout after constantly feeling pressured by deadlines and working long hours. While he was away, doctors diagnosed him with ADHD. At first surprised, Dave now finds comfort in having an explanation for his feelings of being different from others in regards to work habits, social skills, and activity level. Having learned techniques for regulating his cognitive and emotional states in therapy, Dave is excited, yet anxious about his return to the work force. He is motivated to work, but worried that he might relapse into occupational burnout Having established a target user in terms typical of user-centered design projects, work on the design of TimeOutwas started Conceptual Design Drawing on what was found as a problem space from the workshops detailed and interpreted in subsection 4.1.1, a concept of a proposed solution is presented: TimeOut: A digital assistant that in some way senses your affective states, primarily mental stress. When the measured intensity of stress exceeds the individual threshold, the assistant performs an intervention for the user. In that way, the system walks the user through some steps necessary to alleviate their distress and make sense of the situation Evaluation As the concept for TimeOut was articulated, it was necessary to evaluate the feasibility of such a project with regards to adaptability and usefulness in a clinical context. A professor of neuropsychology, affiliated with the INTROMAT research project, was contacted to give an expert evaluation of TimeOut s concept. First of all, the domain expert displayed interest in the project. As adults with ADHD have difficulties regulating emotions, a digital assistant could empower users with better self-regulation, it was said. In terms of how to intervene in the case of distress, GMT (see subsection 2.4.3) was suggested as a protocol for intervention. When speaking of the digital assistant s ability to sense mental stress, it was suggested to compare the output of biosensors to self-reported estimations of mental stress in an experimental study prior to the design phase. 33

44 34 Chapter 4. Development of Prototype 4.2 Second Iteration Continuing the design of TimeOut, establishing some requirements of the design was necessary. Research was done to discern existing biosensors and technologies for detecting mental stress. EDA was chosen as a measure to recognize stress in the user s affective states, as EDA is reflected by sweat glands that are controlled by the sympathetic nervous system (see section 2.5). In pursuit of the evaluation from the previous iteration, some experiments were needed to validate the EDA readings from our chosen wristband device. Mock-ups of TimeOut were designed and presented at a research consortium for INTROMAT researchers Establishing Requirements Considering the conceptual model presented in and its evaluation regarding feasibility, some requirements of the system were determined. The system must provide the user with: 1. detection of the user s stress level 2. an intervention to alleviate feelings of stress when it is detected (a) feedback to alert the user when stress levels are increased (b) tasks given to the user must be in line with the GMT protocol (see 2.4.3) Technical Requisites To achieve requirement 1, a biosensor wearable was necessary. The E4 Wristband by Empatica (Figure 4.2) was acquired for this purpose. Equipped with a photopletysmography sensor that measures blood volume pulse, an accelerometer, an infrared thermopile, and an electrodermal activity sensor, this wristband has an array of biosensors that by themselves, or combined, could give an indication of mental stress. Figure 4.2: Empatica E4 wristband device 34

45 Chapter 4. Development of Prototype 35 Some other wearables were also considered as candidate technologies for TimeOut. Empatica Embrace is a light-weight EDA-sensor equipped wearable wristband with haptic feedback, and Apple Watch 2 is equipped with a blood volume pulse (BVP) sensor from which heart rate and heart rate variability can be derived. Unfortunately no application programming interface (API) or software development kit (SDK) exists for either technology. Furthermore, the E4 wristband communicates with mobile operating systems through an API by low energy Bluetooth (BLE) signals. Empatica has published and currently maintains a Software Development Kit (SDK) for Android. As I have training in Java programming and experience with developing applications for the Android operation system, this mobile operating system was selected as a platform for TimeOut Further Design and Prototyping In an effort to explain the interactional flow of TimeOut, a Business Process Model and Notation (BPMN; see White, 2004) representation of its components was modeled (Figure 4.3). The purpose of this model was to map out the distinct features of such a pervasive affective sensing application, and identify gaps in the flow of interaction. As is apparent from the figure, details on how the user would actively stop, appraise the situation and alleviate feelings of stress are absent in the current iteration. Figure 4.3: BPMN illustrating the components and flow of TimeOut 35

46 36 Chapter 4. Development of Prototype Some mock-ups (Figure 4.4) were made to communicate the concept of TimeOut to an audience at a INTROMAT research consortium at Haukeland University Hospital. While these mock-ups are visually simplistic, the prototype demonstrates how levels of stress are represented by colors. To receive the intervention for alleviating stress the user must actively press the Time out -button. Furthermore, the actions the user should perform to alleviate stress are not apparent, similarly to what is modeled in the interactional flow model (Figure 4.3). Figure 4.4: Low-fidelity prototype of TimeOut. Evaluation of Prototype As mentioned, the interactional flow model and prototype were designed for a presentation at a research consortium at Haukeland University Hospital hosted by the INTROMAT project. While the model was not presented to the audience, it was explained to interested participants. Some input were given and discussed informally both by colleagues of our sub-project within INTROMAT (chapter 1) and participants of other sub-projects within INTROMAT. Below a summary of these discussions are provided: Present the user with a sequence of tasks to recuperate from a stressful situation as: Stop what you are doing. Reflect on your current situation. Follow a mindfulness-inspired meditation, e.g. a breathing exercise. Alert the user with a cue for intervention as a pop-up dialog that may be accepted or dismissed. These summarized points from the evaluation gave some guidelines on how to move forward with the development of TimeOut. To become acquainted with biosensor technology, a small feasibility study of the Empatica E4 wristband was performed to inform future design iterations of TimeOut. 36

47 Chapter 4. Development of Prototype Feasibility Study of Biosensor Wristband Electrodermal Activity (EDA) was found to be a possible output for measuring mental stress, as detailed in section 2.5. The industrial standard of measuring subjects EDA are done by using complicated laboratory equipment. These are intricate and bulky machines, where wires connect electrodes to the subjects palm and fingers. As the Empatica E4 is a compact and relatively discreet wristband device, there is consequently a need of validating that this device provides measurements that correspond with findings presented in existing research that utilize stateof-the-art laboratory equipment. Experiments were designed and carried out in a research design comparative to those of psychological single subject studies. We sought to validate a distinct responses in EDA values as: Increasing galvanic skin conductance while completing a stressful arithmetic task. For the experiment a reversal design was chosen, also known as A-B-A. The letters relate to specific stages in the experiment. For stage A, the subject is instructed to rest while GSR is measured. The data recorded in this period of rest functions as a baseline, to which we compare to the measured response on stimuli introduced in stage B. We then repeat stage A to see how the new baseline compares to the GSR data recorded prior to issuing the stimuli in stage B. Procedure An arithmetic test was designed to induce mental stress in the subject. This test is a part of the Trier Social Stress Test (Birkett, 2011) where the participant is questioned by examiners acting in an authoritative manner to induce mental stress reactions. We extracted the following procedure for our experiment: During the [...] five-minute math portion of this task you will be asked to sequentially subtract the number 13 from 1,022. You will verbally report your answers aloud, and be asked to start over from 1,022 if a mistake is made (Birkett, 2011, p. 2). The test was implemented as a web application by using HTML, CSS and JavaScript, a screenshot can be seen in Appendix B. EDA was measured for rest time, during test, and then for five minutes rest after test. While completing the arithmetic tasks, participants were asked about their stress level on a likert scale of 1 5, where 5 is most stressed, and 1 is least stressed. 37

48 38 Chapter 4. Development of Prototype Three participants, all in their 20 s, were recruited by posters on social media sites. The examiner had no relation to any of the participants. Results Figure 4.5: Recorded EDA levels of participant #1, red line marks test start. The first participant expressed a low level of stress (2) immediately before the test started, but quickly reported higher levels of stress (4, 5) throughout the test. EDA data recorded of the participant show a steady EDA reading in rest time 6, with a rising tonic level in anticipation of the test. While performing the test, EDA levels saw a rapid rise (0.6 µs to 1 µs). This correlates with the participants self-reported stress levels. The participant also stated that as the timer counted down, and I had to start over for every mistake, my feelings of stress rose quickly. The participant never made it below 800. Due to technical difficulties, we have no recording of the participant s arithmetic results. The second participant experienced the arithmetic test as less challenging. Except for a typing error by the examiner he had no errors. In line with these observations, the participant s self-reported stress levels were low throughout the test (1 2). EDA levels were fairly stable throughout the test, but there are limitations as we have no baseline as resting time were never monitored. Measurements of heart rate show varying beats per minute (75 10), and then settling in the rest time after arithmetic testing (90 75). It was noted that the participant seemed stressed, contrary to the participant s self-assessment. Commenting on their experience of participating, the participant said he found the experiment to be a fun challenge and also noted that he is very competitive. Results from our third participant were invalid as the E4 wristband recorded corrupt data. Several attempts were made with the equipment, but for some unexplained reason the wristband would not read the participant s physiological data correctly. Conclusion and Implications for Further Development The tests showed that readings from the E4 wristband were similar to those of prior research. There are some limitations to this experiment as there were technical problems for 38

49 Chapter 4. Development of Prototype 39 each session; the first participant s answers were not recorded, second participant s EDA levels were not monitored while resting prior to the test, and for the third and final participant we were unable to record valid EDA data. Furthermore, the small sample size used is a limitation to this study. That being said, it does not take away from the utility of the experiment for this design process. Several discoveries and implications for further development were found in the course of doing the experiment and interpreting the results. It is interesting to note that the test subjects reacted differently to the arithmetic test in terms of skill and consequentially their self-reported stress levels. The opposite could hold true as well; that stress level impairs arithmetic skills. Many factors could contribute to differing reactions to being tested, such as personality traits. For one who is competitive or goal-oriented, a test of arithmetic skill poses a challenge where the suppression of anxiety, doubt and other emotions related to mental stress are suppressed as a means of coping. When considering the elicited requirements for TimeOut, detailed in subsection 4.2.1, the first requirement of prediction and detection of the user s stress level does not seem feasible within the scope of this research project. As we have learned from chapter 2 and the course of this experiment, the perception of stress is a highly subjective experience. 39

50 40 Chapter 4. Development of Prototype 4.3 Third Iteration Continuing the design of TimeOut, the previous iteration s prototype was further designed to incorporate what was found in the evaluation (subsubsection and the feasibility study of using EDA for supporting emotional and cognitive control (subsection 4.2.3) Refining Requirements Taking into account what was found in the evaluation of the second iteration s prototype presented in subsection 4.2.2, and subsection on the use of Empatica E4 for the detection of mental stress, it is necessary to adjust some of the requirements presented in subsection Regarding the first requirement, that is prediction and detection of the user s stress level, my experience from the previous iteration s controlled stress experiment greatly reinforced the notion that emotions are of a subjective nature and cannot easily be extracted as information. Keeping these results in mind, and building on ideas of the affective interaction perspective as described in section 2.2, I propose some adjusted requirements that reflects the idea of interpretive flexibility (see subsection 2.2.2): 1. Continuous detection of the user s stress level (a) Baseline by selecting lowest recorded GSR value (b) Threshold for stress alleviating intervention at GSR values that exceed an increase of 80 percent from baseline (c) Biofeedback by suggestive visual representations while monitoring 2. An intervention to alleviate feelings of stress when it is detected (a) Given as a set of tasks to be completed by the user: Stop what you are doing. Reflect on your current situation. Follow a breathing exercise. Go back to what you were doing before you accepted to stop Designing for Interpretive Flexibility In an effort to establish interpretive flexibility in the user s feedback loop, an interpretive layer was introduced between the raw data and information displayed. In practical terms, this meant the design of a visual representation that reflects the current GSR readings relative to the computed baseline and the threshold level that cues the intervention. 40

51 Chapter 4. Development of Prototype 41 A green color signifies that the GSR level read from the user s skin is close to the computed baseline. Yellow color signifies that the user s GSR level is halfway between the baseline and the threshold, while red signifies that the GSR level is close to the threshold level (Figure 4.6). Figure 4.6: Suggestive visualizations of the user s EDA data Intervening to Alleviate Stress Based on discussions with researchers in the field of psychology and neuropsychology as described in subsection 4.3.6, and the refined requirements in subsection 4.3.1, a set of tasks for the user were designed. A material card design pattern was used to present the tasks as objects that the user can interact with (as seen in Figure 4.8), with actions such as swipe to delete. Figure 4.7: A cue for intervention that is dismissible by avslå. To cue the intervention a dialog design was used (Figure 4.7). By asking the user Stop? the design seeks to reinforce the interpretive flexibility and incite a sense of self-awareness for the user. To further strengthen its qualities as an application for improving self-awareness, the concept of end-user tailoring was implemented in the design. The intervention dialog is designed for customization of the image and stop-word that is shown in Figure 4.7 (e.g. 41

52 42 Chapter 4. Development of Prototype changing the text cue to Hold up!, or switching the image with something that is personal for the user) Algorithm for Pervasive Affective Sensing In order to have a baseline for continuous comparison with new GSR readings, an algorithm was programmed in Java and tested on an Android phone with the Empatica E4 as the source. Below, the algorithm is represented in pseudocode. Algorithm 1 Basic algorithm for establishing EDA baseline. function Find baseline(newgsr) Called every 10 seconds basel inegsr Initialize return variable and comparator. if newgsr < baselinegsr then basel inegsr newgsr return basel inegsr The parameter newgsr represents an average number of the past 10 seconds GSR readings, which amounts to a total of 40 data points of GSR data. The variable baselinegsr represents a computed baseline for which newgsr are compared with in a separate function Find GSR change. Algorithm 2 Function to compute percentage change in new EDA in regards to baseline EDA. function Find GSR change(newgsr) basel inegsr Find baseline(newgsr) percentagechange percentage change between basel inegsr and newgsr E.g. 40% return percentagechange In this iteration the variable percentagechange was not used for anything other than debugging as the only implemented part of the application thus far was the Empatica E4 wristband API connection and the above algorithms. Evaluating Algorithms Some tests were performed to check the robustness of the algorithm Find baseline. While it did return the lowest GSR reading as a baseline, this proved to be unsatisfactory in reallife settings. Due to hand and other bodily movements the EDA sensor could momentarily lose contact or hold less firmly to the skin, thus resulting in an unnaturally low baseline. This in turn sets the threshold GSR level for cuing an intervention lower giving the user unnecessary cues for stopping and intervening. 42

53 Chapter 4. Development of Prototype 43 The same issues were found for Find GSR change. Manipulating the application to issue false positives was easily achieved by shaking the hand wearing the wristband vigorously. It was found that for the next iteration it would be vital for the stress assessment module s accuracy to completely rewrite the baseline algorithm to handle these unfortunate sources of error in stress monitoring. Furthermore, it was found that fusion of separate sensors, such as EDA, thermometer and accelerometer, could filter out contextual conditions such as removal of wristband and body movement to ensure a more robust affective sensing Prototype A prototype was made to further develop the design of TimeOut (Figure 4.8). It was made using Adobe Experience Design. To create a design that is coherent with other Android applications which users could be accustomed to, Google s design guidelines Material Design were followed as a reference for stylistic and aesthetic design language (Google, 2017). Figure 4.8: Second iteration of lo-fi prototype. While this is a low-fidelity prototype, it is functional in terms of navigation and interaction flow Cognitive Walkthrough Evaluation To evaluate the third iteration s prototype a cognitive walkthrough evaluation were held with colleagues of the INTROMAT ADHD-case team. In attendance were neuropsychologists, expert users, design professionals, and a HCI researcher. The group showed interest in the prototype and was generally positive to the design decisions that had been made. Phrasing the cue for intervention with a question (Figure 4.7) 43

54 44 Chapter 4. Development of Prototype was positively received by those with clinical experience, they found that it leverages the user as an active part in the decision making. It was argued by the neuropsychology experts that use of the word stress should be downplayed because it could denominate negative emotions and experiences. Instead of stress, the term activation (Norwegian: aktivering) was advised because it denotes a neutral measure of emotional and cognitive incitement (see section 2.3 for further reference). There was some confusion on the interactional flow of tasks as swipe-able cards. The team s HCI expert recommended forcing the flow of tasks, instead of giving tasks as a customizable sequence. Furthermore, it was proposed that the next prototype should give users the opportunity to annotate the intervention with a description of their experience (based on Ecological Momentary Assessment as defined by Shiffman, Stone, and Hufford (2008)). The utility of such a feature is to let the user take some ownership over the data that was generated by annotating it with their subjective feelings and experiences that they thought caused the intervention. The expert users showed equal interest in the project as the rest of the group s members, while one user commented that if a machine tells me how I feel then I would view that as a veridical representation of my feelings. This was interesting to the TimeOut project, as it confirmed some concerns of designing pervasive affective sensing systems. While it is a minor comment, it represents an attitude towards computers that reinforces the ethos of designing with considerations from the perspective on affective interaction (see subsection 2.2.2). 44

55 Chapter 4. Development of Prototype Fourth Iteration In the fourth and final iteration of this design project, a fully functional pervasive affective sensing application for Android phones was implemented. This section documents the final development phase of TimeOut for this thesis Incorporating Feedback from Previous Iterations Having had a constructive walkthrough evaluation of the third iteration s prototype, there were several design elements that needed to be reworked and some new features to be designed to meet the new requirements of TimeOut. Refining Requirements These are the adjusted and final requirements of TimeOut: 1. Continuous detection of the user s activation level (a) Find baseline by processing all current GSR values in a moving average algorithm (b) Continuous biofeedback by suggestive visual representations while monitoring the user (c) Cue an intervention to stop and reflect when GSR values exceed an increase of 80 percent from baseline 2. A set of tasks to be completed by the user as an intervention: (a) Stop what you are doing. (b) Follow a breathing exercise. (c) Reflect on your current situation. (d) Assess and describe the experience you had prior to the intervention. (e) Go back to what you were doing before you accepted to stop. 3. Provide a history of previous interventions and the related user assessments (a) Provide a line chart for GSR values: i. prior to intervention ii. while user was completing tasks until saving assessment (b) Display description and metrics provided by user in the assessment Designing for Ecological Momentary Assessment To meet the requirement of affording the user to assess and describe the experience prior to the intervention, a module for Ecological Momentary Assessment was designed for the application. A dialog for assessment of the subjective experience was designed (Figure 4.9). By using likert scales that vary from 1 to 10, where 1 signifies very low or no 45

56 46 Chapter 4. Development of Prototype distraction/unpleasantness and 10 signifies very high distraction/unpleasantness, the user share their subjective assessment of the situation. The EMA dialog was designed in cooperation with domain experts of psychology. Figure 4.9: Assessment dialog for recording subjective experience prior to and during intervention. By asking the user How distracted were your thoughts/emotions at the time of stopping? and How unpleasant was your experience of these thoughts/emotions? the application is capable of annotating the sensor data that was recorded. This was a major gain for this thesis project as this made it possible to measure how precise the application was in detecting highly activated states of emotional and cognitive activity. In addition to likert scale metrics, the user can give a short description of their experience either by direct text input, or the use of speech-to-text functionality. The benefit of this is that the user can go back in their history (see Figure 4.10) and try to make sense of what triggers their emotions and how well the tasks they performed eased activation. With this, they can also annotate that they have experienced a false positive and explain what they think caused it (e.g. someone told a hilarious joke). Lastly, the user can give an assessment of how well it worked for them to follow the stoptechnique on a scale from 4 (very negative consequences) to +5 (very positive consequences). In the middle of the likert scale is 0, which signifies a neutral outcome of following the stop-technique. 46

57 Chapter 4. Development of Prototype 47 Designing History Module for Self-Reflection on Past Interventions To view assessments in retrospect, users may view them in a History -module (Figure 4.10). This displays all the GSR data that was recorded as line chart visualizations, split at the time of accepting the cue for intervention. Figure 4.10: EDA data and description prior to intervention, followed by EDA data while completing the task-based intervention Designing Tasks for Reducing Emotional and Cognitive Activation In cooperation with a PhD candidate in clinical neuropsychology all text for the application was rewritten. It was a goal that the text should be written in a value-neutral wording, and take care not to invoke any additional self-judgmental thoughts that the user may already be experiencing. This was thought to be decisive for the user experience of this application. In addition to writing new texts we also designed the final flow and presentation of tasks (Figure 4.11). Figure 4.11: Intervention design for supporting emotional and cognitive control. While retaining some characteristics of the material card design pattern detailed in subsection 4.3.3, the cards were redesigned to fill the screen layout, and not allow scrolling, as critiqued in the previous iteration s evaluation. This way a static sequence of tasks was achieved, while keeping some aspects of the original card design that was originally well received in the walkthrough evaluation. 47

58 48 Chapter 4. Development of Prototype A welcome-screen for first-time users was written to introduce users, that may have no prior experience of using self-help techniques for relaxation, to the stop-method (Figure 4.12). Figure 4.12: Description of TimeOut for first-time users Improving Algorithm for Pervasive Affective Sensing This iteration introduced sensor fusion to the algorithmic approach to affective sensing. By adding accelerometer data into the algorithm it was possible to filter GSR values for vigorous hand shaking or general physical movement. Furthermore, by adding temperature data the application was able to filter and alert the user if the wristband had a loose connection to the wrist, or was completely removed. Algorithm 3 Movement and temperature-filtering affective sensing algorithm. function Find GSR change(gsrvalues, movementdata, tempereaturedata) Called every 10 s gsrvalues remove all occurrences of movementdata Filters GSR values that were recorded during sharp hand movement. gsrvalues remove all occurrences of temperaturedata Filters GSR values recorded when temperature sensor recorded less than 30 C. For removal, objects stored in gsrvalues are compared with movementdata, temperaturedata on timestamp. newgsr average of all gsrvalues processedvalues add newgsr basel inegsr Find baseline(processedvalues) percentagechange percentage change between baselinegsr and newgsr Average of 40 GSR values spanning 10 seconds. Calls function Find baseline below. E.g. 40% return percentagechange 48

59 Chapter 4. Development of Prototype 49 The return value of Find GSR change is used to control both the visualization of affective state and the cue for intervention. Thresholds for changes in visualization coloring and intervention cues were set. That is, if the average GSR reading surpassed a threshold of 80% above baseline (see pseudocode below), a cue for intervention is issued to the user. The Find baseline algorithm that was presented and evaluated in subsection was improved by performing a moving average. In my implementation of this statistical analysis calculation, a set of previously processed GSR values (the above pseudocode s variable processedvalues) is passed through an algorithm that averages every five consecutive data points of the set ordered by time. These five GSR values represent 50 seconds of EDA monitoring. Each of these returned averages are compared to find the lowest average, which results in a baseline. Algorithm 4 Baseline by moving average. function Find baseline(processedvalues) basel inegsr Called by Find GSR change every 10 s Initialize return variable and comparator. mov ingaverageal gorithm new moving average processor for all gsrvalue s in processedvalues do mov ingaverageal gorithm(gsrvalue) candidatebasel ine mov ingaverageal gorithm.getaverage() end for if candidatebasel ine < basel inegsr then basel inegsr candidatebasel ine return basel inegsr The lowest consecutive 50 seconds of GSR data set as new baseline Tests showed that computation time for the implementation of Algorithm 3 and 4 when run on smartphones averaged at 2.5 milliseconds, even for six hours of GSR data (size of processedvalues > 2000) Implementing a Fully Functional Prototype A fully functional prototype was programmed in Java programming language for the Android platform. Some initial implementation work had been done in the second and third iteration for learning how to use the Empatica E4 Software Development Kit (SDK). In this iteration I rewrote the whole application, and achieved a better baseline detection algorithm as detailed in the above section. The implementation effort was inspired and built upon ideals of loose coupling and high cohesion of software modules. Some rewrites of the code base were necessary throughout the implementation phase to reach the goals of loose coupling and high cohesion. This greatly affected the usability and likely the user experience of TimeOut in positive terms. 49

60 50 Chapter 4. Development of Prototype For example, by implementing the Empatica E4 sensor service in a separate module abstracted from the presentation layer, it was possible to access the wristband from any part of the application which made monitoring the user while completing tasks possible. Libraries Some libraries were used in addition to Google s official Material Design libraries and Empatica s SDK library. These were discrete-seekbar (Claramunt, 2014) for likert scales, MPAndroidChart (Jahoda, 2016) for line chart visualizations, and AppIntro (Quebe, 2016) for the first-time user welcome screen. Furthermore, the moving average algorithm was based on Rosetta Code s implementation of this classic statistical analysis technique (2017) Evaluation A technology probe-inspired evaluation was planned and carried out. Due to the extensiveness of this evaluation the procedure, results, and discussion of results are detailed in chapter 5 and chapter Summary of Development Chapter This chapter described the process of designing and developing TimeOut as a pervasive affective sensing system for monitoring and intervening to support adults with ADHD s emotional and cognitive control. By using methods of HCI research, interaction design practice and programming patterns, a fully functional high-fidelity prototype was developed. The design process involved domain experts of neuropsychology, expert users with ADHD and student participants of a stress experiment. Following this chapter, a final evaluation in the form of a field-trial is documented and discussed in regards to this study s research questions. 50

61 Chapter 5 Evaluation Having constructed TimeOut as a prototype for pervasive affective sensing of a user s emotional and cognitive activation, it was necessary to evaluate the precision of TimeOut s affective sensing, some potential and the experience of use An in-the-wild field trial inspired by technology probes was conducted to research these questions. This chapter describes the method of evaluation, followed by a presentation of the results. The chapter is concluded by a discussion of the field trials limitations. 51

62 52 Chapter 5. Evaluation 5.1 Method To evaluate some real-world experiences of using TimeOut an in the wild field trial inspired by technology probes was conducted (see section 3.5 for a discussion of field trials and technology probes). Six participants were recruited to participate in the field trial, four of which were students and two held full-time jobs. Each participant was instructed to use TimeOut for two working days at their jobs or place of study. The participants were instructed to use the application and wristband as much as they could, but at the same time restrict their use if the implications of use were found to be too distracting in their work. The participants were told that TimeOut is an application for aiding self-regulation of emotions and cognitions, e.g. mental stress or distracting thoughts. They did not receive any training in performing meditative breathing exercises or any related mindfulness-inspired practices such as body scanning. One exception to this is participant #3 who was particularly confused by the breathing exercise. This participant was given an extended instructional video on body scanning in-between the first and second trial day to help the participant s understanding of controlled breathing as a technique for regaining control of activation. Furthermore, participants were encouraged to be critical of TimeOut. This instruction was an effort to minimize demand characteristics (see section 3.5) and to inspire both creative and critical thoughts on the use and interpretations of TimeOut s design. All participants were instructed to use the device solely for running TimeOut while they were participating. This was due to limitations of the current implementation. TimeOut is currently not fully functional when running in the background (e.g. while using a different application on the device). Two Empatica E4 wristbands and one Samsung Galaxy Android phone were acquired for the field trial. A copy of TimeOut was installed on the phones of those participants who had an Android phone. Consequentially, the field trial could be carried out in pairs of participants using TimeOut simultaneously. As the purpose of this evaluation was mainly about the experience of use and the verification of its ability to detect increases in activation, usability errors, and causes of application crashes were fixed in the implementation of TimeOut between each participant as they discovered errors. After each participant had completed their trial of TimeOut, user data was collected from the phones in addition to a debriefing focusing on the participants experiences through a semi-structured interview. As the participant answered questions and elaborated on ex- 52

63 Chapter 5. Evaluation 53 periences of use, the interview was written on a computer as close to verbatim as possible. The interview results are translated into English and hence further distorted from their original wording. Care was taken in the translation to preserve the participant s original intention as perceived at the time of interview Presentation of Results In the next sections of the evaluation chapter, the data gathered from the field trial are presented and analyzed. Definitions In the context of this field trial, a false positive is defined as a cue for intervention where the participant indicated that the affective sensing system did not reflect their personal assessment of activation level. A positive intervention is defined as when the participant reported that the affective sensing was in line with their appraisal of the situation. A cue for intervention is the incident where TimeOut asks the user whether to stop by displaying a dialog overlaying the interface. 5.2 Analysis of User-Generated Data TimeOut lets users save assessments of their experience at the end of each intervention. The application stores sensor data recorded prior to intervention, sensor data recorded during an intervention, a description of their experience prior to receiving a cue for intervention, and some metrics on their experience of activation, distress and benefit of being intervened. These data are stored in a structured database, which the user can retrieve in a history module. The data presented in the following sections are extracted from each participant s collection of assessments as viewed in the history module (see section 4.4 for a thorough description of TimeOut s assessment module and history module) Accuracy of Affective Sensing When comparing the assessments of all participants as presented in Table 5.1, there is a large variation between subjects in how accurate TimeOut was at intervening in moments of distracted thoughts, distress, or other factors that could obstruct their work. 53

64 54 Chapter 5. Evaluation Table 5.1: Quantitative results of field trial. Participant Hours of use Positive intervention False positives Accuracy # % # % # % # % # % # % The collected sessions are distinguished by a number that represents each participant (#1, #2, #3, and so forth) and detailed by hours of use, number of positive interventions, and occurrences of false positives. Some participants had no positive interventions. Participant #6 decided to conclude its contribution to the field trial prematurely. Participant #4 had no cues for interventions at suitable moments but had some rewarding experiences with the continuous biofeedback which are presented in subsection below where participants use of TimeOut is further analyzed. Some assessments were left out when analyzing user-generated data. Participant #3 had some technical issues related to improper use of the wristband which led to several false positives. These issues were quickly sorted out and are not included in the above table as they do not reflect real-world use, rather inadequate field trial instructions Possible Causes of False Positives When looking closer at each participant s collection of recorded interventions, it is possible to discern some plausible causes for their false positives. Participant Table 5.2: Categorization of each participants false positives Behavioral factors Physical movement Focused work Contextual or unknown Social interaction Unknown cause # # #3 1 1 #4 1 2 #5 2 # Table 5.2 presents a categorization of the cause for each false positive as interpreted from the user-generated data. False positives are elicited from the participants self-reported 54

65 Chapter 5. Evaluation 55 assessments of each intervention. These false positives are then categorized in the following order: Behavioral factors: Actions by the participant which led to a cue for intervention. Physical movement: Self-reported incidents of physical movements such as walking, rising from a chair, yawning and so forth. Focused work: When the participant reports working constructively in a state of flow without feeling any distress or other negative affects. Contextual factors: Incidents in the participant s environment that led to a cue for intervention. Social interaction: Arousal from activities such as engaged conversation, humorous amusement, or other people entering the room, which the participant did not regard as a valid reason for intervention. Unknown cause: Recorded interventions where the participant did not state any behavioral or contextual cause for the false positive. E.g. no description with the value 1 on 0 (where applicable) on a likert scales, or a short description such as false alarm. By categorizing the false positives reported in this field trial it is possible to identify some of TimeOut s weaknesses. Firstly, it is apparent from Table 5.2 that the algorithm for affective sensing does not properly filter for movement. Secondly, the algorithm is not designed to discern between positive and negative outcomes of heightened activation, although that is evidently necessary. Lastly, some participants have substantially more false positives than others. Refer to section 6.1 on page 64 for a discussion of these weaknesses. 5.3 Analysis of User Experience The following section gives an account of the participants experiences of using TimeOut, categorized by different aspects of the design Synchronous Visualizations of EDA TimeOut provides a continuous biofeedback to the user while monitoring. Changes in EDA sensor output are visualized by presenting a circular graphical element that gradually transforms the nuances of its color between green, yellow and red (Figure 5.1), shown on the phone running TimeOut s screen. This visualization is changed by TimeOut s algorithm for affective sensing. While the user s 55

66 56 Chapter 5. Evaluation Figure 5.1: The different nuances of colors green, yellow and red visualize the user s current EDA level. (See subsection for further reference) EDA output is at or below the computed baseline, the top left color of Figure 5.1 is shown. As the user s EDA output gradually increases, the color changes by increments of nuances until the color in the bottom right of Figure 5.1 is shown. Participants were not told the details of how and why the visualization changes, only that they were to keep the screen active. This section presents an analysis of the field trial s participants interpretation of TimeOut s continuous biofeedback. Adjusting Behavior According to the Color Nuances All participants reported that they quickly got the sense that TimeOut s visualizations were attempting to reflect their affective state. Several participants tried to adjust their behavior to keep the visualization at a stable hue, preferably at a nuance of green. Contemplating on how the constant monitoring affected behavior, participant #3 said I have tried to adjust my behavior in a way that keeps the color nuance relatively unchanged. [...] I felt constantly reminded [by the visualization] to act constructively, and that was a very positive experience. Even though the visualization was colored yellow most of the time, the participant experienced positive coping: I experienced some control of my thoughts by keeping the color at the same nuance. For the second participant, the changing nuances incited introspective thought that reportedly led to self-awareness, saying each time it [the visualization] turned yellow I tried to control my breathing to change it back to green. It did not always work as I intended. It did make me curious, however, about what happened in my body. One participant in particular had an experience of mastery related to the visualization. Interpreting the colors metaphorically as traffic signals, participant #4 said that in many cases, the colors were reliable [in reflecting affective states]. In the case of a sudden change to yellow, the participant would adjust the problem-solving approach: I would be unsure of what to write next [for a university assignment,] which I presumed caused the yellow color, and would then try to find a new way to approach the issue at hand. 56

67 Chapter 5. Evaluation 57 According to Table 5.1, this particular participant used TimeOut for the longest period of all participants, yet had no positive interventions. Regardless, the participant reported positive experiences of using TimeOut due to the behavior change incited by following the continuous visualization: By adjusting my problem-solving approach I would sometimes successfully transform the visualization s nuance to a green hue. The accomplishment gave a relief. Critical Remarks on the Visualization Some participants were not as impressed with the continuous visualizations. Participant #1 thought the changing nuances were arbitrary. Although at first, the participant felt that I observed myself to a larger degree than I normally would, but then this effect on selfawareness disappeared as the participant felt no correlation between their affective state and the visualization presented. Participant #5 had similar experiences: I did not relate much to the colors, I would see a yellow color and then I would think I am not feeling stressed, so why is it yellow?. Distractions from Viewing the Continuous Visualization Because the participants were instructed to keep TimeOut active on their phone while participating, the visualization was constantly visible. Participant #5, who was not convinced of the visualization s accuracy of affective states, was also distracted by it: I am easily distracted. To fix that I would turn the phone around, so the screen faced the table Positive Interventions When the affective sensing algorithm detects EDA levels that are 80% above baseline, a cue for intervention is issued (Figure 5.2). If the user accepts, some tasks are presented that the user is meant to complete. Whenever a false positive occurred, the participants were to either dismiss the cue, or quickly browse through the tasks and save an assessment that states the occurrence of a false positive. If the cue felt warranted, the participants would complete the tasks including a breathing exercise as instructed. Figure 5.2: A dialog presented to the user represents a cue for intervention. 57

68 58 Chapter 5. Evaluation Constructive Effects of TimeOut s Interventions Four out of six participants reported constructive effects of the intervention cued by Time- Out (Table 5.1). In this section two reports of positive interventions by participant #1 and #2 are presented and analyzed. (a) Realized that I had to rewrite my text. (b) Participant showing a decrease in EDA while being intervened. Figure 5.3: One of the participant #1 s constructive interventions. Participant #1 had a cue for intervention when realizing that a large part of the participant s thesis needed rewriting (Figure 5.3). Notice that the participant evaluated the experience of stopping to a 4 ( Nytte av stopp on a scale between -4 and 5), which would signify that the participant experienced a great benefit of the intervention. Participant #2 had cues for intervention that were correctly aligned with the participant s self-reported affective state: I was mad at a computer program for statistical analysis. Stopping and following a breathing exercise gave me some needed distance from my anger and helped me to focus on the problem in a more constructive manner (Figure 5.4). (a) Still frustrated at [programming with] R. (b) Decreasing EDA output while TimeOut is intervening. Figure 5.4: Participant #2 being intervened at a moment of frustration. Furthermore, participant #2 explained that each time I received an intervention that I felt was warranted, they were received at moments where I experienced very specific problems. Comparing Figure 5.4 with the common psychophysiological characteristics of EDA signals (see subsection 2.5.1), it is reasonable to assume that TimeOut intervened when participant #2 had phasic changes in EDA output. Drawing on this, one could speculate if the 58

69 Chapter 5. Evaluation 59 decrease in EDA output shown in Figure 5.4b is a natural component of a phasic change in EDA. Further research in a controlled environment is necessary to affirm this speculation. Learning How to Stop and Breathe At the first day of taking part in TimeOut s field trial, participant #3 felt no positive effect of the interventions. Reflecting on instructions given in the breathing exercise I would stop and focus on my breath, but my mind would be occupied with thoughts like what does it mean to feel my breath inside my body?. Although the participant thought it was beneficial to take a break by stopping, the participant felt that as soon as work was resumed the participant would feel stressed again. Figure 5.5: Participant #3 s decreased skin conductance while performing breathing exercise at day two of participation. Participant #3 received some homework, a 10 minute long YouTube video giving a thorough mindfulness breathing exercise. This had some effect (Figure 5.5) for the participant: On the second day of using the application, I had had some training in focusing on my breathing. I was more relaxed while completing the exercise, and I think my overall experience of stopping was much more positive than the first day. Analyzing participant #3 s experience of achieving a better effect of TimeOut s intervention on the second day of the trial after receiving homework on meditation, an implication for future designs are extracted. TimeOut s intervention may be more effective for users who have prior experience in following breathing exercises, or perhaps an embodied perception of affect. To accommodate for this implication in future iterations of TimeOut, the breathing exercise could be designed to be interactive, so as to comply with the needs of expert and novice meditation practitioners alike. At the moment the only possible interaction while listening to the breathing exercise is to cancel it. 59

70 60 Chapter 5. Evaluation Performing the Task-Based Intervention The breathing exercise was the best received task of the five tasks given at intervention. Quickly summarized, these are the tasks that TimeOut issue to its user: Stop, breathe, reflect, assess, continue work. For a demonstration of the design of these tasks, see subsection and Figure 4.11 on page 47. As these tasks were designed especially for the benefit of adults with ADHD, most participants explained that some of the tasks given at the time of intervention did not speak to them. Citing participant #1, Tasks stop and reflect made no sense to me. This does not concern me, I thought, as I was well aware of what I was doing, and what I intended to do. [...] I already had a good sense of what my goals were. Participants #1, #2, and #5 had extensive training in mindful meditation prior to their participation in this field trial. For them, the audio playback of the exercise functioned as cues. I used it as a time reference for when to start and stop my controlled breathing, said participant #1. A common complaint from the participants was that they had no feedback from the application of when the breathing exercise ended. Several participants concluded their breathing exercises prematurely because they thought the exercise had ended. The exercise had some 15 and 30 seconds silent breaks which caused the confusion Interruptions from using TimeOut As TimeOut notifies the user of a pending intervention by invoking the vibration motor of the phone, using TimeOut can be experienced as invasive in a work context. As previously discussed, TimeOut s algorithm for affective sensing is not perfectly accurate, and this may lead to false positives that can be distracting. Sitting in a room with other people concentrating on work, participant #3 explained that each time the phone vibrated [due to a cue for intervention] I thought someone could be distracted. The second day I chose a room where I would be alone, as a consequence of this. There I would not have to think about how the phone vibrations affected my colleagues. Participant #6, in particular, had annoying experiences using TimeOut: It [the application] never got it right. Every time I would move around, or speak to somebody, it would interfere. After two hours of use and ten cues for intervention, participant #6 chose not to participate any further in the field-trial in order to stay focused at work. 60

71 Chapter 5. Evaluation History of Previous Interventions Regarding the usage of TimeOut s history module for viewing generated data of past monitoring and interventions, participant #1 expressed interest in this functionality: I tried to recognize the graph with my memory of the actual course of events. [...] It was interesting to see how the graph depicted my performance on the breathing exercise. The same participant was confused when comparing the graph for monitoring before a stop, and graph for monitoring after stop. Participant #3 had a similar experience, The graph showed that my skin conductance levels had dramatically risen while performing the breathing exercise. I was surprised because I experienced great relief from performing the stop sequence. In reality, the GSR levels did not have a dramatic increase, but the reference values of the graph makes it appear that way (Figure 5.6). (a) Skin conductance level prior to intervention. (b) Observe the difference in reference values of y-axis. Figure 5.6: Vague presentation of skin conductance levels after stopping. Other participants had not viewed their history, with participant #5 reporting I never thought of that. Perhaps if I used the application for a longer period, I would be interested in viewing my history. Generally, this was the case for other participants as well Technical Issues Since TimeOut was not extensively tested before being issued to participants as a technology probe, some issues concerning stability were encountered in the field trial. The first and second participant reported several crashes in different parts of the application. Between these and the following participants, efforts were made to resolve these errors in the implementation. The remaining participants had no reports of unexpected crashes, and one participant stated that it was easier to connect the wristband than most other Bluetooth devices. Some participants reported that they were not aware that the connection to the wearable 61

72 62 Chapter 5. Evaluation automatically disconnects when an assessment is saved. Participant #2 spent some time to figure out that it was necessary to manually reconnect to the wristband. Commenting on the experience of using the Empatica E4 wearable, participant #2 experienced some uncomfortable rubbing from wearing the wristband. The uncomfortable rubbing was irritating to me, and made me overly focused on my wrist, the participant added. Participant #3 was conscious not to rest the wrist on the tabletop as the participant thought that it would alter the sensor readings. Other participants did not report any discomfort of wearing the wristband. Some participants had to charge their wristband but expressed no frustration regarding this. 5.4 Limitations There are limitations to any research, also in applied sciences, and this field trial is no exception. While the field trial did produce valuable insights on how TimeOut was used, there are some limitations that are discussed below. Although TimeOut was designed to cater for some special needs of adults with ADHD, none of the participants were diagnosed with ADHD or any similar neurodevelopmental disorder. In addition to this, the task-based design is adapted from an ongoing study of customizing Goal Management Training (see subsection 2.4.3) for non-pharmacological treatment of adults with ADHD, and as such the design may require some training to be utilized in a proper way. Furthermore, the duration of each participant s use of TimeOut is a limiting factor of this field trial. As the prototype was used for two days or less by the participants, some characteristics of use may not yet have been developed and thus be explored in this field trial. One example of this is the history module, which most participants had yet to discover. Furthermore, the participants positive effect of using TimeOut s breathing exercise could be improved by using the application for a longer period. A prolonged field trial (2-4 weeks) could be a more suitable duration for a future field trial of TimeOut. 5.5 Summary of Evaluation Chapter This chapter described the evaluation of TimeOut by presenting the field trial method and results detailing the field trial s participants experiences of using TimeOut. 62

73 Chapter 6 Discussion Following the design, implementation and field trial of TimeOut, this chapter provides a critical discussion of this study s overreaching research question: How can we design interventions supporting emotional and cognitive control in adults with ADHD? To answer this research question, a prototype, TimeOut, was designed and implemented iteratively as documented in chapter 4 inspired by the perspective of affective interaction and other HCI research as reviewed in chapter 2. The prototype was constructed using methods presented in chapter 3. After implementing the final prototype of TimeOut, a field trial was conducted and presented in chapter 5 to evaluate the characteristics of use and the experience of six participants using TimeOut. This chapter presents a discussion of the empirical findings from the field trial of TimeOut and the construction of TimeOut as a pervasive affective sensing system for the support of adults with ADHD s emotional and cognitive control. In the following sections, the discussions are categorized by features of TimeOut s design, each section ending with some implications for designing for supporting emotional and cognitive control and future directions for TimeOut, where applicable. The chapter is concluded with a summary of the design implications elicited from the presented discussions where each of these implications deliberately address aspects of the research question for this study. 63

74 64 Chapter 6. Discussion 6.1 Accuracy of Affective Sensing Algorithm Conducting the field trial was a chance to test the affective sensing algorithm of TimeOut applied in a real-world context. By assessing the participant-generated data presented in Table 5.1, Table 5.2, and the participants accounts recited in section 5.3 some characteristics of its accuracy have surfaced. In general, TimeOut currently works best for users who are occupied with cognitively demanding work that does not involve too much social interaction with colleagues. For some participants in the previously stated context, TimeOut had a convincing accuracy at detecting affective states that negatively impacted the participants work. Participants #2 and #3 had the highest number of positive interventions, although participant #2 had issues of bodily movement leading to eight false positives. The data collected from the field trial does not indicate that participants experienced that TimeOut missed some incidents of loss of emotional and cognitive control. Rather, participants found that TimeOut was overly sensitive to their affective states and gives cues for intervention at moments where the participants did not find this suitable. The effect of these false positives were found to be distractive rather than ensuring safety, as the participants were interrupted in their work by the false positives. Although TimeOut has shown potential in correctly sensing its user s heightened emotional and cognitive activation, the false positives presented in Table 5.2 and analyzed in subsection shows that it is apparent that TimeOut s algorithm for affective sensing has potential for improvement. Figure 6.1: TimeOut mistakenly intervening a participant who experienced a moment of amusement. Even though the algorithm was originally designed to filter for contextual factors such as body movement, 12 false positives across all participants were caused by behavior such as walking, rising from a chair and yawning with arms stretched out (e.g. as seen in Fig- 64

75 Chapter 6. Discussion 65 ure 6.1). These observations indicate that the algorithm for deriving movement from accelerometer data should be further developed to better filter for these kinds of physical behavior. Further development should be aided by experiments in controlled environments to assess the effectiveness of the movement filtering. While the dialog that displayed cues for intervention was designed to be dismissible (see Figure 5.2 on page 57), the affective sensing algorithm did not account for this user action. The application would wait for 60 seconds before reassessing the subject s generated sensor data. Because the baseline would be left unchanged, the user would most likely receive a new cue for intervention and therefore be forced to ignore the cues further, accept the cue for intervention, or exit the application. Furthermore, several participants complained that the visualizations turned to dark hues of yellow although they did not feel anxious, unfocused or stressed. This indicates that the moving average baseline algorithm (see pseudocode on page 49) may need some further research to better find a baseline of the subject s skin conductance Design Implications Following the above discussion of TimeOut s accuracy of affective sensing, a design implication on the costs of false positives in systems for supporting emotional and cognitive control is discussed. When designing pervasive affective sensing systems for supporting emotional and cognitive control, it is important to take precautions regarding the costs of giving users false positives. Emotional and cognitive control is not a matter of life and death, as is the case with systems for detecting epileptic seizures (Poh et al., 2012). For these life-critical systems, it is better to issue an extra possibly false positive than risk missing a critical incident that may, in the extreme consequence, result in sudden unexpected death. In the case of TimeOut, receiving numerous false positives was found to be distracting to its users, rather than critical to assure their well-being. It is argued here that when designing interventions for emotional and cognitive control, designers should take precautions not to load the user with unnecessary cues for interventions as these can be interruptive to the user. This implication for design is in line with related work presented in subsection In their principles for designing for children with ADHD Sonne, Marshall, et al. (2016) and Weisberg et al. (2014) argue that designs should avoid intrusion and take care to limit distractions to prevent the user from losing attention. 65

76 66 Chapter 6. Discussion Potential for Improvements of TimeOut Although TimeOut was shown to be beneficial for some participants in their given context, future development of TimeOut should improve the algorithm for affective sensing. Firstly, the algorithm for deriving body movement from accelerometer data output from Empatica E4 wristband must be improved as the field trials indicate that bodily movement is not always detected properly. Further, the method for filtering EDA data concurring with movement data should be improved. Since arousal from physical movement often affects EDA with a short delay, it should be possible to add some delay to the filtering. Perhaps it is also possible to normalize EDA data which is recorded after body movement, as a means for filtering disturbances in the physiological data. Secondly, baseline analysis of EDA data could be improved to better accommodate for natural variances in EDA. Tonic changes are a component of EDA. EDA varies in intensity throughout the day independent of external stimulus. Therefore, any user will eventually receive a cue for intervention by TimeOut by tonic changes in EDA data. As the current baseline algorithm is designed to look for the longest period of low EDA output in its subject, the slowly increasing EDA will eventually be recognized as an event of heightened activation in the user by TimeOut. Lastly, a completely redesigned approach to computing baseline and detecting heightened activation could be suitable for TimeOut. The current implementation uses the traditional statistical method of simple moving average. Machine learning approaches to data analysis could be more appropriate, as it is possible to train such machine learning algorithms to model user s biosensor patterns for better detection of abnormal biosensor data. 6.2 Synchronous Visualizations of EDA: The Affective Loop While monitoring the user s EDA, TimeOut visualizes changes in EDA continuously relative to a computed baseline (see Figure 5.1 on page 56). This thesis presents users engagement with these visualizations as an affective loop as defined by Sundström et al. (2007) (see subsection for further reference). The rationale for designing a synchronous affective loop in TimeOut was the objective of providing the user with interpretive flexibility (see subsection 2.2.2). The purpose of designing for interpretive flexibility in the case of TimeOut is to allow the user to make his or her interpretations of their affective states and consequently facilitate introspective reflection (see subsection for further reference). Participants #2, #3 and #4 had rewarding experiences of TimeOut s continuous biofeed- 66

77 Chapter 6. Discussion 67 back. In an attempt to control the changing color nuances of the visualization, they would adjust their behavior. E.g, by changing their approach to a work-related problem or by controlling their breathing which would lower sympathetic nervous system activation and thus EDA output. It was reported by the participants that this gave a sense of self-awareness, and that they would often succeed in turning the visualization s color to a greener hue, or keep it at a stable nuance. Others (participant #1 and #5) were not convinced by the continuous visualization, arguing that even though the color changed to yellow, which they interpreted as a detection of their heightened activation, they did not feel particularly activated. Consequentially, participant #5 felt distracted by TimeOut s visualization and would turn the device around so it would face the table and not distract the participant. These findings show that continuous biofeedback in the form of a visualization may be a valuable part of TimeOut s design. As explained, some of the participants succeeded in changing the hue or keeping the visualization s color at the same nuance. In effect showing that TimeOut successfully lead to introspection and an experience of better problem solving, or lowering one s activation by controlled breathing, is a success of TimeOut. Regardless of these positive effects of TimeOut s continuous visualizations, some participants had negative experiences of annoyance or interruption as discussed above. Some design implications are elicited from the presented discussion in the next section Design Implications While some of the participants did experience a sense of introspective reflection by engaging with the synchronous affective loop of TimeOut, others quickly disregarded the continuous visualizations as they felt it did not accurately represent their experienced activation. Höök et al. (2008) argue that affective loops are not designed to infer users emotions, but involve users in emotional interactional process (p. 654) and let the user choose to be involved or not (Höök et al., 2008, p. 654). TimeOut s continuous visualization does infer the users affective states by representing changes in EDA with different colors. TimeOut, similarly to Höök et al. (2008) s designs, generate affective expressions (in TimeOut s case color nuances between green, yellow, and red). The users of TimeOut are free to interpret these affective expressions as they see fit. Wagenknecht (2017) argue that the transformation of biosensor data to familiar, yet ambiguous, affective expressions are a matter of extrospection rather than introspection. These affective expressions are experienced by the user as an outside phenomenon, which the 67

78 68 Chapter 6. Discussion user further interprets. TimeOut s field trial participants engagement with and rejection of the synchronous visualizations may be interpreted as the participants trust in and curiosity about the design. The participants that properly engaged in an affective loop with TimeOut also seemed the most optimistic about TimeOut s ability to sense their affective states. Users engagement with affective loops seem to be contingent on the users trust in and curiosity about the design. Therefore, I would argue that to design engaging affective loop experiences designers should carefully consider the ambiguity of the affective expressions used to incite the affective loop Potential for Improvement of TimeOut As discussed, some of the participants were not convinced by the continuous biofeedback. To accommodate for users that felt estranged by the changing colors, the current algorithm could be expanded to allow for user input in addition to the continuous EDA monitoring. By allowing users to give feedback to TimeOut s visualization, the algorithm could adjust its thresholds for changing color nuance, or perhaps disregard previous sensor data and compute a new baseline. Feedback could be provided to TimeOut in a number of ways. Firstly, a simple touch by the user on the graphical visualization could signal the algorithm to compute a new baseline. Secondly, the user could report its perceived affective state through iconography. This iconographic input would allow the user to not only correct TimeOut in the case where the affective sensing overestimates the user s level of activation, but also in the case where a user feels more activated than what is implied by the visualization. Lastly, tangible interaction such as the Grasp platform from Bryggen Research (Guribye, Gjøsæter, & Bjartli, 2016) could be utilized to provide feedback to TimeOut from the user. To minimize the interruptions experienced when using TimeOut, the design could be moved to a wearable (e.g. a wristband). Designing for wearables opens up for a wider array of feedback modalities, such as tactile feedback. Changes in EDA could be communicated to the user in vibrotactile notifications as a less intrusive sensory feedback than the current visual modality. 6.3 Task-Based Intervention A central part of TimeOut s design is the task-based intervention that is given when the affective sensing algorithm detects a substantial increase in a subject s skin conductance. 68

79 Chapter 6. Discussion 69 These tasks were designed to be used by adults with ADHD, inspired by an ongoing research on adapting Goal Mangement Training (GMT, see subsection 2.4.3) for supporting adults with ADHD s cognitive and emotional control. The presented field trial did not include any adults with ADHD. As TimeOut is a novel, experimental design it was not suitable to include participants with special needs for practical and ethical reasons. Participants of this evaluation were not too involved with completing the tasks at intervention in a step-by-step fashion. As one participant commented, it did not speak to him or her as they were well aware of what their objectives of the day were, and how to reach those objectives. As TimeOut to a large extent emulates the same steps for stopping as given in the ongoing research project on adapting GMT for adults with ADHD s emotional and cognitive control, it would likely require some training for a participant to follow the task-based intervention in a step-by-step manner. GMT treatment is usually given in group-therapy, where the same tasks as TimeOut presents are given as homework for the patients to complete in their daily lives. For future research, it would be necessary to include TimeOut as a part of GMT-based treatment to research how a GMT-based intervention is used and experienced. 6.4 Retrospective of Past Interventions TimeOut lets users view their past interventions in a history module. This module shows the sensor data recorded prior to intervention, and while intervening. The data is annotated by the user at the end of each intervention with metrics on their experience of activation, an indication of their distraction due to activation, and an indication of their experience from completing the tasks given at intervention. Lastly, a description is given by the user to describe their experience prior to being intervened by TimeOut. The history module was designed with the rationale of giving the user some ownership of the recorded sensor data. Participants #1 and #3 were curious of their past interventions. They reported that they tried to correlate the graphical representation of EDA with their recollection of the course of events before their interventions, sometimes leading them to believe they had recognized a pattern. Participant #3 was confused when the history module gave the impression that EDA values had risen dramatically while the participant completed the controlled breathing exercise (see Figure 5.6 on page 61). In reality there was no significant rise in 69

80 70 Chapter 6. Discussion EDA while the participant was following the breathing exercise, it was the line chart visualization that gave that impression. On the other hand, none of the remaining four participants viewed their past interventions in the retrospective history module. Some commented that if they had used TimeOut for a prolonged period they would eventually want to investigate their previous interventions. Further evaluation is needed to properly assess the experiences of using the retrospective history module. 6.5 Designing Interventions for Supporting Emotional and Cognitive Control From what was found in the evaluation of TimeOut (chapter 5) and the discussion presented in this chapter, we can argue that pervasive affective sensing systems may prompt two distinct types of interventions: data-driven and user-initiated. Data-driven intervention: Prompt for a behavioral or cognitive activity issued by the pervasive affective sensing system as a result of some specific data generated by the user. For example, a task-based intervention based on changes in the user s EDA level (see section 6.3). User-initiated intervention: A user s self-instigated behavioral or cognitive action prompted by the user s interpretation of an interaction with the pervasive affective sensing system. For example, when a user changes their problem-solving approach as a reaction to the system s affective expression (see section 6.2). This thesis argue that when designing pervasive affective sensing systems using design principles of affective interaction (see subsection 2.2.1), the user is empowered to initiate its own interventions. These user-initiated interventions are not defined by the system; it is a result of interacting with affective expressions of the system. In this type of intervention, the user intervenes in his or her thoughts or actions with self-defined measures. 6.6 TimeOut s Effect on Emotional and Cognitive Control The design of TimeOut was an effort to support adults with ADHD s emotional and cognitive control. While the field trial did not include any participants diagnosed with ADHD, the iterative design process of creating TimeOut was informed by neuropsychology experts and adults with ADHD. Therefore, this study cannot attest to TimeOut s effectiveness of 70

81 Chapter 6. Discussion 71 supporting adults with ADHD s emotional and cognitive control. What is evident from the field trial, though, is the effects TimeOut had on some of the participants emotional and cognitive control. As documented in subsection 5.3.2, four out of six participants had positive experiences of receiving support from TimeOut for their emotional and cognitive control. Furthermore, as previously discussed in this chapter, some participants had rewarding experiences when engaging with the continuous visualizations of their recorded EDA data. 6.7 Summary of Design Implications for Supporting Emotional and Cognitive Control Below are four design implications elicited from this chapter s discussion for future research on how to design for supporting emotional and cognitive control for adults with ADHD. 1. Consider the costs of prompting the user with interventions (subsection 6.1.1). 2. Avoid interactions that may lead the user to lose attention (subsection 6.1.1), as people with ADHD are particularly prone to becoming inattentive (see section 2.4). 3. Include open interpretations of system-generated affective expressions to invite the user to engage in affective loops (subsection 6.2.1). 4. Include health care professionals and people with ADHD in the design process as their expert insights are vital for the success of designs for supporting adults with ADHD (chapter 4). 6.8 Summary of Discussion Chapter This chapter presented a discussion of TimeOut in regards to its design and the experiences six field trial participants had when using the the design. Implications for designing for supporting emotional and cognitive control were discussed in relation to relevant aspects of the design. Four design principles were presented to guide future research on how to design for supporting emotional and cognitive control for adults with ADHD. 71

82 Chapter 7 Conclusion and Future Work The research presented in this thesis has studied how interventions supporting emotional and cognitive control can be designed to help adults with ADHD. The motivation for this research was the objective of supporting novel non-pharmacological interventions for people with ADHD. By using research through design as a research framework, TimeOut was designed as a fully functional pervasive affective sensing system. The system is a mobile application implemented in Java for the Android operating system. A sensor-equipped wristband is used to provide the application with physiological and contextual data. TimeOut interprets the data and issues a cue for intervention when it detects an indication of emotional or cognitive control. A field trial was conducted to explore how TimeOut was used and experienced by six participants. Some participants experienced a benefit in their work from receiving data-driven interventions from TimeOut. Others reported being distracted from their work when receiving interventions that did not feel warranted by the participants. Furthermore, some participants were engaged in affective loops with TimeOut s synchronous visualizations of EDA. This engagement prompted the participants to initiate interventions aimed at modifying their behavior and thought in an attempt to control the synchronous visualization of EDA. The presented research shows that when designing affective sensing systems for contexts that are not life-critical, one should consider the costs of false positives as these may cause interruptions and negative experiences for the user. Furthermore, designing for open interpretations of system-generated affective expressions may invite users of pervasive affective sensing systems to engage in affective loops that could support their self-awareness. Two distinct types of behavioral and cognitive interventions in TimeOut was uncovered: data-driven interventions and user-initiated interventions. These modalities pose differ- 72

83 Chapter 7. Conclusion and Future Work 73 ent sets of challenges in the design of pervasive affective sensing systems. Further research is needed to uncover how we can successfully design data-driven and user-initiated interventions for assistive technology in the context of mental health care. Finally, while there is optimism concerning the potential of utilizing wearable technologies and sensor data for health care purposes, including mental health care, the nature of affect remain a complex and far from being fully understood phenomena. The presented research shows the complexity of achieving useful sensor-driven technologies for clinical contexts. Therefore, this thesis calls for further interdisciplinary efforts between HCI research and clinicians with an emphasis on achieving evidence-based assistive technologies. 7.1 Future Work TimeOut could be implemented in group-based GMT therapy. A central point of GMT is training executive functions; by practicing the stop technique, new skills for maintaining emotional and cognitive control can be achieved. A strength of TimeOut is its ability to cue an intervention in relevant situations, however, as described in chapter 5 and chapter 6, the algorithms for detecting heightened activation are in need of improvement. Another possibility is to omit the sensor monitoring altogether. Cueing of interventions could be designed to be interval-contingent. With this approach, TimeOut could help people with ADHD practice the stop technique they have been thought in group-based therapy, without the complications of using sensor wearables. Furthermore, new modalities for interaction could be explored for TimeOut. In its current state, TimeOutit is designed for Android phones. These devices bring about sources for distractions to it users, such as notifications from social networking applications and so forth. Using wearable devices such as Apple Watch or Empatica s Embrace offer less obtrusive modalities of interaction such as haptic feedback, which could result in a better user experience. 73

84 Bibliography American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (dsm-5 ). American Psychiatric Pub. Barkley, R. A. (1997). Behavioral inhibition, sustained attention, and executive functions: Constructing a unifying theory of ADHD. Psychological Bulletin, 121(1), doi: / Beaton, R., Merkel, R., Prathipati, J., Weckstein, A., & McCrickard, S. (2014). Tracking Mental Engagement: A Tool for Young People with ADD and ADHD. Proceedings of the 16th International ACM SIGACCESS Conference on Computers & Accessibility, doi: / Biederman, J., & Faraone, S. V. (2006, jul). The effects of attention-deficit/hyperactivity disorder on employment and household income. MedGenMed : Medscape general medicine, 8(3), 12. Retrieved from Birkett, M. A. (2011, oct). The Trier Social Stress Test protocol for inducing psychological stress. Journal of visualized experiments : JoVE(56). doi: /3238 Blomquist, Å., & Arvola, M. (2002). Personas in Action: Ethnography in an Interaction Design Team. In Proceedings of the second nordic conference on human-computer interaction - nordichi 02 (pp ). New York, New York, USA: ACM Press. doi: / Bødker, S. (2000, sep). Scenarios in user-centred designfhsetting the stage for reflection and action. Interacting with Computers, 13(1), doi: /S (00) Bødker, S. (2006). When second wave HCI meets third wave challenges. In Proceedings of the 4th nordic conference on human-computer interaction changing roles - nordichi 06 (pp. 1 8). New York, New York, USA: ACM Press. doi: / Bødker, S., Christiansen, E., Nyvang, T., & Zander, P.-O. (2012). Personas, people and participation - challenges from the trenches of local government. In Proceedings of the 12th participatory design conference on research papers: Volume 1 - pdc 12 (pp ). New York, New York, USA: ACM Press. doi: / Boehner, K., DePaula, R., Dourish, P., & Sengers, P. (2005). Affect: from information to interaction. In Proceedings of the 4th decennial conference on critical computing between sense and sensibility - cc 05 (pp ). New York, New York, USA: ACM Press. doi: / Boucsein, W. (2012). Electrodermal Activity. Boston, MA: Springer US. doi: /

85 Bibliography Brown, B., Reeves, S., & Sherwood, S. (2011). Into the wild: Challenges and opportunities for field trial methods. In Proceedings of the 2011 annual conference on human factors in computing systems - chi 11 (pp ). New York, New York, USA: ACM Press. doi: / Buchenau, M., & Suri, J. F. (2000). Experience prototyping. In Proceedings of the conference on designing interactive systems processes, practices, methods, and techniques - dis 00 (pp ). New York, New York, USA: ACM Press. doi: / Carroll, J. M. (2013). Human Computer Interaction - brief intro. The Encyclopedia of Human- Computer Interaction, 2nd Ed., doi: / Chamberlain, A., Crabtree, A., Rodden, T., Jones, M., & Rogers, Y. (2012). Research in the wild: understanding in the wild approaches to design and development. In Proceedings of the designing interactive systems conference on - dis 12 (pp ). New York, New York, USA: ACM Press. doi: / Churchman, C. W. (1967, dec). Wicked Problems. Management Science, 14(4), doi: / mnsc.14.4.b141 Claramunt, G. (2014). Discreteseekbar. GitHub. Cooper, A. (1999). The inmates are running the asylum. Sams. Crabtree, A., Chamberlain, A., Grinter, R. E., Jones, M., Rodden, T., & Rogers, Y. (2013, jul). Introduction to the Special Issue of fhthe Turn to The Wildfh. ACM Transactions on Computer- Human Interaction, 20(3), 1 4. doi: / Cross, N. (1982, oct). Designerly ways of knowing. Design Studies, 3(4), doi: / X(82) Diamond, A. (2014). Executive Functions. Annual review of clinical psychology, 64, Dibia, V. (2016). FOQUS: A Smartwatch Application for Individuals with ADHD and Mental Health Challenges. In Proceedings of the 18th international acm sigaccess conference on computers and accessibility - assets 16 (pp ). New York, New York, USA: ACM Press. doi: / Dodson, W. W. (2005, may). Pharmacotherapy of adult ADHD. Journal of Clinical Psychology, 61(5), doi: /jclp Donker, T., Petrie, K., Proudfoot, J., Clarke, J., Birch, M.-R., & Christensen, H. (2013, nov). Smartphones for smarter delivery of mental health programs: a systematic review. Journal of medical Internet research, 15(11), e247. doi: /jmir.2791 Empatica. (2016). What should i know to use eda data in my experiment. support.empatica.com/hc/en-us/articles/ what-should-i-know -to-use-eda-data-in-my-experiment-. Empatica. Flobak, E., Guribye, F., Jensen, D. A., & Lundervold, A. J. (2017). Designing Data-Driven Interventions for Mental Health Care. In Proceedings of the 11th international conference on pervasive computing technologies for healthcare (pp. 1 4). Gaver, B., Dunne, T., & Pacenti, E. (1999, jan). Design: Cultural probes. interactions, 6(1),

86 76 Bibliography doi: / Go, K., & Carroll, J. M. (2004, nov). The blind men and the elephant. interactions, 11(6), doi: / Google. (2017). Material Design. Retrieved from [ ] Gould, J. D., & Lewis, C. (1985, mar). Designing for usability: key principles and what designers think. Communications of the ACM, 28(3), doi: / Gregory, S. A. (1966). Design and the Design Method. In S. A. Gregory (Ed.), The design method (pp. 3 10). Boston, MA: Springer US. doi: / Groenman, A. P., Oosterlaan, J., Rommelse, N., et al. (2013). Substance use disorders in adolescents with attention deficit hyperactivity disorder: a 4-year follow-up study. Addiction, 108(8), doi: /add Guribye, F., Gjøsæter, T., & Bjartli, C. (2016). Designing for Tangible Affective Interaction. In Proceedings of the 9th nordic conference on human-computer interaction - nordichi 16 (pp. 1 10). New York, New York, USA: ACM Press. doi: / Harstad, E., & Levy, S. (2014). Attention-deficit/hyperactivity disorder and substance abuse. Pediatrics, 134(1), e293 e301. doi: /peds Hassenzahl, M., & Tractinsky, N. (2006, mar). User experience - a research agenda. Behaviour & Information Technology, 25(2), doi: / Helsedirektoratet. (2014, December). Adhd/hyperkinetisk forstyrrelse - nasjonal faglig retningslinje for utredning, behandling og oppfølging: Rett diagnose - individuell behandling. Hodges, L. J., Walker, J., Kleiboer, A. M., Ramirez, A. J., Richardson, A., Velikova, G., & Sharpe, M. (2011). What is a psychological intervention? A metareview and practical proposal (Vol. 20) (No. 5). Hogg, M. A., Abrams, D., & Martin, G. N. (2007). Social cognition and attitudes. In G. N. Martin, N. R. Carlson, & W. Buskist (Eds.), Psychology (pp ). Pearson Education. Retrieved from Höök, K. (2009). Affective loop experiences: designing for interactional embodiment. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 364(1535), Höök, K., Ståhl, A., Jonsson, M., Mercurio, J., Karlsson, A., & Johnson, E.-C. B. (2015, jun). COVER STORYSomaesthetic design. interactions, 22(4), doi: / Höök, K., Ståhl, A., Sundström, P., & Laaksolaahti, J. (2008). Interactional empowerment. In Proceeding of the twenty-sixth annual chi conference on human factors in computing systems - chi 08 (p. 647). New York, New York, USA: ACM Press. doi: / Houde, S., & Hill, C. (1997). What do prototypes prototype? Handbook of Human Computer Interaction, Hutchinson, H., Hansen, H., Roussel, N., Eiderbäck, B., Mackay, W., Westerlund, B.,... Evans, H. (2003). Technology probes. In Proceedings of the conference on human factors in computing systems - chi 03 (pp ). New York, New York, USA: ACM Press. doi: / In de Braek, D. M. J. M., Dijkstra, J. B., Ponds, R. W., & Jolles, J. (2012). Goal Management Training 76

87 Bibliography 77 in Adults With ADHD: An Intervention Study. Journal of Attention Disorders. doi: / INTROMAT. (2017). INTROMAT: INTROducing Mental health through Adaptive Technology. Retrieved from [ ] Jahoda, P. (2016). Mpandroidchart. GitHub. Kanjo, E., Al-Husain, L., & Chamberlain, A. (2015). Emotions in context: examining pervasive affective sensing systems, applications, and analyses. Personal and Ubiquitous Computing, 19(7), doi: /s Kleinginna, P. R., & Kleinginna, A. M. (1981, sep). A categorized list of motivation definitions, with a suggestion for a consensual definition. Motivation and Emotion, 5(3), doi: /BF Kooij, S. J., Bejerot, S., Blackwell, A., et al. (2010). European consensus statement on diagnosis and treatment of adult adhd: The european network adult adhd. BMC Psychiatry, 10(1), doi: / X Lackschewitz, H., Hüther, G., & Kröner-Herwig, B. (2008). Physiological and psychological stress responses in adults with attention-deficit/hyperactivity disorder (ADHD). Psychoneuroendocrinology, 33(5), doi: /j.psyneuen Levine, B., Schweizer, T. A., O Connor, C., Turner, G., Gillingham, S., Stuss, D. T.,... Robertson, I. H. (2011). Rehabilitation of executive functioning in patients with frontal lobe brain damage with goal management training. Frontiers in human neuroscience, 5, 9. doi: /fnhum Lim, C. G., Lee, T. S., Guan, C., & Fung, D. S. S. (2012, oct). A Brain-Computer Interface Based Attention Training Program for Treating Attention Deficit Hyperactivity Disorder. PLoS ONE, 7(10), e doi: /journal.pone MacLean, D., Roseway, A., & Czerwinski, M. (2013). MoodWings: A Wearable Biofeedback Device for Real- Time Stress Intervention. In Proceedings of the 6th international conference on pervasive technologies related to assistive environments - petra 13 (p. 66). New York, New York, USA: ACM Press. doi: / Malenka, R., Nestler, E., & Hyman, S. (2009). Widely Projecting Systems: Monoamines, Acetylcholine, and Orexin. In A. Sydor & R. Brown (Eds.), Molecular neuropharmacology: A foundation for clinical neuroscience (2nd ed., pp ). New York, New York, USA: McGraw- Hill Medical. Marije Boonstra, A., Oosterlaan, J., Sergeant, J. A., & Buitelaar, J. K. (2005, 08). Executive functioning in adult adhd: a meta-analytic review. Psychological medicine, 35(8), (Copyright - Copyright Cambridge University Press, Publishing Division Aug 2005; Document feature - tables; references; Last updated ; CODEN - PSMDCO) Retrieved from McHugh, B., Dawson, N., Scrafton, A., & Asen, E. (2010). Hearts on their sleeves : The use of systemic biofeedback in school settings. Journal of Family Therapy, 32(1), Mohr, D. C., Burns, M. N., Schueller, S. M., Clarke, G., & Klinkman, M. (2013, jul). Behavioral 77

88 78 Bibliography Intervention Technologies: Evidence review and recommendations for future research in mental health. General Hospital Psychiatry, 35(4), doi: /j.genhosppsych National Collaborating Centre for Mental Health (Great Britain). (2009). Attention deficit hyperactivity disorder : diagnosis and management of ADHD in children, young people, and adults. British Psychological Society. Neumann, E., & Blanton, R. (1970, jan). The early history of electrodermal research. Psychophysiology, 6(4), Retrieved from Nielsen, J. (1993). Usability Engineering (Vol. 44) (No. 3). doi: / Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. In Proceedings of the sigchi conference on human factors in computing systems empowering people - chi 90 (pp ). New York, New York, USA: ACM Press. doi: / Nielsen, L., & Storgaard Hansen, K. (2014). Personas is applicable: A Study on the Use of Personas In Denmark. In Proceedings of the 32nd annual acm conference on human factors in computing systems - chi 14 (pp ). New York, New York, USA: ACM Press. doi: / Olson, J. S., & Kellogg, W. A. (2014). Ways of knowing in HCI. Oulasvirta, A., & Hornbæk, K. (2016). HCI Research as Problem-Solving. In Proceedings of the 2016 chi conference on human factors in computing systems - chi 16 (pp ). New York, New York, USA: ACM Press. doi: / Oxford English Dictionary. (2017). intervention, n. Retrieved from [ ] Passer, M., Smith, R., Holt, N., Bremner, A. J., Sutherland, E., & Vliek, M. (2008). Psychology: the science of mind and behaviour (Vol. 39) (No. 8). Retrieved from Picard, R. W. (1997). Affective computing. MIT Press. Picard, R. W. (2010, jul). Emotion Research by the People, for the People. Emotion Review, 2(3), doi: / Picard, R. W., Papert, S., Bender, W., Blumberg, B., Breazeal, C., Cavallo, D.,... Strohecker, C. (2004). Affective Learning - A Manifesto. BT Technology Journal, 22(4), doi: /B:BTTJ Pina, L., Rowan, K., Johns, P., Roseway, A., Hayes, G., & Czerwinski, M. (2014). In Situ Cues for ADHD Parenting Strategies Using Mobile Technology. In Proceedings of the 8th international conference on pervasive computing technologies for healthcare (pp ). ICST. doi: / icst.pervasivehealth Poh, M.-Z., Loddenkemper, T., Reinsberger, C., Swenson, N. C., Goyal, S., Sabtala, M. C.,... Picard, R. W. (2012, may). Convulsive seizure detection using a wrist-worn electrodermal activity and accelerometry biosensor. Epilepsia, 53(5), e93 e97. doi: /j x Pruitt, J., & Grudin, J. (2003). Personas. In Proceedings of the 2003 conference on designing for user experiences - dux 03 (pp. 1 15). New York, New York, USA: ACM Press. doi: /

89 Bibliography 79 Quebe, A. (2016). Appintro. GitHub. Rogers, Y. (2011, jul). Interaction design gone wild: striving for wild theory. interactions, 18(4), doi: / Rogers, Y., Sharp, H., & Preece, J. (2015). Interaction Design: Beyond Human - Computer Interaction (4th ed.). Wiley Publishing. Rosetta Code. (2017). Simple moving average. Retrieved from [ ] }moving{ }average Rosson, M. B., & Carroll, J. M. (2002). Scenario-Based Design. In The human-computer interaction handbook: Fundamentals, evolving technologies and emerging applications (pp ). Lawrence Erlbaum Associates. Rudd, J., Stern, K., & Isensee, S. (1996, jan). Low vs. high-fidelity prototyping debate. interactions, 3(1), doi: / Sanches, P., Höök, K., Vaara, E., Weymann, C., Bylund, M., Ferreira, P.,... Sjölinder, M. (2010). Mind the body! Designing a Mobile Stress Management Application Encouraging Personal Reflection. In Proceedings of the 8th acm conference on designing interactive systems - dis 10 (pp ). New York, New York, USA: ACM Press. doi: / Sefelin, R., Tscheligi, M., & Giller, V. (2003). Paper Prototyping - What is It Good for?: A Comparison of Paper- and Computer-based Low-fidelity Prototyping. In Chi 03 extended abstracts on human factors in computing systems (pp ). New York, New York, USA: ACM Press. doi: / Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological Momentary Assessment. Annual Review of Clinical Psychology, 4(1), doi: /annurev.clinpsy Simon, H. A. (1969). The sciences of the artificial. Cambridge, MA: MIT Press. Solberg, B. S., Haavik, J., & Halmoy, A. (2015, jun). Health Care Services for Adults With ADHD: Patient Satisfaction and the Role of Psycho-Education. Journal of Attention Disorders, doi: / Sonne, T., & Jensen, M. M. (2016). Chillfish: A respiration game for children with adhd. In Proceedings of the tei 16: Tenth international conference on tangible, embedded, and embodied interaction (pp ). New York, NY, USA: ACM. doi: / Sonne, T., Marshall, P., Obel, C., Thomsen, P. H., & Grønbæk, K. (2016). An assistive technology design framework for ADHD. In Proceedings of the 28th australian conference on computerhuman interaction - ozchi 16 (pp ). New York, New York, USA: ACM Press. doi: / Sonne, T., Müller, J., Marshall, P., Obel, C., & Grønbæk, K. (2016). Changing Family Practices with Assistive Technology: MOBERO Improves Morning and Bedtime Routines for Children with ADHD. In Proceedings of the 2016 chi conference on human factors in computing systems - chi 16 (pp ). New York, New York, USA: ACM Press. doi: / Storni, C. (2015, jun). A personal perspective on research through design. interactions, 22(4), doi: / Sundström, P., Ståhl, A., & Höök, K. (2007, apr). In situ informants exploring an emotional mobile messaging system in their everyday practice. International Journal of Human-Computer 79

90 80 Bibliography Studies, 65(4), doi: /j.ijhcs Tao, J., & Tan, T. (2005). Affective computing: A review (Vol LNCS). Springer, Berlin, Heidelberg. doi: / Wagenknecht, S. (2017). The evocative objectfhintrospection and emotional reflection through computer use. Interacting with Computers, 29(2), doi: /iwc/iww014 Weisberg, O., GalOz, A., Berkowitz, R., Weiss, N., Peretz, O., Azoulai, S.,... Zuckerman, O. (2014). TangiPlan: Designing an Assistive Technology to Enhance Executive Functioning Among Children with ADHD. In Proceedings of the 2014 conference on interaction design and children - idc 14 (pp ). New York, New York, USA: ACM Press. doi: / White, S. A. (2004). Introduction to bpmn. IBM Cooperation, 2(0), 0. Williams, B. R., Ponesse, J. S., Schachar, R. J., Logan, G. D., & Tannock, R. (1999). Development of inhibitory control across the life span. Developmental psychology, 35(1), World Health Organization. (1993). Icd-10 classification of mental and behavioural disorders (the): Diagnostic criteria for research. World Health Organization. Retrieved from Wright, P., Wallace, J., & McCarthy, J. (2008, nov). Aesthetics and experience-centered design. ACM Transactions on Computer-Human Interaction, 15(4), doi: / Young, R. M., Barnard, P., Young, R. M., Barnard, P., Young, R. M., & Barnard, P. (1987, apr). The use of scenarios in human-computer interaction research. ACM SIGCHI Bulletin, 18(4), doi: / Zimmerman, J., Forlizzi, J., & Evenson, S. (2007). Research through design as a method for interaction design research in HCI. In Proceedings of the sigchi conference on human factors in computing systems - chi 07 (pp ). New York, New York, USA: ACM Press. doi: /

91 Appendix A Consent Form Forespørsel om deltakelse i forskningsprosjektet Desgning for cognitive and emotional control Bakgrunn og formål Forskningsprosjektet presenteres som en masteroppgave ved Universitetet i Bergen, institutt for informasjons- og medievitenskap. Målet med oppgaven er å finne interaksjonsteknikker med IKT-verktøy som kan hjelpe brukere med å reflektere over og regulere emosjoner. Prosjektet gjennomføres i samarbeid med Intromat, et forskningsprosjekt finansiert som fyrtårnsprosjekt av Norsk forskningsråd. Formålet med dette eksperimentet er å se kvaliteten på data som gis fra et armbånd ment for monitorering av fysiologiske data. Hva innebærer deltakelse i studien? Et intervju vil utføres der deltaker må løse noen oppgaver. Deltaker må bruke et armbånd under intervjuet som registrerer fysiologiske data. Hele seansen vil vare maksimalt 30 minutter. Hva skjer med informasjonen om deg? Personopplysninger vil bli behandlet konfidensielt. Kun alder og kanidatnummer vil lagres. Kun student og veileder til masteroppgaven vil ha tilgang til personopplysninger. Data fra oppgavebesvarelsen og de fysiologiske dataene fra armbåndet lagres sikkert på en digital lagringsenhet. Prosjektet vil anonymiseres, og deltakere i prosjektet vil ikke kunne gjenkjennes i publikasjon av oppgaven. Prosjektet skal etter planen avsluttes Datamaterialet anonymiseres etter prosjektslutt. Frivillig deltakelse Det er frivillig å delta i studien, og du kan når som helst trekke ditt samtykke uten å oppgi noen grunn. Dersom du trekker deg, vil alle opplysninger om deg bli anonymisert. Dersom du har spørsmål til studien, ta kontakt med: Eivind Flobak ved epost eivind.flobak@student.uib.no / tlf , eller veileder Frode Guribye på epost frode.guribye@uib.no / tlf Studien skal meldes til Personvernombudet for forskning, NSD - Norsk senter for forskningsdata AS. Samtykke til deltakelse Jeg har mottatt og forstått informasjon om studien, og er villig til å delta. Signert av prosjektdeltaker Dato 81

92 Appendix B Trier Social Stress Test Undersøkelse 1 Undersøkelse Om deltaker 1. Deltaker #: 2. Alder år. 3. Hvordan vil du beskrive din dag? 4. Hvordan føler du deg akkurat nå? Rolig Trygg Spent Urolig (engstelig) 5. Hvordan vil du vurdere dine egne regneferdigheter? Bra. Gjennomsnittlig. Dårlig. Vet ikke. Hvor stresset føler du deg nå? 6a. Rett før test 1 5 6b. Under test (4:30) 1 5 6b. Under test (4:00) 1 5 6b. Under test (3:30) 1 5 6b. Under test (3:00) 1 5 6b. Under test (2:30) 1 5 6c. Under test (2:00) 1 5 6c. Under test (1:30) 1 5 6c. Under test (1:00) 1 5 6c. Under test (0:30) 1 5 6c. Under test (0:00) 1 5 6d. Rett etter test er over 1 5 7a. Etter fem minutter hvile 1 5 7b. Hvordan føler du deg akkurat nå? Rolig Trygg Spent Urolig (engstelig) 7b. I dine egne ord, beskriv opplevelsen av testen: 82

93 Appendix C Early Design Workshop 83

94 84 Appendix C. Early Design Workshop 84