Studying six months in two hours

Transcription

1 Studying six months in two hours Virpi Roto Aalto University School of Arts, Design, and Architecture P.O.Box 31000, FI Aalto Finland Sari Kujala Aalto University School of Arts, Design, and Architecture P.O.Box 31000, FI Aalto Finland Copyright is held by the author/owner(s). Workshop on Theories, methods and case studies of longitudinal HCI research, in conjunction with CHI 2012, May 5 10, 2012, Austin, TX, USA. Abstract In this paper we examine the role of longitudinal studies in user experience (UX) research and practice. We present a retrospective method, UX Curve, that allows participants report the most meaningful experiences with a product after a period of use. We claim that retrospective methods are useful for studying long-term UX and for revealing which experiences matter over time. Keywords User experience, Long-term user experience, UX Curve, User experience evaluation methods Introduction In longitudinal usability studies, researchers typically investigate how the usability problems influence the use of a system over time. When the focus shifts from the ease and efficiency of use to experiential aspects, also the focus of longitudinal studies changes. In longitudinal user experience (UX) studies, researchers investigate how participants feel about the system in different phases of the study. They do not need to investigate the details of interaction, as they do in usability studies, but the goal is often to gain a holistic understanding of the reasons behind changing attitude towards the product as a whole. Longitudinal UX research methods may thus differ from those used for studying usability.

2 2 Figure 1. UX Curve Template Figure 2. An example of a general UX Curve for a mobile phone. By long-term UX we mean how the user s experience and relationship with a product evolves over time from the early learning and enthusiasm to becoming a part of daily life [4]. At best, good long-term UX means that the person loves the product, is proud of owning and using it, and takes actions in favor of the product or the manufacturer. Long-term UX is important for industry, since happy customers are loyal to the company, may help in developing the product further, and bring in new customers e.g. by recommending the product and company to others. UX evaluation methods in general can investigate momentary, episodic, or cumulative (i.e. long-term) experiences [8]. All these can be studied before, during, or after the actual interaction. For example, one may study anticipated episodic experience by asking what kind of experience the participant expects to have about task X with the system. Or, one may study momentary experiences during interaction by psychophysiological measurements. Also long-term UX can be studied before, during, or after a longer period of use. Before taking a product into use, we may ask participants to imagine how it would feel to have such a product in daily use, and thereby study anticipated long-term UX. We may study long-term UX during use by running a longitudinal field study and probing participants long-term UX at different phases of the study by Experience Sampling Method. Or, we may use a retrospective evaluation method and study UX e.g. 6 months after taking the product into use. Some scholars are skeptical about some combinations of the UX time spans (momentary, episodic, or longterm UX) and the moment of evaluation (before, during, or after interaction). Retrospective UX studies have been criticized, as it is known that people do not remember their actual experiences afterwards, and the chronological order of memorized experiences is often wrong. Studying momentary UX long after the interaction can hardly produce valid results. We claim, however, that retrospective methods are in many cases the best for studying long-term UX. They also work well in finding out which usage episodes are the most meaningful in the long run. We support the view that memory is sometimes more important than actuality [6],[3]. Also Oishi and Sullivan [7] have shown that retrospective evaluations predict later human behavior better than daily ratings do. If we evaluate experiences as they happen, we do not know how meaningful those experiences will be in the long run. Also, conducting longitudinal studies to evaluate UX during use is very expensive and prone to Hawthorne effect (participants knowledge that they are in an experiment modifies their behavior). This is why we chose to study long-term UX by using a retrospective method, UX Curve, which also reveals the most meaningful episodic experiences [4]. UX Curve method The idea of the UX Curve method is to ask the participant draw a curve that describes how UX of a product has changed over time. The participant is provided an empty graph on paper with a timeline (Figure 1) and asked to memorize the moment just before taking the product into use. While drawing the UX curve, the participant explains the reasons behind the changes in the curve. In the resulted curve, the left end of the curve shows the expected UX at the point of taking the product into use, and the right end the current situation. When the long-term UX was

3 3 Figure 3. Improving Attractiveness Curves for a mobile phone (with user IDs). Figure 4. Deteriorating and stable Attractiveness Curves for a mobile phone. Figure 5. Mean curves of improving and deteriorating curves from Fig. 3 and 4. worsening, the curve goes down, and with positive experiences the curve goes up (Figure 2) [4]. It is notable that the curve does not indicate the UX of the described incidents per se, but how they influenced the long-term UX at that point. We have tested the UX Curve method with a physical product (mobile phone) and with an intangible digital service (Facebook) [5]. In addition to the general UX curve that describes how the overall relationship has evolved between the participant and the product, we asked the participants draw a curve also about usability, utility, attractiveness, and usage volume. We found out that UX Curve is an effective method to reveal especially the hedonic experiences, while a questionnaire brought up more pragmatic experiences. An improving attractiveness curve had a statistically significant association with users satisfaction and their willingness to recommend the product or service to others [5]. The data collected with UX Curve method is mainly qualitative but the curve shapes can be analyzed quantitatively. In industry, the incidents that participants reported affecting UX and comparison of the curve trends of different products can give valuable guidance for future product development. Researchers can analyze the sources for the most meaningful experiences and use the curve trends for hypothesis testing. For example, we wanted to see if an improving UX Curve correlates with customer loyalty; it did [4]. Figures 3 and 4 give an idea of the various curve shapes analyzed in our study [4], and Figure 5 shows mean curves of the ones in the earlier figures. Although the set of individual curves seems messy, the mean curves show a general trend that is easier to analyze. Especially comparisons to other products may show interesting differences in the mean curves, in our case mobile phones (Fig. 5) and Facebook (Fig. 6). The attractiveness of a mobile phone rises quickest in the beginning, while the improvement is steadier with Facebook. With the deteriorating experience, the case is the opposite: mobile phone attractiveness decreases steadily, while there is a sudden drop with Facebook after the initial use period. The reasons behind the drop included simply losing interest, less than expected updates from friends, and noticing that Facebook takes too much time. For the mobile phone, reasons for steadily deteriorating interest were related e.g. to the value of the phone as a status symbol and losing the charm of novelty. There is still room for improvement in the curve analysis methods, since each curve is telling its own story and detailed categorization of the curve shapes may be difficult. In any case, researchers should not draw conclusions based on the exact timing and order of the incidents, as UX Curve is an approximation based on participant s memories. This is why we have analyzed the curve shapes on a very general level: classifying the curves to improving and deteriorating ones, and generating a mean curve summing all individual curves. Tools In light of our studies with UX Curve, it seems that most people are able to draw the curve easily. They enjoy this type of reporting, compared to traditional questionnaire filling. Compared to free recall, chronological curve drawing increases the amount and richness of the recalled information [3].

4 4 Figure 6. Mean curves of improving and deteriorating curves of Facebook. There are two similar methods to which we can compare UX Curve: Critical Incident Technique (CIT) [1] and iscale [3]. CIT is similar to UX Curve, as both methods aim at retrospectively identifying incidents that influence a phenomenon. CIT has been mostly used in service research, and it typically investigates single events or short-term interactions [2], while UX Curve focuses on how UX has developed over time. The iscale tool can be used to draw UX Curve via a Web browser, e.g. in remote online studies [3]. According to our studies, paper and pen are still easier and more flexible than the tool, although the tool would allow the participant to correct the curve, and the researchers to analyse the curve shapes more easily. Conclusions We discussed the approaches for investigating longterm UX and the reasons behind, and claim that retrospective methods are the most useful ones for this purpose. The criticism on retrospective evaluation is based on the assumption that the research needs to find out the details of the actual experience instead of the recalled experience. We claim, however, that studying momentary UX differs from studying longterm UX in this respect. In the first case, we should know exactly what the participants felt at the different points of interaction; this is certainly hard to study retrospectively. In the latter case, we need to know participants current feelings and thoughts about the product as a whole; this would be hard, and not very sensible, to investigate in the middle of interaction. A retrospective method is more suitable for this case. As an example of a retrospective method to study longterm UX, we briefly described a method called UX Curve. Based on our first studies with the method, it seems to be an effective method for studying long-term UX and the critical incidents influencing it. We were able to get an understanding of how and why a participant s UX had changed over six months, in less than two hours. References [1] Flanagan, John C. (1954). The Critical Incident Technique. Psychological Bulletin, 51 (July), [2] Gremler, D.D. (2004). The Critical Incident Technique in Service Research. Journal of Service Research, 7(1) [3] Karapanos, E., Zimmerman, J., Forlizzi, J., Martens, J-B. (2010). Measuring the dynamics of remembered experience over time. Interacting with Computers, 22(5) [4] Kujala, S., Roto, V., Vaananen-Vainio-Mattila, K., Karapanos, E., Sinnela, A. (2011). UX Curve: A Method for Evaluating Long-Term User Experience. Interacting with Computers, 23(5) [5] Kujala, S., Roto, V., Väänänen-Vainio-Mattila, K. and Sinnelä, A. (2011). Identifying Hedonic Factors in Long-Term User Experience. In Proceedings of the International Conference on Designing Pleasurable Products and Interfaces (DPPI 11), pp [6] Norman, D.A. (2009). Memory is more important than actuality. Interactions, March+April 2009: p [7] Oishi, S. and Sullivan, H.W. (2006). The predictive value of daily vs. retrospective well-being judgments in relationship stability. Journal of Experimental Social Psychology, 2006, 42, 4, [8] Roto, V., Law, E., Vermeeren, A., and Hoonhout, J. (eds) (2011). User Experience White Paper Bringing clarity to the concept of user experience. Outcome of the Dagstuhl Seminar on Demarcating User Experience, Germany.

5 5 Virpi Roto is a Post-doctoral Researcher at Aalto University School of Arts, Design and Architecture, Finland. Her background is in computer science and she has 18 years of experience in the HCI field, most of that time in Nokia Research Center. She has employed various usability and user experience evaluation methods in her work, and she understands about the applicability of different methods in industrial setting. Her list of publications and the allaboutux.org site demonstrates some results of her mission of making user experience work more systematic. Sari Kujala is a Post-doctoral Researcher at Aalto University School of Arts, Design and Architecture, Finland. Her background is in psychology and her research work has focused on getting human point of view to design: gathering and analyzing user needs, identifying user values and symbolic meanings of products and evaluating user experience.