Mobile learning with location aware augmented reality business games

Transcription

1 Mobile learning with location aware augmented reality business games Dr. David Parsons - Massey University Dr. Krassie Petrova Auckland University of Technology December 2011

2 Contents Introduction...4 Using Mobile Devices for Learning...4 Mobile Learning with Serious Games...4 Methodology... 6 Experimental results...8 Questionnaire responses...8 Value of mobility...9 Fundamental levels - Bloom s taxonomy...9 Flow experience...10 Social flow...10 Critical thinking Interview responses...11 Learning experience...13 Learning outcomes...15 Mobile learning...16 Challenges...17 Suggestions for improvement...19 Critical incidents...20 Information quality...21 Data Logs P age

3 Observations...25 Evaluation and outcomes...25 Conclusions...26 References...28 Appendices P age

4 Introduction The goal for this project was to create a mobile learning application for undergraduate students in business and related disciplines that simulates a real world consulting exercise. The game was designed to support contextual learning, to be freely available, and be easy for both teachers and learners to use in any physical environment. The project involved developing the software required for delivering content on mobile devices based on the learners location, using established theories of game design to make the game engaging and motivating, and creating learning materials within the game that supported the development of higher level thinking skills. The game was designed to provide a learning experience similar to a real world business consulting exercise that can be used by any group of learners using readily available mobile devices. The game (based on a scenario designed by Bos and Gordon, 2005) was designed so that it could be played in any environment (such as a university campus) where predefined locations could be chosen to act as destinations in the game. The game augments the physical location to represent a virtual company. Players take the role of business consultants hired by this company to help it address its problems, initially presented to the players through the medium of a negative story in the press about the company. Players interview (through videos and multi choice questions) virtual employees located around the campus, obtaining information and physical artifacts. From these interviews and artifacts, players must infer the problems behind the symptoms the company is facing, and offer change recommendations, utilizing higher level thinking skills. Our findings suggest that learners found the game engaging and motivating, and we were successful in providing a context within which students brought their higher level thinking skills to bear on the problems presented by the game. The evaluations indicate, however, that we were less successful in providing a game that worked well as a team activity. Since teamwork is an important soft skill that we hope to develop within the game, further work on the game design and implementation is needed to address this current limitation. In this report we begin by outlining the reason for trying to implement a mobile learning activity using a serious game. We then explain our methodology and provide the results from evaluating the game created for this project. We conclude with some reflections and recommendations for practice. Using Mobile Devices for Learning Extensive research into mobile learning, where devices such as mobile phones and tablet computers are used as part of a learning activity, has shown that it can be used to encourage both independent and collaborative learning experiences, and raise self-esteem and self-confidence (Attewell, 2005). The ability to take a mobile device into any environment means that they have proved particularly useful in teaching subjects that can be explored in a real world context, such as applied maths, language learning, environmental studies, urban history and geography, but with imagination, mobile learning can be effectively used in any discipline. 4 P age

5 Mobile learning practice is increasingly moving towards location aware and augmented reality systems that enable learners to explore situated learning environments. Learning with mobile devices is most effective when it supports the learner within a real world context. As mobile devices increasingly support new technologies such as location awareness, we can more effectively integrate the learning process with its surroundings, and support collaborative learning with mobile communication. Situated learning, whereby the transfer of knowledge is situated where it is actually used, has long been recognised as a valuable way of teaching (Brown et al., 1989). Mobile devices and their associated software and services enable situated learning experiences to be enhanced with context relevant learning content overlaid on the learner s perception of reality (i.e., augmented reality). Although many one-off projects have explored this area, they have not addressed the important issues of embedding and sustainability, whereby mobile learning interventions can go beyond a single project and become reusable learning tools across the tertiary sector. The tools that have so far been developed to enable augmented reality mobile learning systems are often limited in their functionality, or in the range of supported mobile devices, or by both. Sustainability has also been an issue here, due to withdrawal of vendor support (e.g. the withdrawal of support for the popular MScapes tool by Hewlett Packard). Many of the existing tools are also poor in supporting collaborative mobile learning. A further issue is that some tools rely exclusively on continuous internet connectivity, limiting their applicability and incurring additional running costs. Given these various constraints, the project described here aimed to provide a mobile learning tool that was freely available, sustainable and could be deployed on a large number of mobile devices, without requiring internet access. Mobile Learning with Serious Games The concept of digital, game-based learning has become increasingly important in education (Prensky, 2001). Serious games, which are designed for the purpose of solving a problem, have been shown to be a powerful approach to mobile learning. They have been increasingly used for education and training, for example in the military (Bright, 2009) and for training fire fighters (Kankaanranta and Neittaanmaki, 2009) and are increasingly finding their way onto mobile devices. Although serious games can be entertaining, their main purpose is to teach. Unlike games that are designed purely for entertainment, in a serious game the entertainment aspect is included to increase the motivation to learn. Serious games are often used to simulate a learning environment where providing access to the equivalent real world environment would be too difficult, dangerous or expensive. We chose the domain of serious, business-related games to explore in our mobile learning project because such games have been shown in the literature to be useful activities within a business curriculum (Gilgeous and D'Cruz, 1996). However, no work has previously been demonstrated on how mobile business games may help students to learn. We therefore identified this as an important aspect of our project. We also considered the issue of collaborative learning to be an important feature of gaining critical thinking skills, and the business game that we identified as a useful exemplar incorporates this mode of learning (Bos and Gordon, 2005). 5 P age

6 Our objective for this project was to use a mobile serious game to provide a learning experience similar to a real world business consulting exercise that can be used by any group of learners using readily available mobile devices. In this game, (based on a scenario designed by Bos and Gordon, 2005), any campus can be used to represent a simulated organisation. Playing the role of teams of consultants, students are given a business problem to investigate, using mobile devices to move around the campus gathering information. Various locations reveal different information, and students need to collaborate in teams to collect and synthesize this information, in order to achieve the required learning objectives based on applying higher order thinking skills. The gathering of information is based on geo-tagging whereby access to resources is linked to particular locations, and triggered by the GPS system within the mobile device. The resources gathered are varied in terms of media and presentation, and include aspects of augmented reality, where information is presented overlaid on the real world. For example in the game virtual video interviews occur at physical locations where real world artifacts are collected. A final issue we should address in this introduction is our definition of a game. Thus far, we have indicated that the learning activity includes exploring a real world environment using both virtual and real world resources. In what way can this be classified as a game? It should be noted that this is specifically designed as a simulation game. Such games; contain multiple game-like elements but retain some environmental fidelity. The environment, objects and rules simulate a performance environment creating lifelife environments and populating them with objects that emulate the real world.... At the heart of a simulation game is a pertinent context that s aligned to learning and business needs. Typically, a complex decision-making tree provides that context; players navigate the tree by interacting with the environment and the elements that populate it. (Upside Learning, 2011) Thus the activity is game-like, rather than embodying all the features that might be expected of a purely recreational or casual game. It incorporates a decision tree and elements of the environment aligned to the learning objectives. Our work also relates to the broader area of business simulations and construction and management simulation games. Neef et al. (2001) assert that possible activities in such games relate to procurement (sometimes called acquisition), production, distribution, management, and construction. Our learning activity include procurement of resources (both virtual and real), management of these resources and construction of an analysis that can address the underlying problems of production and distribution that face the virtual company represented in the gameplay. In summary we have created and evaluated a game-like activity that addresses issues common to business simulations by leveraging the contextual learning made possible by mobile devices. Methodology A design science research method was used to develop and evaluate the mobile business game. The software was developed collaboratively by researchers at both universities (Massey and AUT) and was the empirically evaluated by user testing 6 P age

7 with both staff and students. Both quantitative and qualitative data was gathered using multiple approaches to data collection. 1. A questionnaire was administered to the participants after the practical activity 2. Semi structured interviews were conducted with the participants after the practical activity 3. Observations were made of participants carrying out the practical activity 4. Data logs from the mobile devices were analysed. The iterative design science cycle was applied through the development of the project artifacts and their evaluations. There were two major design cycles, each one consisting of smaller iterations of design, implementation and evaluation. An existing implementation of a location aware activity was taken as the baseline software architecture. This implementation was technically functional but lacked an effective game narrative and had failed to engage learners. The first iteration focused on technical testing of this platform in order to identify its reusable elements and areas for development in later iterations. During this cycle, we began to develop our own game design within the context of testing and reflection. This part of the project asked key questions about; what can serious mobile games hope to achieve for learners? What kinds of learning can serious mobile games support? And what innovations can be brought to bear in serious mobile learning games? At this stage, the purpose of our evaluation was to test the usability, functionality and perceived educational value of the continually evolving game design and the software. The second major cycle followed once the software framework and the game narrative had been developed to a point where testing and qualitative evaluation by the development team suggested that more rigorous empirical evaluation could take place using test subjects. At this stage our basic research had developed a series of testable hypotheses about the learning benefits of serious mobile gaming which we were able to encapsulate into a set of research questions, mapped onto questionnaire and interview questions for our experimental subjects. To test these hypotheses we undertook a series of experiments to evaluate the potential learning outcomes of the mobile game. The participants in this evaluation phase were tertiary students recruited from the two universities involved in the project, and the purpose of the evaluation was to test the learning effects of the system. It should be noted that this stage of the research required full ethics committee approval by both universities before it could be undertaken. Our main research questions related to the evaluation of the mobile game were as follows: Does mobility contribute to the learner s experience of the game? Does the game provide learning support at the fundamental levels of Bloom s taxonomy (knowledge, comprehension, application)? Does the game provide learning support at the higher levels of Bloom s taxonomy (analysis, synthesis, critical thinking)? 7 P age

8 Is the mobile game able to create a context within which learners experience flow and/or social flow? Does the game provide effective learning triggers? How do participants perceive the learning experience, including ease of use and information quality? Does the game successfully engage learners? These questions were answered using a combination of questionnaires, interviews, observations and analysis of data logs. In the following section we report on the experimental results of our evaluation. Experimental results This section summarises the results of our evaluation tests. We ran seven evaluation sessions, each with two participants, so we had 14 sets of data, including both quantitative (Likert scale questionnaire responses, data logs) and qualitative (semi structured interviews.) Questionnaire responses Figure 1 shows the average questionnaire responses for each of the 20 questions from the 14 participants. The vertical axis shows the mean average of the responses for each question from the respondents using a Likert scale questionnaire. The horizontal axis is the question number (see Appendix 1 for details of the questionnaire). It should be noted that for all responses, 1 equates to strongly disagree and 7 to strongly agree therefore the neutral / don t know value for each question is 4. It should be noted that for questions 5, 11 and 16 some level of disagreement was considered the preferred outcome. Figure 1: Average questionnaire responses from 14 respondents (Likert scale 1-7) The questions attempted to address different aspects of the game evaluation, but were deliberately intermingled in the questionnaire. In this analysis, the questions are addressed in their categories of analysis rather than their original order. The 8 P age

9 categories of question relate to the value of mobility, various levels of Bloom s taxonomy, and flow experience. Questions related to the value of mobility These questions were intended to investigate, from different perspectives, the value of using a mobile game, as opposed to other delivery methods, for delivering the learning goals in the activity. Table 1 shows the three questions relating to this part of the evaluation. The responses to question 1 reveal that the respondents saw no unique value in delivering these learning goals using a mobile solution, but nevertheless see a clear advantage over using a more traditional PC based elearning solution, and the respondents anticipated an increase in popularity for this type of game in the near future. From this we can see that the participants valued the situated learning aspect of the game, but perhaps could see that mobile devices were not the only way of achieving this. Table 1 responses to questions related to the value of mobility Question Question Average response Number 1. My learning about the business ideas 3.5 (neutral) covered by the game would be difficult to achieve using other methods. 16. The game would be better played on a PC. 2.0 (disagree) 20. Games like this one will become popular in the near future. 5.5 (agree) Questions related to the fundamental levels of Bloom s taxonomy At the fundamental levels of Bloom s taxonomy, we wished to evaluate knowledge, comprehension and application. The questions in Table 2 address aspects of these levels of knowledge. Generally the responses are positive, but for question 18 ( The information provided was always to the point ) the response is almost neutral. This is interesting, since the game is deliberately designed to provide conflicting information to require the learner to apply higher levels skills of critical thinking. Responses to this question suggest that the equivocal nature of the information supplied is recognised. Table 2 responses to questions related to knowledge, comprehensions and application Question Number Question Average response 3. Using the game improved my understanding of certain 4.86 business issues. 17. The information provided was helpful to playing the game The information provided was always to the point The information provided was easy to understand P age

10 Flow experience questions The questions listed in table 3 were intended to ascertain if the students responded positively to the questions relating to characteristics of flow experience, namely control, enjoyment and engagement. Two of these questions (5 and 11) tried to ascertain if the users felt frustrated or bored. High scores on these particular questions would have suggested that flow experience has not been achieved, but in fact both results indicate disagreement with these statements. Overall the responses suggest that positive aspects of flow were experienced by the learners, and that negative indications were low. Table 3 responses to questions related to individual flow experience Question Number Question Average response 2. Using the game gives me a feeling of control over my 4.36 learning about business issues. 4. I found the game provided an enjoyable way to learn Time seemed to pass slowly while I was playing the game I received adequate feedback from the game while I was 4.86 playing it. 9. I felt engaged in the activity of playing the game Interacting with the game is often frustrating Social flow questions Literature on individual flow experience has been around for a long time and there have been many publications relating to how individuals might experience flow. The concept of social flow, however is more recent and thus far has been less explored. The questions in table 4 were intended to investigate whether the team aspect of the game was important and could contribute to social, as well as individual flow experiences. The responses to these questions are largely neutral suggesting that the team aspect of the game has not yet been developed to an extent where it is adding value. Table 4 responses to questions related to social flow experience Question Number Question Average response 8. I would have preferred to have played the game as an 4.21 individual rather than in a team. 14. The game was well suited for playing as a team I enjoyed collaborating with my partner in the game Critical thinking questions Perhaps the most important questions asked in the questionnaire related to the higher levels of Bloom s taxonomy. The questions in table 5 were intended to ascertain if the game had encouraged the application of the higher level skills; analysis, synthesis and critical thinking. This appears to have been moderately 10 P age

11 successful, but the overall responses suggest only weak levels of agreement with these statements. Table 5 responses to questions related to higher level thinking skills Question Question Number 6. I felt able to identify some major business issues being presented in the game. 10. I felt that some information sources in the game were more reliable than others. 12. I was able to identify solutions to the problems faced by the fictional company presented by the game. 13 I was able to gather items of information from different stages of the game and identify relationships between them Average response In terms of ascertaining the consistency of responses, Figure 2 shows the range of average responses from the questionnaires, the lowest responses averaged 3.55, the highest 4.55, but there are no major outliers, and 10 of the 14 responses have average responses in the range 4-5. We therefore conclude that the responses are consistent enough to provide a baseline for further development. Figure 2: Range of average responses across 14 participants Interview responses The qualitative component of the research design involved administering a semistructured interview and analysing the data in order to investigate further participant perceptions about the game in terms of the value of mobility, game flow, and resulting knowledge acquisition and development. 11 P age

12 The interviews were conducted on the day of the evaluation. Eleven of the 14 respondents to the survey questionnaire were also available to be interviewed. The interviews were conducted in the following chronological sequence: MAP1-MAP8, AUP1-AUP4. Due to time constraints the interview with MAP6 was not conducted as planned; for similar reasons participant MAP8 could not provide responses to Q3a- Q3e. The interview instrument (Table 6, also see Appendix 2) included a group of artifact evaluation questions (Part 1), and questions that attempt to identify the effective learning triggers provided by the game, derived from critical incident theory (Part 2). Table 6 interview questions Part 1 Question 1 Question 2 Question 3 (general) Question 3a Question 3b Question 3c Question 3d Question 3e Question 4 Question 5 Part 2 Question 1 Question 2 Question 3 Artifact evaluation questions What are your general feelings about playing the game? What did you most like or dislike about the experience of playing the game? Do you feel that you gained any new knowledge or skills from playing the game? If so, what were they? How do you think playing the mobile game might compare with other learning experiences intended to teach the same knowledge and skills, for example, doing an activity in a face to face classroom situation, or using an e-learning system, or performing a real world consulting exercise? How useful in the specific context the context was the information provided? What do you think about the amount of information you received as a participant? How relevant was the information provided? How adequate was the information provided? How easy was it to use the information provided? What do you think were the main challenges in the game? How easy were those challenges to overcome? Do you think that the game could be improved in any way? If so, how? Critical incident theory questions Could you describe an incident that you remember that was an example of effective learning? What were the general circumstances leading up to this incident? Can you tell me exactly what the mobile learning game did that was so effective at the time? How did this incident contribute to the overall goal or effort of yourself and/or your team in playing the game? Other comments 12 P age

13 The questions in Part 1 were informed by prior research undertaken by the project team, and questions in Part 2 were adapted from Jonassen, Tessmer & Hannum, (1999). Participant responses to the interview questions were analysed qualitatively with respect to learner perceptions about the mobile learning artifact usefulness and performance (learning experience and new knowledge), artifact ease of use (challenges related to playing the game), information quality, further requirements, learner sense of achievement, and the effectiveness of the learning experience. In order to present the findings, a thematic analysis was initially performed that included Part 1 questions 1, 2 and 3 (general part only), questions 4 and 5 and any Other comments. A deductive / inductive approach was applied (Fereday & Muir- Cochrane, 2006). First the interview questions were used to identify and create data domains (Zhang, Von Dran, Blake, & Veerapong, 2001) which allowed us to structure the initial analysis: learning experience, learning outcomes, mobile learning, challenges, and suggestions. The responses were further divided into utterances and a hierarchy of codes was developed inductively and iteratively based on the themes emerging from the individual utterances. The hierarchy includes categories and factors within the categories. A final cross-check was done across the domains in order to map all utterances to the appropriate domain code regardless of the response position with respect to the interview question. A similar treatment was applied to Other comments responses. Next, the responses to questions 2 and 3 in Part 2 were analysed inductively applying and developing further the inductive coding framework, with question 1 responses providing instances of the category effective learning. Finally the responses to questions 3a-3e were analysed applying the deductively determined category information quality. The tables further in the text provide complete summaries of the participant responses and substantiate the coding. No answer responses were not included in the summaries. The identification codes of the interviewees were retained for further reference and analysis. Learning experience Overall, the participants liked and enjoyed the game:...good game,...playing it was awesome... The idea was wonderful (MAP4);...liked phone used (MAP7); It is a good simulator and will be a good exercise for the students (AUP2) with only one participant indicating that This may be fun for some people to go around and use the GPS but I [would] prefer playing it on a PC (AUP2). In their responses the interviewees identified a number of factors that contributed to the learning experience positively or negatively (Table 6). In summary, imperfections in the implementation (such as the amateur video recordings) and some characteristics of the physical environment (e.g. noise due to building works on one campus) may distract participants. While the prototype could be improved along these lines, factors such as GPS precision are related closely to the technology platform and any future development will be dependent on its location positioning capability. With respect to the key game concepts, participants valued the interactivity and the need to be proactive, enjoyed the innovative way of learning, and found the game engaging. The game design however needs to be enhanced with respect to features such as team work, and feedback on the overall outcome. It may also be necessary to develop inbuilt support for learners not entirely comfortable with the technology used. 13 P age

14 Table6 learning experience Category codes Navigation /GPS Key game concepts Factor codes ( affecting learner experience positively) Direction...good game interesting to use GPS (MAP7). Compass Compass, when I understood it, excites me, makes me search, as in hunting (MAP1). Active...liked walking around, not sitting (MAP3);.. liked moving around to places (MAP7); [liked]...moving from point to point (MAP1). Interactive liked...interactivity (MAP2); Liked the interactive part (MAP3). Engaging I liked going around and finding things, more informal (MAP 3);...good game,...playing it was awesome... The idea was wonderful (MAP4);...nice way to trace, much more involving (MAP7);...liked the initial brief...liked Artifacts (MAP 1). Innovative I liked it because it was a different way to go about solving problems (MAP2); It was definitely a novel experience (AUP3); It was interesting and novel experience (AUP4). Choice of technology...liked phone used (MAP7). Factor codes (affecting learner experience negatively) Direction What to do if wrong direction chosen? (MAP1); [difficult]... moving using the guide... does not point to the start. (MAP1); On some steps I began to doubt whether I am going all right (AUP1). Compass The compass was frustrating, I disliked the compass (MAP2). Precision GPS precision not that good, I was more than 20 feet from the target and it opened 4th interview (MAP4); The second interview was opened not when it showed (MAP4); [the game] finished a bit too fast (AUP1). Team work It is not a team game, It is an individual game. It did not affect my perception, having another team mate (MAP4); Not enough communication with the team member (AUP1). Individual? Or collaborative? Why was the other guy there?.. No idea what other guy was doing... Did not know it I am individual or competing as in a game (MAP1). Assessment/ feedback I did not understand how correct I was and how was I estimated. (AUP1) Choice of technology This may be fun for some people to go around and use the GPS but I prefer playing it on a PC (AUP2). Self-efficacy... is the game for the Navigation Savvy only? (MAP1); I got more confident after I played for a while. If there were more stops it would be better for me, to learn first (like 6 or 7) (MAP4);...would like more checkpoints (MAP7); [it was]... a bit of a learning curve...not possible to understand compass device during the briefing indoors (MAP1). 14 P age

15 Simulation...It is a good simulator and will be a good exercise for the students (AUP2). Use of video The video - a bit annoying. I would prefer a description on the display (MAP5). Interviews within the business It would have been better if the interviews were in a text format (AUP3). Prototype Video quality The video was very slow (MAP 5) Easy to see its videos [would be better](map7);...watch the video for the second time would be better (MAP 7);...the phone interviews were not clear as there were noises in the background [as pre-recorded KP] (AUP4); Text on screen quality Some questions are not possible to be read in full (AUP1) Need for earphones Frustrated with earphones (MAP1) Need to use a headset because the characters in the game not only talk but some pictures are showing (AUP1). Phone quality Better phone would be good (MAP7). Environment Cues The Kiwi sign was missing at stop 1 and 2 (MAP1);...on one occasion - Slightly off, the board had moved - The clue was misleading (MAP3). Noise Due to the noisy environment it was really hard to listen to the interviews..{aup3} Learning outcomes With respect to developing new skills and gaining new knowledge as a result of playing the game, the answers of the participants can be grouped into three categories, from none or very low to high (Table 7). The majority of the participants perceived the game as an effective vehicle for learner development: Mobile learning is very effective and also involves thinking (MAP7). Liked the game, the ideas, effective (MAP7). A significant number of participants perceived the game as beneficial to the development of skills in the higher levels of the Bloom s taxonomy. 15 P age

16 The comment about developing listening skills may need further exploration as it may reveal a gap in learner skills. The comment about team work skills is controversial at a glance given the evidence about insufficient emphasis on team work; however as the evaluation progressed the researchers became more skilled themselves in introducing participants to the game and encouraging them not to rush to the finish but stop where instructed and engage in a discussion with their game partner. Table 7 learning outcomes Category codes Factor codes Level Effective learning Key game concepts Business knowledge Knowledge about problems in the company (MAP2);... business dynamics, issues for large companies (MAP7); [I] picked up on a conflict (MAP8). Critical thinking...trying to ask the right questions...applying the right questions.. (MAP3). How to look at the problems (AUP2); Problem solving Yes...find solutions (AUP2);...question/problem solving (MAP4) Skill development New [skills]: Listening skills (MAP3); good...skills: deep thinking (MAP4). Team work Not too much. May be in team work skills (MAP5). Hard to tell.. no [new skills/knowledge] (AUP1) ; Not exactly (AUP3). High Low None or very low Mobile learning Participant perceptions about how the use of mobile technology facilitated learning and how it compared with other learning approaches are summarised in Table 8. All emerging themes related to key game concepts and information quality. Some limited evidence exists to indicate that perceptions about the game being rewarding, motivating and enjoyable as well as the active involvement add mobility value to the game which may be also appealing to a particular learner style (i.e. with a preference to an outdoors lifestyle). Learners also appreciated the condensed format of the game which allows just-in-time learning. Table 8 mobile learning Category codes Key game concepts Factor codes (Mobile learning adds value) Condensed The mobile version is faster than it Factor codes (Mobile learning does not add value) Simulation [Interviews] are not Factor codes (Mobile learning has potential) Team work Team work if 16 P age

17 would be in a face to face situation because the interviews are to the point (MAP2); In class: work hard to acquire the same in two or even three sessions. This was fast (MAP4).. Outdoors Since it is an outdoor activity it might interest quite a few people (AUP4). Player empowerment There was a paper (i took) we were asked to listen and understand but we had instructions. This is not the same, it is more. In the paper, Where we were, what to do, we were told all and what was right. I prefer learning not knowing the answer in advance (MAP3). Simulation..this was also real world (MAP4). Self-efficacy Mobile learning is easy, new skill, good to be using (MAP7). Rewarding...[mobile learning is] motivating [more than] just reading (MAP7). Motivating...[mobile learning is ] more rewarding (MAP7). Active The moving takes in more. (MAP8);... the active participation will help in the learning experience (AUP4). Enjoyment More fun (AUP1). directly interactive (MAP2). Self efficacy I was too focused on the technical issue... Did not listen to the interviews properly (MAP1). Simulation The mobile phone experience would be similar to a face to face classroom situation (AUP3);...not as educating as a real world consulting exercise (AUP3). more enhanced (MAP5). Simulation...not as effective as performing a real world consulting exercise (AUP1); Challenges With respect to how easy it was to play the game, the perceived challenges were coded using the codes developed so far. While some challenges were relatively easy to overcome (e.g. technical aspects of the environment such as clutter on display boards) others (cognitive) were more significant as they involved critical thinking and analytical skills (Table 9). While the technical challenges can be addressed as not to distract players, it may be argued that a certain level of challenge makes the game more engaging and rewarding, and that a fine balance between easy to play and challenging needs to be maintained with both technical and intellectual challenges providing motivation and facilitating effective learning. 17 P age

18 Table 9 challenges Category Easy to overcome: marked by a * Effective learning Problem solving Interpret the poster rather than make a decision before the end (not to be too quick) (MAP3); To get everything together to make conclusions AUP1). Critical thinking Ask the right question (MAP3). Navigation/GPS Direction GPS was difficult to use, could be more interesting (MAP4). Compass Trying to use the compass navigation (MAP2) Key game concepts Interviews with the business The [interview] questions (AUP2)* Flow [The] point of collaboration needs to be more clear (MAP8);...finding the artifacts (MAP7)*;...finding points (MAP5)*. Self-efficacy Learning about the application (MAP1); [it was]... easy to learn the platform mobile learning... the game requires a even much easier [approach] (MAP7). Environment Campus layout Identify places, especially at the start (MAP1);.. maybe it would have been harder on a different campus that I didn t already know my way around.. (MAP2). Cues Too many papers on the boards (MAP1); The Kiwi sign missing (MAP1). Suggestions for improvement Participants were asked to suggest further improvements. Several directions for improvement emerged (Table 10). These include the already mentioned emphasis on collaboration and team work, better navigation and navigation tools, enhancing the video material, and using a technically better platform. Most of the suggestions specific to the game design provide recommendations about how to make the game even more engaging by enriching the content, using a mix of media, and giving the player more control powers, and increasing the level of challenge by adding a 18 P age

19 completion time constraint. Only one of the responses contained a suggestion to change the game design radically by making it competitive. Table 10 suggestions Category Emerging directions Navigation/GPS Compass The compass could be better (MAP2); Compass - not working properly (MAP5). Precision Better precision (MAP4); Prototype Phone quality Key concepts game Smart screen would be better (e.g. ipad), [this one is] hanging up when you press the exit (MAP3) ; The Button; not good (MAP3); A new platform, better, more user friendly (MAP7). Flow By the time you get to the third building, you know the game flow, but you know you have to go indoors for the artifact but outdoors for the interview location. This means you have to go indoors and lose the location. It might be better if you could have the location indoors or the information outside (MAP2); Interactive The GPS could give me a confirm that I have collected the artifact (MAP4); More check points (MAP7). Rewarding [make it] more like a treasure hunt (MAP4). Use of video The information in the videos could be shorter; a bit more to the point (MAP2). Team work Either make it [more] collaborative or competitive (MAP1). Interviews within the business Alternating the questions (MAP8); May be it would be nice to have a list of seen artifacts and asked questions with answers (i.e. a summary at the end) (AUP1); Questions can be written up; and the correct answers as well (AUP2); To have the interviews in a different way, may be as an (AUP4). Player empowerment Request the player to OK opening the interview, not to open automatically (MAP4);...split and choose / [add] more choice in game to alter the outcome 19 P age

20 (MAP8); Let the participant choose in which direction the interview should go in (AUP3);... choose whom to interview...it would have been better if we were given the opportunity to choose whom to interview (AUP3). Time limit Introducing time? [to complete] May be a good idea (MAP4) Simulation...more work on programming part so it will be closer to a real classroom, e.g. not just a pre-recorded video (AUP2). Critical incidents demonstrating effective learning The critical incidents identified by respondents are presented in Table 11. Although only four respondents were able to identify an effective learning episode, the responses highlight the positive role of the artifact and the opportunity to ask questions after the video recording of the interview as well as the relevance of the information supplied. Three participants considered the incidents critical for the overall outcome of the game, while one participant considered it critical in terms of enhancing their learning experience. Table 11 critical incidents Effective learning episode Effective learning trigger Significance MAP1 I thought all of a sudden Why are they not receiving the number of the phones? Have they not done enough research?, which made me change my mind. I was thinking before that about flaws, batteries! The Third point, they are having more mobile phones. [ key game concepts: use of artifact, flow] The number of models [solutions] the information was not in the interview but in the poster [artifact]. [ effective learning: critical thinking solving ] MAP2 The game started to give me an idea about company politics The video and log at the third location [ key game concepts, use of artifact] At that point in the game I knew what was happening, issues in the company that needed to be sorted out. [ effective learning : problem solving ] MAP3 When I asked the question about other problems with batteries and they said yes. Helped me to see as not an isolated case but as part of a larger problem Interview + article [key game concepts: interviews with the business, artifact] The Questions and answers the question I asked stood out as the logical one. [effective learning : critical thinking ] MAP7 Artifacts - they triggers the thought purpose. Finding the artifacts; asking question, when you get the right answer [key game concepts: Team, engagement, hunt. [key game concepts: teamwork, engagement, 20 P age

21 interviews with the business, artifact] player empowerment ] Information quality The following factors contributing to information quality were investigated: Usefulness, density, relevance, adequacy, and ease of use. As seen from the response summaries, while information was found to be mostly well connected to the game, reasonably sufficient in order to help solve the problem, and relatively easy, there was a perceived lack of balance in the amount of the information provided at different stages of the game (density) which may also have affected the perceptions about the overall usefulness of the information provided. Second, some of the information provided was not found to be clearly presented, affecting the perceived level of relevance (Table 12). Table 12 information quality Usefulness (overall) MAP1 MAP2 MAP3 It was useful - pretty much to the point Some more than others; videos + artifacts + but not all; floor (4) useful, not the others; the questions useful, e.g. the first interview, but not the middle ones; videos- useful as I got the info not trusting each other. MAP4 Step-by-step information was useful. I doubted the article initially but then got convinced from the interviews. MAP5 Useful? 7 on a scale from 1 to 10 MAP7 AUP1 AUP2 AUP3 AUP4 MAP1 MAP2 MAP3 MAP4 MAP5 MAP7 Yes I think i could not realize it ( understand and use ) useful enough useful useful Density (amount of relevant information provided) Enough, satisfied, good context. There was too much information in each video clip Fine; no pressure to try to remember all of it. Videos a bit longer than it needed, other precise. Good. enough; more could take you away. AUP1 may be need some more information about the quest. I had different expectations before the start AUP2 more info will be needed and also in the question part. More and better questions should be asked cause i was thinking of another question which was not there. Relevance MAP1 Relevant. Not a problem to understand MAP2 I would give it 8 out of 10 MAP3 Relevant yes. 21 P age

22 MAP4 MAP5 MAP7 AUP1 AUP2 AUP3 AUP4 MAP1 MAP2 MAP3 MAP4 MAP5 MAP7 AUP1 AUP2 AUP3 AUP4 MAP1 MAP2 MAP3 MAP4 MAP5 MAP7 AUP1 AUP2 AUP3 AUP4 Relevance yes. Yes. some questions may have been misleading but this was not the issue. relevant relevant it was relevant;... was sometimes a bit confusing it was relevant....would be better if it was more clear Adequacy Adequate. Not a problem to understand. Adequate Adequate enough, in fact more than enough Yes but I found it by trial, after asking the wrong question. adequate, especially the artifacts. it was adequate. Sometimes, to a point it was. yes. adequate adequate Adequate. It was sufficient. adequate Ease of use Easy. But I could not select 3 options. (had a technical issue) There was a story. I knew what to do after picking up the first artifact. This information was still useful after you went on to collect the other artifacts Easy yes; navigation was on, compass ok Easiness: 2nd video had a picture, but I did not know where to go. The visuals info on the video should be more precise as to where I need to go. Reminded me of the Da Vinci Code book!! Easy. easy; happened without me, not much input needed- nice, No skills needed. easy very easy it was easy but it would have been better if the questions form which we were supposed to choose were displayed in full. I would read only part of the question. This made it hard to decide which questions to ask. It was quite easy. Table 13 shows the complete set of codes use to present the data at this initial stage of the qualitative analysis. As seen, there exist some many-to-many relationships between the set deductively defined domains and the inductively developed categories. These relationships can be explored in more detail in order to corroborate and explain the findings of the quantitative analysis and to build new theories. The responses about the quality of the information provide provides further highlights to be used in planning future work, alongside with the explicit suggestions of the participants and some of the perceived challenges. Additional analysis of the data may reveal relationships between some categories and the relevant comprising factors as well as between pairs of factors within a category and may lead to breaking categories such as key game features into new categories related to different aspects of the mobile artifact design, or redefining categories (e.g. prototype and navigation/gps may merge). A numeric representation of the number of utterances supporting each code with a domain/ category may add to understanding its perceived importance and impact (Zhang et al., 2001). 22 P age

23 Table 13 coding framework Data domains Category Factors Learning experience, suggestions Learning experience, learning outcomes, mobile learning, challenges, suggestions Learning experience, suggestions Learning experience, challenges Learning outcomes, challenges, critical incidents Navigation/ GPS Key game features Prototype Environment Effective learning Direction, compass, precision Condensed, outdoors, player empowerment. simulation, self efficacy, rewarding, motivating, active, enjoyment, flow, engaging, choice of technology, interactive, team work, assessmentfeedback, innovative, use of video, interviews with the business, time limit, artifact Video quality, text on screen quality, need for earphones, phone quality Cues, noise, campus layout Business knowledge, critical thinking, problem solving, listening skills Information quality Information quality Usefulness, density, relevance, adequacy, easiness of use Data Logs The information recorded in the data logs proved difficult to analyse effectively due to some limitations in the way the devices had been configured. Due to issues with the timestamps in the logs on the different devices it was not possible to identify which devices had been paired (device usage was totally anonymous), so we were unable to draw any conclusions about the way that the devices had been used within specific pairs. However we were able to derive the overall time taken for the activity by each person and the number of interactions that took place with the device. These interactions included events like watching a video and answering questions, as well as various interactions required for navigation. The minimum number of interactions required to complete all the activities in the game was 34. Table 14 shows information retrieved from the data logs, with the overall time taken to complete the task along with the number of events. Whilst this data is not rich enough for much analysis, we can see that all participants completed all the activities in the game, and that only 4 out of the 14 participants restricted their interactions with the system to a minimum, suggesting that the majority of participants were interested enough to do more than the minimum requirement. The times taken vary widely. These times do not provide very reliable data since sometimes one member of a pair would terminate the game on the device before the final discussion, with the other partner terminating the game afterwards. However we can at least infer from the times taken that only two participants seem to have rushed through the activity, 23 P age

24 and that most participants were engaged enough to spend 25 minutes or more on the activity, which was in line with our expectations. Table 14 data logs of duration and events Duration Number of events Figure 3 shows the mean number of interactions per second for the 14 participants, Although this data needs to be treated with caution, given the various factors that affected the collection of logged data, we can at least see that participants who rushed the activities, suggesting that they were not very interested in them and wanted to get done as soon as possible, were in a minority. This suggests that most participants were engaged in the activity Series1 Figure 3: Mean interactions per second In future evaluations we intend to use richer and more robust logging mechanism that will allow us to more directly triangulate the data logs from the devices with other learner data. Observations Our observations of the participants were informal, and were intended to ensure that we were aware of the issues faced by the participants and any aspects of the game that could be improved to enhance the learning experience. The most significant factors that arose from these observations were; the difficulty of positioning physical artifacts relative to the geo tagged locations so that they could be easily located, the impact of external environmental factors (buses, building work, exposure to the weather) on the choice of geo tagged locations, and aspects that needed to be better prepared for in the introductions to the learning task, for example learning to use the 24 P age