Key factors of heuristic evaluation for game design: Towards massively multi-player online role-playing game

Transcription

1 Int. J. Human-Computer Studies 65 (2007) Key factors of heuristic evaluation for game design: Towards massively multi-player online role-playing game Seungkeun Song, Joohyeon Lee Cognitive Science Program, The Graduate School, Yonsei University, #134, Shinchon-dong, Seodaemoon-ku, Seoul , Korea Received 14 December 2005; received in revised form 25 December 2006; accepted 3 January 2007 Communicated by S. Wiedenbeck Available online 2 February 2007 Abstract The computer game industry has become the fastest growing field of the entertainment industry. However, only a very small number of computer game products overcome the costs of production and generate earnings. According to traditional marketing wisdom, customers preferences must be analyzed correctly to create successful products, and in the gaming industry, such information must be considered during the design process. This research aims to explore key factors of heuristic evaluation for game design. A review of literature pertaining to computer games and HCI was conducted along with an empirical research of a Massively Multi-player Online Role-playing Game (MMORPG). We identified 18 usability issues in MMORPG and presented a recommendation relevant to the issues. Empirical data were applied to a new heuristic evaluation framework. We determined the relationship between key factors and four game categories, such as game interface, game play, game narrative, and game mechanics. Moreover, the results presented 54 key factors for a new heuristic evaluation framework for game design. The conclusion presents key implications of our research in a game design context, particularly related to early design processes. r 2007 Elsevier Ltd. All rights reserved. Keywords: MMORPG game design; Usability; Heuristics evaluation; Design process 1. Introduction The computer game industry has become the fastest growing field of the entertainment industry. However, only a very small number of computer game products overcome the costs of production and generate earnings (Armstrong and Hamm, 1997). According to traditional marketing wisdom, customer preference is a core issue affecting the creation of successful products (Kotler, 1994), and the game design process is crucial to ensuring gamer satisfaction (Fabricatore et al., 2002). Recent studies have shown that the business model for Massively Multi-player Online Role-playing Game (MMORPG) relies mainly on gamer subscription fees, and that most gamers subscribe to only one game (Yee, Corresponding author. Tel.: ; fax: addresses: songsk@yonsei.ac.kr (S. Song), ljhyeon@yonsei.ac.kr (J. Lee). 2003). The number of new subscribers tends to grow slowly relative to other types of games (Woodcock, 2003). Since the success of any future game title depends on how much interest it can garner in the traditional game market (Wagner, 2002), game novices should be able to enjoy games easily by lowering the game entry threshold. Consequently, researching game novices is as important as researching game experts (Cornett, 2004). When current game design processes are examined, they proceed in the following order: synopsis, background and context research, script writing, visualization and concept art, interaction design, level design, materials, models and animation, media editing, programming, integration, beta test, and publishing (modified from Manninen, 2004). One major problem of this process is that user satisfaction, or whether the game is fun or not, can only be determined after all the procedures have been completed. The time and cost of game development could be reduced if user satisfaction were known at the early stages of the design /$ - see front matter r 2007 Elsevier Ltd. All rights reserved. doi: /j.ijhcs

2 710 ARTICLE IN PRESS S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) process, or even during game development. One way to solve this problem is to provide an iterative evaluation during specific design phases (Dumas, 2003). The goals of this study, therefore, are to (1) explore the key factors of heuristic evaluation for MMORPG design and (2) to present key implications of evaluative support during the early stages of the game design process. 2. Literature review 2.1. Current evaluation methods for computer games Gamers can provide important feedback regarding game play that will significantly enhance the fun they have while playing it. Evaluation methods for acquiring gamer feedback have been applied to computer games with varying success. Some evaluation methods, such as surveys, have been used frequently to collect information from gamers. A game designer would conduct a survey of gamers to find out which game features are liked or disliked. Correctly conducted surveys yield critical data that designers can employ in the generation of game features. However, surveys are difficult to conduct well. Survey questions are difficult to write in a manner that yields high-quality data. The sample for the survey must be large enough to conduct statistically significant analyses and comparisons, and it must be representative of the population of interest (Bradburn and Sudman, 1988; Sudman et al., 1996). Furthermore, surveys have retrospective attribute. Gamers are asked to think about their past experiences with a game when responding to a survey regarding a new game. It is difficult to ensure that survey responses are unbiased as respondents base their perceptions on long-time game interactions that serve to distort their long-term memory and allow for re-interpretation (Ericsson and Simon, 1993; Van Someren et al., 1994). Therefore, it is difficult to draw general conclusions from the data of such surveys. The focus group interview has also been used in the game design process for various purposes, including helping game designers determine how gamers perceive a game. Eight twelve gamers meet to discuss topics that the game designers are interested in. Focus group interviews are typically useful for concept generation during the early stages of the design process (Krueger, 2000). However, focus group interviews do not adequately aid game improvement because participants respond to abstract gaming concepts rather than concept development (Fulton and Medlock, 2002). Additionally, interviewees may be sensitive to group dynamics and pressures that could impact the quality of survey results. Because group pressures can influence individual participants, results obtained from group interviews may provide misleading and biased data although they had not intended to (Fulton and Medlock, 2002). The most popular source of gamer opinion is beta testing conducted early during the development process. In general, beta testers are volunteers who are recruited from various fields to play an early version of a computer game. They are asked to provide information about technical bugs in the software, but they can also provide feedback about issues relating to game play. Although the results of beta tests are crucial for identifying important bugs in games, they are less effective in helping to identify and fix game play issues or issues relating to fun. Beta testers are generally highly advanced players rather than members of the general gamer population. Moreover, feedback from beta tests is not collected in a systematical manner. For example, beta testers are asked to play the game as much as they can and provide feedback about any problems they encounter. Consequently, Pagulayan et al. (2003) mention that game designers have no direct control over their game play because beta testers play at home or work. These limitations prohibit obtaining specific feedback from beta testers. So far there have been some discussions about the strengths and weakness of user-based evaluation versus heuristic evaluation in Human Computer Interaction field (Kantner and Rosenbaum, 1997). User-based usability evaluation can be an excellent source of behavioral information about gamers. Usability research can help identify issues that obstruct gamers from experiencing the fun or enjoyment of a computer game. However, the typically small sample sizes used in usability evaluations prohibit generalization to a wider population. Moreover, as usability feedback often comes late after development decisions have been made, it is difficult to provide feedback relevant to game development. That is, the delay between when a product feature is developed and when usability feedback is conveyed to the developers precludes the use of those recommendations (Medlock et al., 2002). In order to overcome this limitation, the heuristic evaluation method has been developed to provide game designers with immediate feedback to identify the general issues at the beginning of the computer game design process. Desurvire et al. (2004) conducted a comparison between user testing and Heuristic Evaluation for Playability (HEP) based on the literature related to video games, computer games, and board games. The efficiency of the heuristics method was assessed with a new prototype game at the beginning of the design process. The HEP was reviewed by several game designers and experts. The playability evaluator conducted HEP while focusing on how each heuristic was supported or violated, and then identified the usability issue. At the same time, four subjects were given instructions to begin the game and think aloud while being asked several probing questions during 2-h game play sessions. The results of the experiment revealed the total number of issues identified from HEP to be greater than the number of issues found from the user study while user study issues, such as terminology, characters, and verbiage, were found to be more gamespecific. Most notably, the study revealed a key implication for usability evaluation methods in computer game. Specifically, HEP was found to maintain qualitative similarities and differences with user testing, but HEP was found to be best suited for evaluating general issues at the

3 S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) beginning of a game development cycle, during the prototyping process. Although there is the strength of HEP, it holds many issues that are cosmetic positives. Consequently, HEP generates most issues that are only part of the truth. In addition, the drawback of HEP is that it uses surrogate users, not typical users of online game. The results of heuristic evaluation are not primary user data and thus are slightly suspected (Nielsen, 1993; Dumas and Redish, 1999). Moreover, HEP is highly dependent on the skills and experience of the evaluators. Usability specialists may lack domain expertise, and domain specialists are rarely trained or experienced in usability methodology. In spite of the weakness of heuristic evaluation, it is more proper to detect general issues than user-based evaluation at the beginning of a game development (Desurvire et al., 2004). Thus, it might be a better idea for the evaluators to select the most appropriate method between heuristic evaluation and usedbased evaluation according to the features of each stage in game design process. Prevalent game evaluation methods, such as surveys, focus group interviews, beta-testing, and usability evaluations, have been reviewed briefly, with the positive and negative aspects of each method criticized. The primary purpose of this study is to extend the work of Desurvire et al. (2004), to critically review all aspects of their methods, and to develop a new heuristic evaluation method that minimizes the negative aspects and employs the positive aspects of each method. To this end, the first step to be carried out is to extract key factors in evaluation methods based on the characteristics of games and to use them as evaluation methods New aspects of usability evaluation for computer games The study of software usability has expanded rapidly in the last 10 years, but there have been comparatively few studies of usability in games (Federoff, 2002). This disparity may result from the differences that exist between computer games and traditional software productivity. Usability of traditional software is primarily focused on productivity, which comprises ease to learn and ease to use as its goal. Quite differently, the goal of usability in games is to reduce obstacles against fun, or to enhance the enjoyment of participation. Games are fundamentally different than general software. Games are mostly used to enjoy leisure time or have fun. Eventually, games are enjoyed in a manner similar to movies or novels. As a result, the goal of usability evaluation in games must be considered differently. Moreover, software productivity requires an effort for consistency. If users use different methods or tools each time, then they will suffer from having to learn something new each time. Games, however, have to provide a variety of experiences. Each time gamers play a game, they should be led to learn new rules and attempt new strategies to achieve goals. Otherwise, gamers may easily become bored. Since they include the feature of learning and exploration, games should provide a variety Game Narrative Game Play of experiences each time they are played. Furthermore, some constraints are deliberately imposed on a game, while they must be eliminated from traditional software. Such constraints may cause unexpected design problems in software productivity. A game, however, requires some constraints to increase enjoyment. Consequently, we must consider the manners in which usability differs between computer games and traditional software, and thus how usability should be evaluated (Pagulayan et al., 2003). In summary, although traditional usability evaluation is said to be superior to other evaluation methods, such as surveys, focus groups, and beta-testing, it is necessary to review the applicability of usability evaluation based on the nature of games. As previously mentioned, traditional usability evaluation, such as that used for software productivity, may not be directly applied to computer games. The primary focus of usability evaluation for computer games should be on reducing obstructive factors and promoting enjoyment instead of making the game easy to use with minimal mistakes. Given the above, what aspects of games can we evaluate for usability? According to Clanton (1998), the different usability issues in games include the game interface, game play, and the game mechanics. Game interface comprises all devices that gamers come into contact with as they interact with a game. Game play signifies all the processes that gamers go through to achieve the goal of a game. Game mechanics comprises the physics of a game, which is fabricated through the aggregation of artwork and program. Besides these issues, usability is also influenced by the tasks that gamers must or desire to accomplish, and the narratives that yield fun, enjoyment, and curiosity (Jenkins, 2002; Desurvire et al., 2004). Fig. 1 represents aspects of usability evaluation as determined from a review of literature pertinent to usability evaluation of games. 3. Empirical research 3.1. Overview of research Game Task Game Interface Game Mechanics Fig. 1. The aspects of usability evaluation of computer games. This research aims to explore key factors affecting usability evaluation on the basis of empirical data. To this end, we conducted an experimental study including the

4 712 ARTICLE IN PRESS S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) analysis of gamer tasks and a post-questionnaire. Usability issues were revealed, and a solution for the issues was explored. Empirical data were applied to an initial heuristic evaluation framework. Through correspondence analysis, we discovered a relationship between 54 key factors in four game categories, being the game interface, game play, the game narrative, and game mechanics. The 54 key factors were derived from both empirical results and literature review. This section presents the specific setting for our experiment. Section 4 describes the experimental results. In particular, Section 4.4 details the relationships between known key factors, the game categories, and the 54 key factors derived from a new framework of heuristic evaluation for MMORPG Experiment material Among the various MMORPG titles available, we selected World of Warcrafts (WOW) for its commercial success (ESA, 2005) and its record of excellent playability as established by various game critics. 1 According to Kasavin (2006), a famous reviewer for Gamespot.com, WOW is superficially similar to numerous other games. The game s interface is so slick and easy to learn and understand, and the game play itself is so quickly intuitive, that there is not even a tutorial to wade through; there are just some helpful, optional pop-up tool tips, as well as an excellent printed reference manual that goes into specific detail about most of the various aspects of play. WOW provides immediate feedbacks of the current status using audio and visual interface. This function serves as the criterion to measure correctly the performance of tasks such as success, error, and failure. WOW provides gamers with log data related to the extent of their damage during combat, which was different from other MMORPGs. This log data can be used as the objective index for the empirical research. In WOW, game progresses through quests. Quests signify another game comprised of small units (i.e., a game within a game). In case of novel or a play, they are often subdivided through the concept of chapter. In the game design, stage and map are utilized to reduce tedium and to induce gamers immersion. In addition, the total content is divided into subcontents by mission or quest. What gamers should do for each quest is apparent because the total game is divided into smaller units using quests in WOW. Traditional MMORPGs, before the appearance of WOW, had the problem that gamers often forgot their goal due to the endless story and a wide game space. This problem was solved through the concept of quests by WOW, which is the first time in the game industry. Moreover, gamers may perceive the problems and challenges easily that they must face to win the game because 1 / pc.gamespy.com/pc/world-of-warcraft/571585p1.htmls. WOW has an excellent design that allows for proper constraints per game content. These design factors in WOW are regarded as the important nature of game play. WOW is MMORPG based on the myth of North Europe that is the motif of the Lord of ring or Harry Potter. WOW has a perfect narrative structure with its sophisticated design. These aspects play an important role in assessing game narrative in terms of usability. WOW is a spectacular game due to its various extensions in terms of content and design although it has a concise, definite structure. In addition, WOW is unique because of the introduction of new game features, such as auction system, mail system, and diverse transportations. Such things were not considered in other MMORPGs. These aspects are relevant to game mechanics, which is achieved through a combination of animation and programming. WOW, therefore, is worth studying for game interface, game play, game narrative, and game mechanics in usability evaluation for MMORPG. For this reason, we selected WOW for our experiment Subjects Eighteen undergraduate students participated in the experiment at Kyonggi University, Seoul, Korea. All subjects were students attending a course, titled Understanding of Contemporary Multimedia. The experiment was conducted at the end of the term, and all the subjects were given tasks related to computer games several times. In this experiment, the subjects were classified into two groups according to their level of expertise with MMORPGs. The first level represented novices who were new to MMORPGs, and the second level represented experts that were experienced with MMORPGs but new to WOW games. Subjects were divided into two levels because playing patterns could differ by experience level, and because game design and evaluation would have to differ accordingly Experimental task We developed experimental tasks based on the analysis of the help function, or tutorial, for WOW, which is represented by an exclamation mark during play of the Korean version of WOW. For instance, if a gamer tries to hunt for a monster, clicking on the monster results in two exclamation marks in the middle of the bottom of the computer screen. The first exclamation mark refers to a battle mode, meaning, Right clicking on the enemy and approaching it initiates fighting. General fighting occurs automatically. The second exclamation mark is associated with magic and power, meaning, magic and power can be applied by clicking on the various icons at the bottom on the left-hand side. After the battle ends, another exclamation mark appears. This mark signifies the obtainment of trophies, and in the exclamation mark it states, If you want to search for items on the dead body, then you can click the right button of the mouse on the corpse.

5 S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) Table 1 Fundamental tasks of the MMORPG game WOW No. Task Detailed description Task 1 Acquire items Right click a dead body, and then right click items on loot panel Task 2 Skilled-attack Use a relevant skill after generating a rage Task 3 Chat with a NPC with! Right click annpc with an orange exclamation point (i.e.,!) Task 4 Equip an item Right click an item on the backpack panel Task 5 Trade items Click a merchant NPC, and then right click an item on the backpack Task 6 Identify quest Click a Holy Grail shaped-icon on the right button part (Menu Bar) in the interface Task 7 Adjust viewpoint Hold down the left mouse button to look around the world. Hold down the right mouse button to turn your character using the mouse Task 8 Find inventory Click a backpack-shaped con on the right bottom part (storage) of the interface Task 9 Register the ability Drag the ability to your action bar (i.e., quick slots) Task 10 First fight Right click the enemy and approach it Task 11 Movement Movement driven by mouse right click, Movement driven by keyboard (press mk-, AWSD) Task 12 Select a target Left click the target Task 13 Learn abilities After going looking for Lein Bishare NPC, gamers learn the ability suited for his or her own level Task 14 Find party fellow Chat with other PCs or use the community function Task 15 Party play Join party, and conduct the role of warrior for party play Right clicking on the item again leads to the gamer s inventory. As explained above, instructions for playing a game are typically provided at the beginning of a game, but in WOW, game instructions are provided peculiarly via the exclamation mark, or help icon in a unique way. This study involved observation of all exclamation marks, which in sum acted as a tutorial, and tasks were subsequently based on tutorial observation. The experimental tasks are provided in Table 1. Tasks frequently used in real game play are developed for the early stages of the game so that they may be learned quickly and naturally, because it would often be uncertain how the first-time WOW gamers would begin the game. In addition, we developed several tasks that might be deeply associated with factors affecting a usability evaluation of the game Experiment procedure The experimental sessions were divided into four sections. First, subjects were given instructions about the general nature of the experiment. They were asked to fill out a prequestionnaire as well as a written consent to experiment participation. The pre-questionnaire included the subjects demographic backgrounds, gaming experience and level of expertise. Second, all subjects were asked to select only the warrior class from the human race of the Alliance, 2 one of the player characters in the game, in order to guarantee a 2 Two character types, either alliance or horde members, exist in the game of WOW. Alliance characters represent good, and horde characters represent evil. The two character types face off in conflict. Our participants were instructed to choose to be members of the alliance, and then to be human from among several races, and then to be a warrior from among some available classes. With these choices made, the characters enter the game in the North-shire Valley, Elwynn Forest, a region of the virtual world in which the game is played. If a different race and type of character were chosen, then the gamers would begin the game in a different location, thus leading to a different gaming experience. As such, subjects were asked to choose the same character type, race, and class to ensure a consistent experimental environment. homogeneous experimental environment for the participants. The actual playing session ranged from 1 to 2 h. The researcher recorded subjects utterance and actions during game play using audio and video equipment. Third, a postquestionnaire was provided to determine subjects subjective attitude towards game play after playing session. The postquestionnaire was divided into three sections: game interface, game play, and game narrative. The post-questionnaire was based on 7-point Likert Scale. Finally, the participants were debriefed by the researcher about the primary purpose of this experiment, the need for confidentiality, the participant s responses and feelings, and so on. The experiment station carried two PCs with Windows XP, Intel Pentium IV 2.8 GHz processors with 512 MB RAM, GeForce graphics card with 128 MB RAM, Highspeed LAN connections, dual 200 GB HDD, two video camcorders, amplified goose-neck microphones, a movie clip capture program called Fraps 2.5, and Premier 6.5 which is a digital movie clip editor Data analysis The purpose of the study is to investigate the behavioral differences between the expert and novice gamer groups, both of whose members were encountering the game for the first time. To this end, for the 15 tasks in the experiment, we conducted a frequency analysis, computed the arithmetic mean, and performed a t-test to compare the difference between the performance of the novice and expert groups (i.e., the elapsed time required to complete the 15 tasks) The criterion for task experiment We measured subject performance in completing their tasks. Tasks completed with errors were those requiring minimal prompting, and subjects were considered to have failed a task when its completion required explicit direction.

6 714 ARTICLE IN PRESS S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) Empirical result 4.1. Pre-questionnaire results The pre-questionnaire was administered to ascertain subject profiles, including information such as age, sex, online gaming experience, and general knowledge of online games. Table 2 summarizes these profiles. The subjects chosen for the study represent the general demographic distribution of computer gamers (ESA, 2005). Mean age of the subjects is with maximum 24, minimum 18, and standard deviation The behavioral differences between both groups Table 3 summarizes the subjects performance of tasks. The analysis of tasks shows significant differences between Table 2 Subject s profiles Profile Expert Novice Total No. of subjects Age Mean: Mean: Mean: Max: 20 Max: 24 Max: 24 Min: 18 Min: 19 Min: 18 SD: SD: SD: Sex Male: 8 Male: 4 Male: 12 Female: 1 Female: 5 Female: 6 Experience with online games General knowledge of online games Never: 0 Never: 3 Never: years: years: years: 10 Over 2 years: 5 Over 2 years: 0 Over 2 years: 5 Beginner: 3 Beginner: 9 Beginner: 12 Intermediate: 3 Intermediate: 0 Intermediate: 3 Advanced: 3 Advanced: 0 Advanced: 3 novice and expert groups. However, a detailed presentation of experimental task performance is not presented for brevity. Instead, we present an analysis of some tasks with frequent errors and failures. Task 1, which involved the acquisition of items, was performed with two procedures. First, the subject was required to click the right button of the mouse on a dead monster, and then click on the applicable item when the trophy panel appeared. The average elapsed time required to complete the task was 1 min 11 s for novices 24 s for experts. The novices took about three times longer than the experts, representing a significant difference in task performance time (t(16) ¼ 1.917, po0.05). The task showed high error (7/18) and failure (4/18) rates relative to other tasks. Many subjects (11/18) thought that right clicking the dead monster, which caused the appearance of the trophy panel, was enough to obtain answers such as the item has been acquired, or it s showing the item I acquired (see Fig. 2). There was also another case related to subject error in acquiring items. When subjects (6/18) first clicked the right button of the mouse on the dead monster, causing the trophy panel to appear, they clicked the description part of the item rather than the icon-shaped item on the trophy panel. This behavior is attributed to the size of the item explanation part being three times larger than that of the icon-shaped item part in terms of the visual interface (see Fig. 3). Task 5 involved trading items acquired during hunting for sale of purchase of goods from a merchant Non-Player Character (NPC). For the novice group, task 5 took an average of 1 min and 35 s, while the expert group took only 58 s. The novices spent twice longer than the expert, again representing a significant difference in completion time (t(14) ¼ 0.708, po0.05). As with case 3, the error (4/16) and failure (3/16) rates were higher relative to those of other tasks. Table 3 Behavioral differences between novice and expert groups No. Tasks Error Failure Average elapsed time (min:s) T-value Novice Expert Task 1 Acquire items 7/18 4/18 1:11 0:24 T (16) ¼ 1.917, po0.05 Task 2 Skilled-attack 3/18 8/18 1:02 0:10 T (16) ¼ 0.032, po0.05 Task 3 Chat with a NPC with! 3/18 5/18 0:41 0:07 T (16) ¼ 0.705, p40.05 Task 4 Equip items 2/18 4/18 0:30 0:35 T (16) ¼ 0.316, p40.05 Task 5 Trade items 4/16 3/16 1:35 0:58 T (14) ¼ 0.708, po0.05 Task 6 Identify quest 3/18 2/18 0:20 0:15 T (16) ¼ 0.553, p40.05 Task 7 Adjust viewpoint 1/12 2/12 0:16 0:08 T (10) ¼ 1.201, po0.05 Task 8 Find inventory 1/17 2/17 0:21 0:10 T (15) ¼ 1.195, po0.05 Task 9 Register abilities 0/7 3/7 1:08 0:35 T (5) ¼ 0.839, p40.05 Task 10 First fight 0/18 1/18 0:37 0:35 T (16) ¼ 0.212, p40.05 Task 11 Movement 0/18 0/18 0:27 0:13 T (16) ¼ 2.022, p40.05 Task 12 Select a target 0/18 0/18 0:00 0:00 Null Task 13 Learn abilities 1/9 0/9 0:45 0:27 T (7) ¼ 0.644, p40.05 Task 14 Find party fellow 1/8 0/8 0:14 4:20 T (6) ¼ 1.108, p40.05 Task 15 Party play 0/5 1/5 19:56 11:18 T (3) ¼ 1.092, p40.05

7 S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) Fig. 2. Case 1, error in acquiring items. Fig. 4. Case 3, error in trading an item. Fig. 3. Case 2, error in acquiring items. Figs. 4 and 5 reveal many common errors. Fig. 4 shows subjects dragging the item they wanted to sell from their inventory to the trading panel. Some subjects (5/16) would then click on that item repeatedly, reporting that they continually clicked the item because there was no indication that it had already been sold. Fig. 5 shows some subjects (5/16) dragging the item to the merchant NPC, rather than dragging it to the trading panel, or right clicking on the items in inventory. Such errors suggest that the interface for trading items is not intuitive or natural. The other task experiment results are presented in Table Eighteen usability issues Table 4 shows 18 usability issues derived through the analysis of the post-questionnaire (see Appendix A) and the experiment based on the 15 tasks during the game play session. We identified usability issues, measured their scope and severity, and appraised their ranking as shown in Table 4. Scope defines how widely a usability problem is distributed throughout a product, and severity signifies the degree of seriousness of the problem (Dumas and Redish, 1999). Fig. 5. Case 4, error in trading an item. Scope was split into local usability issues and global usability issues. Local usability issues represent problems that occur only within a limited range of a product. Global issues in usability represent overall design flaws. Severity is divided into the following three levels. If the severity causes such difficulty that the gamer can no longer play the game, it is critical level. If it causes the gamer to become confused and slows the game progress, it is major level. If it causes problems but does not impede the way a gamer plays the game, it is minor level (Dumas and Redish, 1999; Barnum, 2002). In the way seriousness was ranged in the postquestionnaire, points below 4 3 were recorded on 7-point Likert scale, and a descending order ranking was adopted in terms of the number of subjects who had been affected by the seriousness. Usability issues were summarized in Table 4, based on the task-based experiment and the postquestionnaire. For brevity, we present only selected recommendations for solving the most critical usability issues. A description of all usability issues and recommendations is beyond the 3 Values below 4 signify a negative subjective attitude.

8 716 ARTICLE IN PRESS S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) Table 4 Eighteen usability issues No. Usability issues Severity Scope Users affected Issue 1 Acquiring items caused confusion Critical level Local 17/18 Issue 2 There were many incongruent actions initiated by right-clicking the mouse Critical level Global 15/18 To chat with the NPC To equip items Trading items Movement driven by mouse 13/18 13/18 9/18 Issue 3 There was no clear feedback when trading items Critical level Local 13/16 Issue 4 For warriors, there was no detailed instruction or tutorial explaining how to use skills Critical level Global 12/18 such as Heroic Strike at the beginning of the game Issue 5 Use of the Holy Grail-shaped icon to access the quest log was not intuitive Critical level Global 12/18 Issue 6 It was difficult to build story content 11/18 Issue 7 It was difficult to understand one control (Number 1 key) mapping onto two functions Critical level Global 10/18 (on/off) Issue 8 It was difficult to have empathy for his or her character image 9/18 Issue 9 There were too many quests that required only hunting the Monster at the game s 9/18 beginning Issue 10 After identifying the help exclamation point, when the gamers wanted to see the 8/18 relevant content again, it was difficult to find that information Issue 11 It was difficult to learn at the beginning of the game 8/18 Issue 12 There were too many hints in the game 8/18 Issue 13 It was difficult to read and understand the words presented within the game 8/18 Issue 14 It was difficult to change the automatic view Critical level Global 7/12 Issue 15 Game setting and key-binding setting were complicated 6/18 Issue 16 It was not easy to remember the game s names of places and people 6/18 Issue 17 The item size on the loot panel was smaller than its description area Major level Local 6/18 Issue 18 In the party play, the method of acquiring items was not fair 3/8 Note: Total size o18, and number 16 of 13/16 means that two subjects were missing among total sample size of 18. scope of this research; we refer the reader to Song and Lee (2006) for a more detailed account. Issue 1 in Table 4 was the most critical issue. Most of the subjects were asked to indicate their level of confusion related to the execution of item acquisition task. Subjects indicated that their confusion stemmed from a lack of confirmation of item acquisition. In order to solve this problem, the game should provide clear visual representation about how to obtain items. Otherwise, novices might give up game play at the beginning of game. The acquisition of items is critical to game play. All games including WOW should offer some visual indication of item acquisition, or for any other task completion. Issue 2 highlights the difference between games and general software products (i.e., MS Office, Web Browsers, Graphic editing tools, etc.), which often assign special actions to right clicking. In games, actions such as movement, conversing with NPC, trade decisions, and equipping items are not special occurrences, but rather basic actions that are used frequently. To support such actions, it would be more appropriate to provide left clicking as the basic value, rather than right clicking. Issue 3 showed many instances of the subjects dragging an item from the inventory to the trading panel, and clicking on it again only to have it returned to the inventory. This problem occurred because the subjects did not have an assurance as to whether the item was sold or not. In WOW, the sale of an item yields a sound effect representing the accumulation of money, and the visual interface shows a tiny number that increases incrementally with each sale. However, the sound and incremental number change had no effect on gamers confirmation with respect to trading items, and thus gamers were left to wonder whether their trade was successful. To fix this issue, a clear instruction or expression should be provided upon the completion of a trade. Clear feedback is a critical issue in the design of any game interface Key factors derived from a new framework of heuristic evaluation in MMORPG A process to derive key factors affecting game design from literature studies and experimental results is presented in this section. First, the game category was divided into four subcategories by reviewing literatures related to HCI and computer gaming by adopting a top-down methodology. The key factors of each category were then derived from empirical data by adopting a bottom-up methodology. The top-down method to obtain four game subcategories and the details pertaining to them are first discussed.

9 S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) Table 5 Key factors derived from empirical data No. Empirical data Key factors in usability evaluation Task experiment Post-questionnaire (See Appendix) Detailed key factors Category Issue 1 Task 1 (Acquire items) Clear feedback Game interface Minimal control Issue 2 Task 3 (Chat) Affordance of objects Game interface Task 4 (Equip items) Task 5 (Trade items) Task 11(Movement) Issue 3 Task 5 (Trade items) Clear feedback Game interface Issue 4 Task 2 (Skilled-Attack) Adequate expression in help Game interface Issue 5 Task 6 (Identify quest) Usage of metaphors similar to the real world Game interface Issue 6 C3 Building story content as emergent method Game narrative Issue 7 Task 10 (First fight) Natural mapping Game interface Issue 8 B10, B11 The gamer empathizes through game play Game Play The gamer empathizes with the game character Issue 9 C3 Building story content as emergent method Game narrative Issue 10 Task 6 (Identify quest) A10 Retrievable within the game Game interface Issue 11 B3, B4 Easy to learn at the beginning of the game Game play Issue 12 B9 Hints presented within the game too numerous Game interface Issue 13 A6 Suitable names Game interface Issue 14 Task7(Adjust Viewpoint) Proper grouping, depth and breadth of menu items Game interface Issue 15 A8,A9 Flexible usage according to Expertise Game interface Issue 16 A6 Suitable names Game interface Issue 17 Task 1 (Acquire items) Proper grouping Game interface Issue 18 B12 Fair play between gamers. Game play The categories of the game interface, game play, game narrative, and game mechanics were included in the MMORPG usability framework, as shown in Fig. 1. This kind of structure combines the three categories asserted by Clanton (1998) and Federoff (2002) with the game narrative suggested by Jenkins (2002) and Desurvire et al. (2004). For the game interface, eight of the 10 heuristic principles suggested by Nielsen (1994) were adopted, with the exclusion of Error prevention (Heuristic 5) and Recovering from errors (Heuristic 9). The two excluded heuristics do not seem to apply to computer games, but admittedly could aid traditional software productivity. Niesen s Heuristics 5 and 9 do not related to game design because gamer errors are closely tied to the final consequences of a game in game play. Game play and game mechanics have been devised based on the empirical studies of Malone (1980, 1982), who studied factors triggering fun in computer games, a subject also contributed to by the game developer Crawford (1982). Game narrative comprises the four criteria 4 of Jenkins (2002) and some assertions of Desurvire et al. (2004). To obtain heuristic evaluation factors through empirical study, we identified 18 usability issues through the analysis of task completion and post-questionnaires in Section 4.3. The usability issues were identified to be factors that preclude a gamer from continuing play or cause a delay in game play associated with confusion. For example, gamers may want to quit a game if completion of a task, such as acquiring items, is continually confusing. Or, if gamers suffer from frustration in trying to buy and sell items due to the lack of clear feedback, they may also wish to discontinue game play. It is essential that factors contributing to game cessation be accounted for during game design. As such, they must be adopted as usability evaluation factors (Table 5). Based on the 18 usability issues, we examined the relationships among usability evaluation factors as determined by top-down methods. This examination showed similarities between the 18 usability issues determined from the bottom-up approach and the usability evaluation factors set using the top-down approach. For example, issue 1 (confusion relating to item acquisition) was related to a clear feedback and the minimal controls during the completion of task 1 (item acquisition). Issue 2, which related to incongruent right-click results, was connected to affordance of objects. 5 This was derived from Task 3 (chat with NPC), Task 4 (equip items), Task 5 (trade items) and Task 11 (movement). Issue 6, which showed story content to be difficult to build, was 4 In his article, Jenkins (2002) noted four criteria, being embedded narratives, evocative space, emergent narrative, and enacting stories; game design as a narrative architecture. 5 Affordance of objects refers to the perceived and actual properties of things, primarily those fundamental properties that determine just how the thing could possibly be used (Norman, 1990).

10 718 ARTICLE IN PRESS S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) correlated to emergent narrative. 6 This result was obtained through the post-questionnaire (C3). 7 What s noteworthy is that issue 2 (affordance) and issue 7 (natural mapping) 8 are newly discovered usability evaluation factors which could not have been established using a top-down approach. The final version of the usability evaluation framework for MMORPGs was derived using the following approach. In the initial framework, correspondence analysis was conducted to determine the relationships between game subcategories including the game interface, game play, the game narrative, game mechanics and the 31 usability factors determined from empirical data. From a statistical viewpoint, the interface, game play, game narrative, game mechanics, and 31 usability factors were nominal, categorical data that could serve as the basis for study results. That is, it is possible to identify the relationship between two variables with correspondence analysis of nominal, categorical data. In all, 30 game designers and experts with more than 3 years experience in the MMORPG field were asked to choose appropriate cells 9 in matrix comprised of factors and subcategories in Table 6, which means the correlation between four subcategories and 31 usability factors. Each cell in Table 6 has the repeated, accumulated frequency. For example, when we examined the relationship between vividness factor and four subcategories, the frequency of interface for vividness factor is 10, the frequency of play for vividness factor is 3, the frequency of narrative for vividness factor is 2, and the frequency of mechanics for vividness factor is 18. This statistic result implies that there is the highest relevancy between vividness and mechanics, and the second highest relevancy between vividness and interface. Moreover, we can find that play and narrative have low relevancy between them. Table 6 summarizes the results of the expert survey. The correspondence analysis was conducted to identify the relationships among 31 factors and the four game 6 The Sims as a sandbox or dolhouse game, suggesting that it should be understood as a kind of authoring environment within which players can define their own goals and write their own stories. Thus, emergent narrative is not pre-structured or preprogrammed, taking shape through the game play, yet they are not as unstructured, chaotic, and frustrating as life itself (Jenkins, 2002). 7 Readers are advised to refer to Appendix A for the post-questionnaire. 8 Mapping is a technical term meaning the relationship between two things, in this case between the controls and their movements and the results in the world. Natural mapping, by which it was meant taking advantage of physical analogies and cultural standards, leads to immediate understanding (Norman, 1990). 9 Appropriate cells in matrix comprised of factors and subcategories in Table 6 could be chosen by experts more than twice when they were confident in the existence of duplicate relationships between the factors and subcategories. For example, when experts thought that there are two relationships between a factor (vividness) and two subcategories (Interface, Mechanics), they can respond to chose two times onto 2 cells (i.e., first cell means vividness in first column and interface in first row, second cell means vividness in first column and Mechanics in fourth row) simultaneously. Table 6 Frequency between subcategories and factors Factors Subcategories Interface Play Narrative Mechanics Active Margin Vividness Reward Replay Recog Physic Percept.-motor skill Pace Natural mapping Modeless Match metaphor Learn Interaction with narr Immed. display Help Goal Flexibility Feedback Fair Evocative sp Enacting st Empathy Emergemt narr Embedded narr Difficulty Curiosity Control Consistency Challenge Balance Affordance Aesthetics Active margin Note: Appropriate cells in matrix comprised of factors and subcategories in Table 6 could be chosen by experts more than twice when they were confident in the existence of a relationship between the additional choices. For example, when experts thought that there are two relationships between a factor (vividness) and 2 subcategories (Interface, Mechanics), they can respond by selecting two 2 (i.e., while the first cell means vividness in the first column and interface in the first row, the second cell means vividness in the first column and Mechanics in the fourth row) simultaneously. Therefore, we can examine the relationship between vividness factor and four subcategories. According to frequency between the first column (vividness factor) and four subcategories in Table 6, the frequency of interface for vividness factor is 10, the frequency of play for vividness factor is 3, the frequency of narrative for vividness factor is 2, and the frequency of mechanics for vividness factor is 18. This statistic result implies that there is the highest relevancy between vividness and mechanics, and the second highest relevancy between vividness and interface. Moreover, we can find that play and narrative have low relevancy between them. Table 6 summarizes the results of the expert survey. subcategories. Fig. 6 shows the 31 factors for the four subcategories using the coordinate system method. Fig. 6 is a location map of each variable category computed by the score of the row variable and the column variable.

11 S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) review of literature relevant to computer game design and HCI. We identified 54 key factors for game design and broadly divided them into four subcategories, being Game Interface, Game Play, Game Narrative, and Game Mechanics. Game Interface includes such factors as feedback, metaphor, control, consistency, recognition, flexibility, esthetics & minimalist design, help, affordance, and natural mapping. Game Play involves factors such as goals, learning, rewards, challenges, pace & pressure, re-playability, empathy, fairness, difficulty, balance, perceptual-motor skill. Game Narrative consists of embedded narratives, evocative space, emergent narratives, enacting stories, interaction between gamers and the narrative, curiosity, and modeless. Game Mechanics comprises factors including immediate display, physics, and vividness. This study offers a new method for the field of game design that may aid in the creation of more usable, fun games Discussions and implications According to this location map, game interface is closely related to feedback, metaphor matching, control, consistency, recognition, flexibility, esthetics, help, affordance, and natural mapping. On the other hand, game play is strongly related to game goals, learning, reward, challenge, pace, re-playability, empathy, fairness, difficulty, balance, and perceptual-motor skill. Game narrative is strongly related to embedded narratives, evocative space, emergent narratives, enacting stories, interaction between the gamer and the narrative, curiosity, and modeless. 10 Game mechanics were closely related to immediate display, physics, and vividness. From the correspondence analysis, it was possible to obtain 31 factors in four subcategories. In addition, feedback, control, consistency, recognition, flexibility, esthetics, and help were subclassified based on the heuristics studies of Nielsen (1994). In the end, 54 usability evaluation factors were devised from the 31 usability factors under the four subcategories (see Table 7). Consequently, the final version of the heuristic evaluation framework for MMORPGs was developed by effectively using both top-down and bottom-up approaches. 5. Conclusion Fig. 6. Correspondence analysis results Summary of the results This study presents key factors for game design. We derived key factors through a usability evaluation and a 10 If an application is modeless, it allows the user to do whatever they want when they want without being restricted to certain features in certain modes. The limitations of this study stem from cultural factors. This study was conducted in a distinctly Korean setting, which is relevant to the results. For example, the subjects did not naturally maintain empathy for their character images (Issue 8) because Korean gamers tend to prefer characters of a traditionally beautiful appearance, such as those in Lineage 2 published in NC soft, Korea. According to McGregor (2006), Lineage 2 received 10 of 10 points for graphic quality in a MMORPG.COM review. It was also difficult for subjects to remember place and person names (Issue 16) because Korean novices were not familiar with WOW game terminology (see Table 4). Therefore, our future works will include a more elaborate experimental design that is capable of handling such cultural complexities. Despite the limitation, this research offers both strong theoretical and practical contributions. The theoretical contributions of this study relate to the new framework for heuristic evaluation in MMORPG games, which is an objective method for reducing obstacles to fun, rather than obstacles to accomplishment (as in productivity applications). This research confirms and extends the current line of computer game usability research. Previous usability researches have focused on efficiency, ease to learn, and ease to use because they have been concerned with product performance in terms of traditional usability. This research extends the current usability framework by examining the fun of a game, rather than just productivity. In terms of practical application, this research offers 54 key factors derived from the study for heuristic evaluation in MMORPG games, as first examined by Desurvire et al. (2004). As discussed in Section 2.1, current computer game evaluation methods are limited to surveys, focus groups, beta-testing, and other traditional user-centered

12 720 ARTICLE IN PRESS S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) Table 7 Fifty-four key factors derived from a new framework of heuristic evaluation in MMORPGs Category Key factors Source I. Game interface A. Feedback of the current status 1. Clear feedback Nielsen s Heuristic 1 (1994), Malone (1980, 1982): 2. Adequate feedback Shneiderman (1992), Norman (1990) Issue 1 B. Metaphor matching 3. Usage of metaphors similar to the real world Nielsen s Heuristic 2 (1994), Issue 5 C. Control and freedom 4. Immediate control Nielsen s Heuristic 3 (1994), 5. The default of control Bickford (1997) 6. The degree of freedom D.Consistency and standards 7. Consistent name, information, structure, and expressions 8. The trends set by the game community Nielsen s Heuristic 4 (1994), 9. Consistent in control, color, typography, and Sanchez-Crespo Dalmau (1999) dialog design E. Recognition rather than recall 10. Retrievable within the game Nielsen s Heuristic 6 (1994), 11. Suitable names Norman (1990), 12. Proper grouping Issue 10; 13. A visual hierarchical structure Issue 13; 14. Clear visual distinctions Issue 14; 15. Proper depth and breath of menu Issue 16; Issue 17 F. Flexibility and efficiency of use 16. Flexible usage according to expertise Nielsen s Heuristic 7 (1994), 17. Options Issue Personalization 19. Quick performance methods 20. Automatic performances Nielsen s Heuristic 8(1994), G. Aesthetic and minimalist design 21. Aesthetic appreciation Issue 1; 22. Minimal expression Issue 7; 23. Minimal control Issue 8; Issue 15 H. Help 24. The adequate help content Nielsen s Heuristic 10 (1994), 25. Adequate expression in help Issue 4; 26. Hints presented within the game not too numerous Issue 10; Issue 12 I. Affordance of objects 27. Primarily fundamental properties that determine just how the thing could possibly be used J. Natural mapping 28. The relationship between two things, in this case between the controls and their movements and the results in the world II. Game play K. Goal 29. A clear over-riding goal of the game at the beginning of the game Norman (1990), Issue 2 Norman (1990), Federoff (2002), Issue 7 Clanton (1998), Malone (1982) 30. Multiple goals on each level L. Learning 31. Easy to learn at the beginning of the game. Crawford (1982) 32. Hard to master the game Issue 11 M. Reward 33. Rewards that are more deeply immersed in the game N. Challenge 34. Keep playing more in spite of high difficulty in the game Bickford (1997) Myers (1990)

13 S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) Table 7 (continued ) Category Key factors Source O. Pace and 35. Pace to apply adequate pressure to the gamer Clanton (1998), without discouraging them Pressure Shelley (2001), P. Re-playability 36. It should play the game again Shelley (2001) Q. Empathy 37. The gamer empathizes through game play Desurvire et al. (2004) 38. The gamer empathizes through game character Issue 8 R. Fairness 39. The fair play between gamers Clanton (1998), Issue 18 III. Game narrative S. Difficulty 40. Various difficulty levelsn according to each gamer s level T. Balance 41. Game play should be balanced with Multiple ways to win U. Perceptual-motor skill 42. Behaviors are quick and well-timed responses to changes and threats in the environment V. Embedded 43. The story should be adequately laid on the space narratives W. Evocative space 44. Evoking the materials of other famous works, such as the Lord of ring, warcraft, or star war etc. Malone (1980, 1982): Crawford (1982), Malone (1980, 1982): Pagulayan et al. (2003) Jenkins (2002) Jenkins (2002) X. Emergent 45. Building story content as emergent method Jenkins (2002), narrative Shelley (2001), Issue 6; Issue 9 Y. Enacting stories 46. The short stories reaffirm gamer interest after they have become bored with general game play Jenkins (2002) Z. Interaction between gamer and narrative 47. Discovering the story as part of game play Desurvire et al. (2004) AA. Curiosity 48. Maintaining curiosity regarding story outcome Desurvire et al. (2004) BB. Modeless 49. Perceived as modeless even if the game cannot be modeless Federoff (2002) IV. Game mechanic CC. Immediate display 50. Feedback given immediately to display gamer s control Malone (1980, 1982): DD. Physics 51. Feeling natural mechanics Federoff (2002) 52. Correcting weight and momentum. EE. Vividness 53. Unpredictable behaviors of Monster or NPCs Crawford (1982), 54. Unexpected game outcomes Bickford (1997), Malone (1980, 1982): usability evaluation methods that are employed after product development. However, framework developed herein has the potential to identify critical problems with game design, even when applied at early stages of the design process. Therefore, this research extends current computer game evaluation methods. For example, the possibility of immediate feedback will enable game designers to improve the quality of the game content quickly, without resorting to the complex procedures and longstanding computer game evaluation methods. The results of this study have clear implications for overall game development, and should serve to improve game quality at any stage of the design process. This study examined subjects playing WOW at a specified game entry point with a specified character class for approximately 2 h. Consequently, the study results cannot be extended to the explanation or evaluation of Deep Play or game play with other races or classes. Future researches will include an assessment of key factors for late-game design based on a usability evaluation game play in the latter stages of the game.

14 722 ARTICLE IN PRESS S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) Appendix A. Post-questionnaire for game interface, game play, and game narrative See Table A1 for further details. Table A1 Post-questionnaire for game interface, game play, and game narrative List A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 B11 B12 B13 B14 B15 B16 B17 B18 C1 C2 C3 C4 C5 C6 Post-questionnaire Is the visual feedback, such as their status/score, and visual cue clearly presented? Is the audio feedback, such as their NPC s voice, and audio cue adequately provided? Does it help that interactions with other characters or targets are analogous or similar to real-world situations? Do you feel in control of the actions occurring on the screen? Are consistent Names/Information/Structure/ Representation being provided? Are instructions for the system retrievable within the game? Is the flexibility provided with variable difficulty level? Are game control tools and the on-screen interface unobtrusive and conducive to easy access to the game environment? Are game controls and on-screen interface simple to provide easy access to the game environment? Is help needed to engage in game play displayed through a tutorial? Does it provide a clear overriding goal of the game at the beginning of the game? Are there multiple goals on each level? Is it easy to learn at the beginning of the game? Is it hard to master the game? Are there rewards for becoming more deeply immersed in the game? Do you want to keep playing more in spite great game difficulty? Does the game pace apply adequate pressure without discouragement? Does it make the gamer want to play the game again? Are the hints presented within the game not too numerous? Does it make the gamer empathize with the character during game play? Does it make the gamer empathize through game character? Is the play between gamers fair? Are the behaviors of Mob or NPCs predictable? Are the outcomes of the game not predictable? Does it provide variable difficulty levels that are suited to all levels of gamers? Is game play balanced with multiple ways to win? Is it perceived as modeless even if the game cannot be modeless? Are you taught skills early that you expect to use later? Is the story evenly distributed throughout the game? Can you evoke the materials of other famous works, such as the Lord of Rings, Warcraft, Star Wars and so on? Can you build story content via the emergent method? Do the short stories cause you to become refocused on the game after becoming bored with game play? Can you discover the story as part of game play? Is the storyline designed so that it may create curiosity? Note: A1 A10 game interface part; B1 B18 game play part; and C1 C6 game narrative part. References Armstrong, L., Hamm, S., CD-ROMs: The Giants Rule. Business Week. The McGraw-Hill Companies Inc., New York. Barnum, C.M., Usability Testing and Research. Pearson Education Inc. Bickford, P., Interface Design: The Art of Developing Easy-to-Use Software. Academic Press, Chestnut Hill, MA. Bradburn, N.M., Sudman, S., Polls and Surveys: Understanding What they Tell Us. Jossey-Bass, San Francisco. Clanton, C., An interpreted demonstration of computer game design. In: Proceedings of the Conference on CHI 98 Summary: Human Factors in Computing Systems, pp Cornett, S., The usability of massively multigamer online roleplaying games: designing for new users. In: Proceedings of the Conference on CHI 2004, vol. 6(1), pp Crawford, C., The Art of Computer Game Design. Retrieved from / htmls. Desurvire, H., Caplan, M., Toth, J.A., Using heuristics to evaluate the playability of games. In: Proceedings of the Conference on CHI 2004, pp Dumas, J.S., User-based evaluations. In: Jacko, J., Sears, A. (Eds.), Handbook for Human Computer Interaction in Interactive Systems. Erlbaum, London, pp Dumas, J.S., Redish, J.C., A Practical Guide to Usability Testing, revised ed. Intellect Ltd., Great Britain. Entertainment Software Association, Essential Facts About the Computer and Video Game at E3. Ericsson, K.A., Simon, H.A., Protocol Analysis. MIT Press, Cambridge, MA.

15 S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) Fabricatore, C., Nussbaum, M., Rosas, R., Playability in action videogames: a qualitative design model. Human Computer Interaction 17 (4), Federoff, M.A., Heuristics and usability guidelines for the creation and evaluation of fun in video games. Master Thesis, Indiana University, Bloomington. Fulton, B., Medlock, M., Beyond psychological theory: getting data that improve games. In: Proceedings of Game Developers Conference, Jenkins, H., Game design as a narrative architecture. In: Wardip- Fruin, N., Harrigan, P. (Eds.), First Person: New Media as Story, Performance, and Game. The MIT Press, Cambridge, MA. Kantner, L., Rosenbaum, S., Usability studies of WWW Sites: heuristic evaluation vs. laboratory testing. In: Proceedings of the 15th Annual International Conference on Computer Documentation Table of Contents, Salt Lake City, Utah, USA, pp Kasavin, G., Here is the Online Role-Playing Game You Should Play, No Matter Who You Are. Gamespot.com. Kotler, P., Marketing Management, Analysis, Planning, Implementation, & Control, eighth ed. Prentice-Hall, Englewood Cliffs, NJ. Krueger, R.A., Focus Groups: A Practical Guide for Applied Research. Sage, Thousand Oaks, CA. Malone, T.W., What makes things fun to learn? A study of intrinsically motivating computer games. Ph.D. dissertation, Department of Psychology, Stanford University. Malone, T.W., Heuristics for designing enjoyable user interfaces: lessons from computer games. In: Thomas, J.C., Schneider, M.L. (Eds.), Human Factors in Computing Systems. Ablex Publishing Corporation, Norwood, NJ. Manninen, T., Rich interaction model for game and virtual environment design. Ph.D. Thesis, Oulu University Press, Oulu, Finland. McGregor, C., MMORPG.COM. Retrieved from / mmorpg.com/gamelist.cfm/setview/ratings/setrview/none/gameid/46s. Medlock, M.C., Wixon, D., Terrano, M., Romero, R., Fulton, B., Using the RITE method to improve products: a definition and a case study. In: Proceedings of Usability Professional s Association. Myers, D., A Q-study of game player aesthetics. Simulation & Gaming 21 (4), Nielsen, J., Usability Engineering. Academic Press Inc., New York, NY. Nielsen, J., Heuristic evaluation. In: Nielsen, J., Mark, R.L. (Eds.), Usability Inspection Methods. Wiley, New York. Norman, D., The Design of Everyday Things. Doubleday, New York. Pagulayan, R.J., Keeker, K., Wixon, D., Romero, R., Fuller, T., User-centered design in games. In: Jacko, J., Sears, A. (Eds.), Handbook for Human Computer Interaction in Interactive Systems. Lawrence Erlbaum Associates, Inc., Mahwah, NJ, pp Sanchez-Crespo Dalmau, D., Learn faster to play better: how to shorten the learning cycle. Retrieved from / com/features/ /shelley_01.htms. Shelley, B., Guidelines for developing successful games. Retrieved from / Shneiderman, B., Designing the Human Interface: Strategies for Effective Human Computer Interaction. Addison-Wesley Publishing Company, Inc, Reading, MA. Song, S., Lee, J., An exploratory study of the critical design factors for MMO game design based on usability evaluation. In: Proceedings of the Conference on HCI2006 Metamorphosis, vol. 2, pp Sudman, S., Bradburn, N.M., Schwarz, N., Thinking About Answers: The Application of Cognitive Processes to Survey Methodology. Jossey-Bass, San Francisco. Van Someren, M.W., Barnard, Y.F., Sandberg, J.A.C., The Think Aloud Method: A Practical Guide to Modeling Cognitive Processes. Academic Press Prof., Inc., San Diego, CA. Wagner, J.A., Showdown in cyberspace: star wars vs. the sims. Salon.com. Retrieved from / 07/09/mmorpg/index.htmlS. Woodcock, B.S., An analysis of MMOG subscription growth. Retrieved from / htmls. Yee, N., Gender and age distribution. The Daedalus Project 1 2. Retrieved from / phps.