Data Cracker: Developing a Visual Game Analytic Tool for Analyzing Online Gameplay

Transcription

1 Data Cracker: Developing a Visual Game Analytic Tool for Analyzing Online Gameplay Ben Medler Georgia Institute of Technology Atlanta, GA benmedler@gatech.edu Michael John Electronic Arts, Inc. (EA) Redwood City, CA MJohn@ea.com Jeff Lane Great Northern Way Vancouver, BC, Canada jefflane@post.harvard.edu ABSTRACT Game analytics is a domain that focuses on the systems and methods used to analyze game-related data. In this paper we present how a visual game analytic tool can be developed to analyze player gameplay behavior. Our tool, Data Cracker, was built for monitoring gameplay in Dead Space 2, the latest game in the Dead Space franchise. We use Data Cracker as a case study to inform a larger discussion of designing a visual game analytic tool while working with a game team. Our design approach focuses on increasing the data literacy of a game team. This means getting an entire team interested and involved with game analytics. We found that building our tool during the early game development cycle, creating multiple early visual prototypes and branding the tool to the Dead Space team caused more team members to become interested in our tool. Increasing interest in analytics is also a means, we argue, for changing the common occurrence within the game industry to disband teams after a game is released. Instead, we promote the creation of live teams which stay attached to a game long after it is release in order to continue the analysis process. Additionally, we discuss the barriers one might face when developing game analytic tools, such as prejudice against analytics or the technical issues involved when collecting large data sets. All of these examples are presented as insights we gained while coupling analytic tool design to game development. Author Keywords Game analytics, visual analytics, information visualization, team communication, game design, player behavior. ACM Classification Keywords K8.0. General: Games; H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION Analytics is currently a hot topic within the game industry, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI 2011, May 7 12, 2011, Vancouver, BC, Canada. Copyright 2011 ACM /11/05...$5.00. driven in part by the extensive use of web-style analytics by the social gaming powerhouses [2, 20], but adoption of analytics in mainstream games tends to be more cocktail conversation than actual ongoing use. Game analytics is a domain that focuses on the systems and methods used to analyze game-related data (also referred to as game telemetry) and therefore can be a large area to cover. Analyzing a game s financial records, development progression and software metrics can all be considered game analytics. Instead, we refer to the type of game analytics which influences how games are designed by analyzing players, an area of analysis that is very prominent among industry professionals [17, 21, 23, 42] and academic researchers, with a good overview in [27]. Monitoring player behavior means logging in-game events, or metrics, such as when a player begins a level or performs an action like jumping while in the midst of gameplay [24]. From a game design point of view, behavior data is useful for determining how players are actually playing a game in their natural environment, versus other evaluation methods like self-reporting surveys or controlled play tests. We have developed a visual game analytic tool, Data Cracker, in order to explore how visual game analytic practices based on player behavior analysis can augment the game design process and seek to use the tool as a case study to instigate a wider discussion of game analytics. Data Cracker is a prototype system built at Electronic Arts (EA), one of the world s largest game publisher/developer, as part of the company s initiative to provide its game studios with analytic tools. In the past many EA studios have built analytic tools for analyzing player data but have typically been focused on recording metrics for monitoring financial prosperity. Only a few visual systems built at EA track player behavior and with a few notable exceptions, prior to development of the Data Cracker, these had largely been side projects initiated by various EA employees who built the systems outside of their normal duties. As a response, the Data Cracker initiative is a formal attempt at EA to innovate and create best practices for building game analytic tools. However, development of Data Cracker was more than an experiment to determine how to best develop a game analytic tool, it was a chance to explore the difficulties that can arise when inserting analytic tools into a creative process such as game design.

2 Data Cracker was built by the authors while employed by EA for an internal EA game studio Visceral Games and deployed to monitor player behavior from the game Dead Space 2 (DS2) [45], the sequel to the critically acclaimed game Dead Space [44]. Our team worked directly with the DS2 multiplayer team in order to help monitor online multiplayer gameplay. This is the first time that a multiplayer component has been offered in a Dead Space game, which is one reason way Visceral s DS2 team was eager to use a tool for analyzing player gameplay behavior. Through our interaction with the DS2 team and the development of Data Cracker we gained many insights related to working with an external game team in addition to the best practices we discovered related to developing a visual game analytic tool. This joint effort format between a game development team and an analytic team will likely continue at EA making the insights we gained valuable for future analytic projects. In this paper we present the insights gained from working with the DS2 team and how the design of the visual game analytic tool Data Cracker shifted as the game, and team, shifted. We begin with a review of other game analytic systems and relate them to the field of visual analytics which revolves around the exploration of data through visualization. In the past, EA teams would typically analyze data using spread sheets, or hastily created web-based visualizations. With Data Cracker we provide the DS2 team with multiple visualizations that help them explore their data from a different perspective and makes data analysis much more accessible. Next, we describe DS2 s multiplayer gameplay and present the final Data Cracker architecture. Using our development cycle as a case study informs the subsequent discussion of important guidelines for implementing a visual game analytic tool as part of a game team s development process. This includes covering the production methods we found necessary to build Data Cracker along with the technological and human barriers we faced when working with a team during their development cycle. Finally, we critique our game analytic insights gained from this project in order separate the idiosyncrasies of our situation from game analytics at large. This will help other game teams gauge how our insights may or may not apply to their game or development process. DEFINING VISUAL GAME ANALYTICS Perhaps the most predominant examples of visual game analytic tools are the services available for games built on the Flash platform. Mochibot [26], Nonoba [28] and Playtomic [29] are three examples of visual analytic services Flash developers can use to monitor player gameplay behavior. In the past, earlier services like Mochibot were used to determine the number of users loading a particular game, which justified ad revenue for game developers who placed ads in their games. These analytic services have expanded now to include features such as player achievements, micro-transactions systems and player behavior tracking through monitoring gameplay events. However, these systems are only available for games built on the Flash platform and act as basic analysis systems similar to other, offline, visual analytic tools like Tableau [39] or Spotfire [37]. Shifting to the PC and console platforms, Zoeller [49] presented an example of a developer-facing visual analytic tool, SkyNet, used by the game developer Bioware. This system focuses on the game development process by monitoring developer behavior rather than players. Bug tracking, software metrics and social networking features are combined to create an online-portal where developers can both stay in touch with one another and have consistent access to their game s development status. While many features in SkyNet use spreadsheets to represent data, visual callouts such as color-coding important information are used, in addition to other visual graphs, to display information such as the number of bugs filed and fixed. Heat maps are also used to denote where crashes occur in a game s space, even giving developers the option to click on the map in SkyNet and be transported to the location within the game environment. SkyNet is an example of how visual game analytics can be used to monitor other game-related data, such as the developer behavior and software metrics. Data Cracker, in contrast, is used to analyze the player s experience, both during development (e.g. player testing and beta tests) and after the game is released. Beyond a single game studio s tools, user research groups at larger game publishers also use game analytic systems for analyzing player behavior. Microsoft s TRUE system [17] is one example. TRUE is a game analytic system built for both user testing and beta testing games. When used for user testing, the system can record player metrics, capture their video output and prompt players with surveys to gauge their attitude. Collecting both video recordings and player attitudinal data help inform the behavior metrics which could otherwise be misleading without proper context. However, video and attitudinal data is only collected within an onsite lab-based setting. When TRUE is used to capture data from beta testers it essential performs in the same way as Data Cracker, only logging player metric data. Additionally, TRUE has means of visualizing the data that is collected and has proved useful for game designers when making decisions on how to alter a game s mechanics. In order to expand on systems like TRUE we seek to bolster the uniqueness and life cycle of game analytic tools on top of features like video recording and data visualization. As we discuss in the following sections, branding tools for specific game teams and promoting data literacy are needed to further advance game analytics. Stepping away from game analytics for a moment, two other larger areas devoted to visualizing and analyze data include visual analytics [40] and information visualization [3, 6, 8, 41, 46]. These growing research areas explore the advantages of visualizing data [46] which include enhancing a person s: comprehension of large datasets,

3 perception of emergent properties, ability to find problems within datasets, and capability to form hypotheses. Vision is not only the fastest and most nuanced sensory portal too the world, it is also the one most intimately connected with cognition, [6] which is why we feel that visual analytics is an important area to pursue when providing game analytic tools for developers. Visual game analytics is also a way to provide greater universal usability [36] and data literacy for members of a game team. From interacting with the DS2 team we found that if analytic tools can be created to provide greater accessibility to more members of the game team those tools become significantly more useful in advancing design discussions amongst the team. Providing visualized examples of gameplay data is less intimidating than handing a game developer a spread sheet of numbers. While there may be specific personal qualities that make someone adept at becoming data analysts [6] there are certainly examples of how casual information visualization can increase a person s literacy of data analysis and allow further reflection [30] that can be beneficial for game teams. DS2 GAMEPLAY & DATA CRACKER S ARCHITECTURE In this section we describe the type of gameplay found in Dead Space 2 and the architecture behind Data Cracker. This information provides context to our discussion of design insights and procedures for working with the DS2 team in the next section. Dead Space 2 Gameplay Dead Space 2 multiplayer gameplay is based in the thirdperson shooter genre and places two asymmetric teams against each other in battle, the human security forces verses an alien team known as the necromorphs (necros). Matches between these two teams consist of two rounds where one team plays as the humans first, the other team as the necros, and these teams switch sides in the subsequent round. Humans and necros are asymmetric teams because each side has their own unique abilities and different goals for winning. For example, the human team has a series of objectives (holding a position, turning on a machine, gathering components) that they must complete within a given timespan, while the necro team attempts to stall the human team until time expires. This makes balancing these two teams a subtle and very challenging task for the game s designers because the diversity of abilities can make one side more powerful than the other causing an unfair game. These and other differences between the teams affected what Data Cracker needed to monitor. The human security team uses an assortment of projectile weapons (i.e. guns) as their primary weapons and each team member has the ability to heal one another with health packs. Humans can take multiple weapons with them into each match which makes monitoring which weapons they choose an important factor. Moreover, players on the necro team must choose from four different classes with fixed abilities and weapons. For instance, the class known as the Pack is fast, lightly armored and only has a short-ranged, melee attack. Necromorph players thus cannot switch between different weapons while playing and instead are forced to choose one of the four classes each time they are killed (before they respawn and re-enter gameplay). Additionally, players gain experience for performing actions during play, some only available for either humans or necros, which lead to further rewards. Normally, gathering data about these various distinctions happens through play testing a limited population, which can be time consuming and difficult to analyze in a statistically significant way. However by recording data telemetrically, data from hundreds of thousands of games can be analyzed. Figure 1. The Data Cracker architecture combines both server-side systems analyzing data from Dead Space 2 and a client-side web interface for data visualization. Recording Players Player events from DS2 multiplayer matches are recorded to monitor how each team is behaving and allows designers to balance the human and necromorph teams. Recording events in DS2 works by adding telemetric hooks or functions into a game s code which send event data to a separate server location whenever an event in the game is triggered. The basic data format that is sent by a hook in Dead Space 2 includes: an anonymous unique user identifier; region information; event identifiers such as an event s name and type; a timestamp of when the event occurred both on the local machine and on the server; and, finally, specific information about the event which are often represented using key-value pairs (i.e. a variable name and its value). In DS2 these key-value pairs hold information related to:

4 Identifying when unique matches start and end. When the human team completes objectives. The position of players as they traverse the play area. When players equip weapons or classes. How much weapon damage players inflict on other players. When players are killed by or kill another player. When players respawn after being killed. Whether a team wins or loses a match. How much experience each player gains from a playing in a match. After events are logged, the server-side portion of Data Cracker begins organizing the collected player data. Data Cracker Server-side Architecture Data Cracker uses a client-server architecture (Figure 1), processing event logs from DS2 on the server-side and presenting a visual analytic tool on the client-side. The server portion organizes and aggregates events together into data formats required for the client-side analysis tool. Beginning with raw text logs containing recorded player events from DS2, each event is stored within a MySQL table as a single element. The organized events are used to create additional aggregated tables representing the data and is similar to an online analytic processing (OLAP) system [5]. OLAPs aggregate large datasets along a single measurement, creating a separate list of multiple dimensions to query the measurement against. In DS2, each weapon type found in the game is an example of a measurement and two dimensions that are created include how long players held a weapon during a match and the amount of damage a weapon caused during a match. Even though each individual weapon event is stored in one table, additional aggregated weapon tables are created as a preprocess to ease the burden of later analysis and improve responsiveness of the tool. The processes through which these aggregated tables are created are called Extraction, Transformation, Loading (ETL) processes. ETLs are pieces of software responsible for the extraction of data from several sources, their cleansing, customization and insertion into databases [43]. ETLs in Data Cracker create aggregated tables, often representing averages and totals, that are separated by day. We found that aggregating based on single days was detailed enough for the DS2 team to gain insight from the data (however this process can be duplicated using shorter or longer time frames). The aggregated data produced by the ETLs are stored in separate MySQL tables which are directly referenced by the client-side website. Figure 2. Screenshot of the Data Cracker client-side website.

5 Data Cracker Client-side Website The client-side website (Figure 1 and 2) is an online visual analytic tool designed using a user-centered approach. Our team worked alongside the DS2 multiplayer team as they developed their gameplay, regularly meeting to design and test the tool (we go into further detail about the design of the tool in the next section). Data Cracker s client-side website visualizes the aggregated data created by the server ETLs using Protovis [31] and jquery [14], both javascript libraries that run on the client s machine. Queries are served by AJAX calls to PHP scripts which grabs data from the aggregated tables in the MySQL database. The basic interface and feature layout of the tool includes: The Summary Graphs at the top of the page show overview values about the collected player gameplay data. This includes how many players have been tracked, how many matches have been played, the winning percentages of the human and necro teams and the number of matches played on each map in the game across time. These summary graphs change whenever the user changes the time frame they are viewing using the timeline. The Timeline is an interactive selection table that displays the range of days which are currently being analyzed. When users click and drag across a range of days every graph in Data Cracker alters itself to display data from that time frame. Once a date range has been selected it can also be dragged to encompass other dates, giving users the ability to always view a fixed number of days (such as a week s worth of data). Data Cracker also allows user to mark events on the timeline, such as acknowledging when a new patch was released for the game, which may affect how a user should interpret data from that time period. The Main Graph area displays the graphs built for analyzing player gameplay data. These graphs display data related to: unique/total users being monitored, kill/death ratios, number of rounds played and won, experience points gained by players, weapon statistics, and objective completion rates. Following the Shniderman visualization mantra of overview first, zoom and filter, then details-ondemand, [35] each data area, for example number of rounds played, has an overview graph which display the total values for the current date range selected. Users can then navigate to more detailed graphs which allow them to zoom and filter the data, for instance data can be separated based on a specific map. Some graphs also offer details-ondemand through the ability to sort values or only displays values as users interact with the graph. Finally, each graph displays a title, its date range and, if necessary, its axes/key values so that screenshots of graphs can be taken and added to gameplay reports, which are often created by game development teams. GUIDELINES FOR DEVELOPING A VISUAL GAME ANALYTIC TOOL Working with the Dead Space 2 team brought to light many important design factors about visual game analytic tool development. In this section we list a number of insights we gained from our development process, each one discussing specific features that should appear within a game analytic tool or how the development process should be structured. Insights are broken into three categories: production, functional and game team integration. Production insights revolve around the goals and development procedures used to build Data Cracker. These production insights are the most novel in comparison to the other game analytic systems covered. Functional insights highlight the actual features and interaction capabilities of the tool while game team integration insights focus on our interactions with the DS2 game team. These insights are perhaps less novel, and more apparent to seasoned analytic tool developers, but are listed in order to reaffirm their necessity while developing a game analytic system. Production Insights Create analytic tools in parallel with game development Commonly, analytic tools are created after a dataset, or the procedure for creating that dataset, is finalized. Building a tool for investigating airline ticket ordering [22], housing prices [48], document analysis [16], or network structures [7] can rely on the datasets being produced in a standard manner (e.g. airline ticket data will not suddenly change formats). However, in these investigative cases analytics tool are used to gain insight from data in order to make decisions which will not affect the data s content [16]. Someone analyzing documents for security threats will not alter the contents of those documents. Metric analysis of player behavior in games, in contrast, can be used as a powerful extension of the traditional design process of iteration and refinement, altering the very nature of the data players can produce. For example, the DS2 team continually added new telemetric hooks and altered mechanics like the types of weapons available in the game, altering the data produced along with those gameplay changes. The consideration and specification of the telemetric data format became a positive feedback loop for the game s design team during this iterative process. Having access to gameplay data forced them to quantify their experiential expectations of each gameplay alteration. Developers therefore can benefit from analytic tools not only after a game is released, where the tools may benefit the creation of patches to make gameplay alterations, but as soon as a team begins testing their gameplay, which generally happens very early in the development cycle. For example, during the DS2 beta test the team was able to use our tool to find an exploit players were using involving a particular weapon called the javelin gun. Using Data Cracker to review the weapons usage percentage and damage dealt the team was able to tweak the weapons power to balance gameplay. At the same time players were also complaining online that the necro team was too weak in multiplayer matches. The DS2 team was able to verify these complaints using Data Cracker and subsequently

6 knew how much to tune the necro class health and damage by analyzing the values visualized by the tool. Produce early visual prototypes Just like with game development prototyping [32], presenting the graphs and analytic features of a tool early is vital to help communication between the analytic team and the game team. Each graph produced for Data Cracker was first prototyped and discussed in our weekly meetings with our DS2 team contacts. The entire process, from events being triggered to visualizing the data, was discussed at our meeting as we prototyped each graph through the development process. Giving the DS2 team high fidelity prototypes that functioned and displayed real data helped the team understand how the tool would be beneficial to them once completed. We also used prototypes to win arguments about adding certain features or what data should be collected. For example, after presenting our graphs for analyzing player experience points gained during matches we were able to show that the points needed to be separated by team type in order to show inequities between the human and necro teams, even though experience points are only awarded after matches where a team had played as both sides. Prototypes brought out these types of situations in our discussions with the DS2 team: thinking of different ways of interpreting the data and what might be missing from our analysis. Build for a Broad Audience Building a game analytic tool during a game development process means the data being collected could affect a large number of members on a game s team. Designers and producers may have to change the game s mechanics which affects programmers who have to code the mechanics and artists who have to create content related to the mechanics. Giving the power to analyze game data to each of these groups during the development process can help teams communicate changes more effectively. Early game analytic tools should thus be designed with the expectation that a broad audience will use them, not just a set of superusers who intuitively understand the game s underlying structure. This means providing tiers of analysis, or ways that users of different levels can access the data. With Data Cracker we achieve this by providing graphs that visualize the same set of data but offer different levels of interaction. For example, users can view one graph that shows the total number of rounds played, a second graph that breaks down the total rounds played by map and the win percentage for each team, and a third set of graphs for each map breaking down the percentage of human teams that completed that map s series of objectives. Game producers may only wish to view the total number of rounds played while a game designer may want to know how often teams are completing a certain objective on a specific map. Performing usability tests on tools are also invaluable in proving and iterating the tool s broad accessibility. We conducted tests both internally with multiple members of the DS2 team and with outside EA employees to gauge how quickly they could understand the interface and features. As a result we added features such as help icons that explain graphs to users and help indoctrinate team members who had never used our tool before. We also focused on using simple visualizations methods like bar graphs and line charts which can decrease data interpretation errors [11]. Certainly, as tool development continues, features that dive deeper into the data can be added but continuing to support the earlier broader features will keep a team focused on the benefits of using analytics. Brand a tool to a particular team Data Cracker could have been visually designed in a generic fashion so any game s data could be connected to the tool and analyzed. Other data visualization software takes this approach [37, 39] and offers users a common interface no matter what data is being analyzed. However, for Data Cracker we took a different approach and branded the tool specifically for Dead Space 2. The main reason for designing the tool in such a way was to make sure that the DS2 team knew that this was their personal tool. Bateman et al. [1] found that users find charts and graphs that are visually embellished (with unnecessary visual elements) to be more enjoyable and help with retention. While most of the graphs in Data Cracker are not embellished with extreme graphics, certain DS2 symbols were added to mark Data Cracker as a DS2 tool. First, the term Data Cracker was actually decided upon because there is a term called Planet Cracking that plays a major role in the lore of the Dead Space franchise. Second, the title graphic for Data Cracker also uses artwork from DS2 depicting the main character and other DS2 artwork is used throughout the tool. Third, the color scheme used in the tool is the same color scheme used in DS2. For instance, a certain shade of blue is often associated with safety in the game and is used to depict the human team s data, while red symbolizes distress and used to show data associated with the necro team. Finally, we also made use of DS2 artwork when ever creating material describing or promoting the tool. Each of these examples works to create a common identity for Data Cracker that links it directly with the DS2 team. Encourage live teams As the final insight into producing a game analytic tool, it is important to encourage the game team to create live teams which will incorporate analytic tools into the development process. A live team is a portion of a game development team that supports a game after it is released. Often times supporting a game after release refers to customer service representatives or development team that create patches or extra downloadable content (DLC). Alternatively, when referring to live teams in relation to game analytics it means a team that continues to analyze the game after release and develop further tools. Their duties should go beyond finding bugs to fix. Live teams should monitor gameplay for months after the game is released

7 keeping track of the ebb and flow of player experiences, altering tools as required. Analysis reports should be created detailing how players are reacting to the game s mechanics and should be shared with other teams creating DLC or working on games within the same genre. Game teams are often dissolved after a game is released which is one leading cause for why game analytics has been slow to take off, there is no one around after the game is released to analyze player reactions. By promoting live teams which stay active after a game comes out it ensures that analytic tools are properly used and have a greater impact on future games produced. Functional Insights Aggregate data for quick analysis Query time is often a big problem when dealing with large datasets where searching through each element can be time consuming [5]. DS2 could have upwards of hundreds of thousands of players after the game is released, each of which is reporting their behavior events back to EA s servers. The common practices of OLAP [5] and ETL processing [43] were implemented in Data Cracker in order to get around this problem. For example, one set of graphs shows how many matches were played on the different multiplayer maps available. The calculations to add the total matches played on every map, each day would have been extremely costly to perform multiple times. As a result, an ETL was produced that added the total matches played on every map, each day and placed in a separate table. Data from thousands of events are combined into a single element making querying and visualization much faster while still leaving those events intact, within another table, for other systems to access or data mine. Use time-based queries Time emerged as a key dimension for almost all the gameplay analytics we came across for DS2. In most cases, time provides the key added dimension to give context to gameplay data which has been the case for other game analytic systems [17]. For the Data Cracker tool, the importance of time as a data dimension is why the timeline is the highest-order interactive element of the tool (affecting all graphs in the tool), and why it received the most attention in terms of usability. Our initial designs included two text boxes that marked the start and end dates of the time frame shown on each individual graph. This eventually changed to a single set of boxes that governed the time frame for all graphs in the tool because the DS2 team wanted to compare different graphs within a single time period without having to set each graphs time frame individually. After more iterations, the time frame was shown on an interactive timeline making it easier to select a date range, move the range to different dates, as well as mark the timeline with key events that could affect the interpretation of the data (e.g. when downloadable content, DLC, is added to the game and may affect player behavior). Debug the game and data hooks with the tool The creation of telemetric hooks to track player behavior is a software process requiring the use of an API and as such is as prone to error as any other part of the software. Insuring that event hooks accurately represent player behavior is a key part of the analytic system development process [15]. Having game analytic tools functioning early within the development cycle allows another avenue to debug both the metric system and the game. At one point during Data Cracker s development, for instance, we started receiving additional weapon related events from players who had already completed a match, events that should not have been occurring. When we reported this to the DS2 engineers it was found that the game was creating multiple copies of each player and each was sending telemetry data. The engineers were quickly able to rectify the mistake thanks to our tool. If that data had been allowed to persist our aggregated calculations would have been skewed, affecting how the data was to be interpreted. Create functionality to deal with legal Issues When game companies collect player data they have to follow different rules depending on the geographic location of the player. Compliance with these rules is extremely important in the current climate where online privacy is heatedly being debated [4, 19]. EA, for instance, cannot collect data from players in over 100 different countries. In North American countries like the U.S., Canada and Mexico users must agree to a Terms of Service document in order for the company to track data. Other countries have an opt-out policy where data is collected by default but can be turned off by the player. Procedures within an analytic system must be in place to handle these different geographic categories. Another legal issue surrounding recording player data is keeping the player s identity safe. An event tracking system has to know which unique user an event is coming from in order to maintain an accurate depiction of the events that took place in a game. However this must be stored in such a fashion that the identifier (in our case, an integer) does not identify the player personally. Data Cracker, working in concert with EA s centralized online infrastructure, maintains compliance with both the geographic and identity related restrictions. Game Team Integration Insights Meet with an interdisciplinary team Interdisciplinary teams are a necessity within the game industry as much as they are within other design and media professions [18]. Game analytics too is an interdisciplinary practice and each game development discipline use analytics for different purposes [25]. While developing Data Cracker we regularly met with a team consisting of the lead programmer, producer and designer on the DS2 multiplayer team. Each was invested in the tool differently. Our main programmer adding the telemetric hooks into the game needed to know what data we wanted to collect and voiced his own opinions on what data was possible to

8 collect. The lead producer and designer were more interested in how the tool was going to help them balance the game s mechanics. Meeting with an interdisciplinary team meant we were both able to make relevant design choices that supported multiple members while also promoting the transfer of game analytic knowledge between the members themselves. This helped bypass common interdisciplinary team communication barriers [9] by allowing our DS2 team members to understand how the tool functioned and everyone s analytic needs. Prepare for analytic prejudices Collecting player data is certainly not a new concept and EA already had an entire infrastructure built for monitoring player events. One persistent problem, however, was many game teams incorporated data collection but never analyzed the data. Understandably, these meant some Dead Space 2 team members were skeptical about our promise to provide a tool that would make use of player data, because they had seen how ineffective previous efforts had been. This type of prejudice against a new procedure or technology is common in other research. Herbsleb et al., for example, had similar experience while adding an instant messaging technology to a distributed team [12]. In our case, this prejudice made development difficult because we relied on the team to provide us with the necessary gameplay information and telemetric hooks that sent a player s data to our server. In order to get around this opposition we create example scenarios of how each data point would be used and how they would inform the team. For example, showing how increased fidelity of the experience point data would offer a better picture of which team, humans or necros, were achieving certain experience point events more often. Anticipate the effects of game design changes Development of Data Cracker was happening at the same time as the DS2 multiplayer gameplay was being developed: if the gameplay changed, the analytics had to change. Weekly meetings with our DS2 contacts were essential and kept us informed about changes to the gameplay. Most of these changes were cosmetic (e.g., maps or weapons being added or taken out) and only affected how certain data points were labeled. However, at one point the DS2 team limited human players to only two weapons and were forced to choose one specific weapon. This radically altered how often players switched weapons and the variety of weapons they had to choose from during play. Analyzing weapon data had to change as a result and while the graphs didn t change visually the interpretations of the graphs changed. Tracking those interpretation changes was vital for keeping the team updated about our progress and, in the future, would help anyone attempting to analyze DS2 data with our tool as well. Update the team regularly While staying connected to the DS2 multiplayer team helped us keep track of gameplay changes it was also important to stay connected with the DS2 team as a whole. Simple ways of doing this was to talk with other DS2 team members whenever possible and to participate on internal lists, generally acting like a fellow team member. The other major way we connected with the DS2 team at large was distributing a weekly data analysis report that covered the progress of Data Cracker and interesting information we gleaned from analyzing DS2 player data. Like Data Cracker, these weekly reports were DS2 branded, often incorporating artwork from the game, and were designed to look handmade to distinguish them from standard analysis reports that game teams receive from marketing or management departments. Each issue had a certain theme based on an important feature added to Data Cracker or a specific anomaly we found in the data that week. One weekly issue focused on the Fourth of July holiday weekend, for example, where no data was logged from DS2, meaning everyone successfully stayed away from the office. We often received comments that these reports kept people interested in the tool even if they were not affiliated with the DS2 multiplayer team or DS2 in general. DISSCUSSION Data Cracker has been in service since August There are current plans to continue iterating on the design of the tool and to provide each team within Visceral Games with their own version. The tool has already proved useful by helping find bugs in DS2 s multiplayer environment, allowing analysis of player tests and through our efforts in promoting the tool has caused other EA teams to begin designing their own analytic systems. The DS2 single player team has also used the Data Cracker s systems and web services to create complex analytics for their in-house play tests, over and above our work. While Data Cracker has been a useful tool for the DS2 team, further critiques and evaluations are required to understand how game analytics combines with the game development process. Our insights reflect the design properties we focused on while building Data Cracker: accessibility, branding and longevity. An entire game team should have access to analytic tools throughout the game development process. Systems like TRUE seem to be particular to only user research teams [17]. We instead argue each game team must have basic data literacy skills and be able to use analytic tools as the amount of data games collect and use increases [24]. One way to raise data literacy is through branding tools for specific teams. The Skynet system [49] is an example that demonstrates how a tool can create a community around game analytics. With Data Cracker, we sought to bring a semblance of that community feature by branding our tool to the DS2 team which boosted the team s interest in the tool. Finally, analytic tools should be built to work across the entire development process. Analytics should represent a reliable means of testing a game and live teams can be created to handle the analysis of data after a game is released. A regular occurrence in the game industry is to dissolve a team after completing a game. However, the new industry emphasis on supporting games after their release (with DLC, patches and online player dossier

9 systems [25]) demands live teams as necessary for postrelease content development. There are also limitations to some of the insights we presented. First, building a tool for broad audiences and aggregating data would not be ideal for developing data mining analytic tools. Data mining player events would require different methods [47] and would likely be built for users with special analytic or statistics training [6]. Second, creating branded systems and establishing live teams may create significant cost overhead that game studios are not prepared to invest. While we argue that game analytics are vital, worth the initial investment, others question the overzealous use of metric-driven design [10, 13]. In contrast to the limitations of our insights, there are a number of avenues where our insights can be expanded upon in the future. Data Cracker was built to promote data literacy among development teams and as their literacy rises developers will be prepared for more robust analytic systems. Adding in data mining functionality for more automatic analysis, such as player modeling [33], could increase the complexity of the available analysis that a tool provides to developers. Other visual representations can be added as well. For instance, player events happen within a hierarchy (organized by map, level, play session, etc.) and graphs such as sunbursts [38] or treemaps [34] work well for displaying hierarchical data. Finally, there has been an increase in recent years in the number of game analytic systems that are provided to players instead of only game developers [25]. Future game analytic systems may combine developer-facing and player-facing systems with each audience given different analytical functionalities. CONCLUSION Data Cracker is a tool for monitoring player behavior by tracking and organizing in-game player events. The tool was built using a client-server model where ETLs are used to aggregate data on the server-side and a client-side game analytic website visualizes the collected data with a series of interactive graphs. Similar systems have been built before, such as the TRUE system [17], but unlike other game analytic tools our design approach for building Data Cracker was to increase the Dead Space 2 team s data literacy. As digital games increasingly utilize player data for gameplay and as part of their development process it is not enough to design for small subsets of superusers who understand the importance of data analysis. Therefore we set out to build an analytic tool for monitoring player behavior that is accessible to an entire game team. The insights we gain from our design and development process were split into three separate categories: production, functionality and game team integration. Through our production process we found that building our tool in parallel with the game development cycle and having early visualization prototypes familiarized the DS2 team with Data Cracker. We used tiers of analysis to lay out visualizations offering different levels of detail so team members were not inundated by too much data at once. Branding the tool to Dead Space 2 also helped identify the tool as part of the DS2 team and created further interest. For Data Cracker s functionally we built a system that aggregated data across different dimensions including time, in-game maps and gaming platform. This meant the tool was quick to respond to queries further lowering the barrier for team members to access their data. The tool had to also follow a number of legal requirements since a broader audience would have access to potentially sensitive data. Finally, we had to maintain communications through our interactions with the game team. Initially we were met with prejudice against analytic tools and had to watch out for unexpected changes made to the game that affected our tool s functionality. We were able to minimize these problems by continually meeting with an interdisciplinary team working on DS2 to present functional prototypes that showed the effectiveness of the tool and determine if any game changes would affect the design. The insights into designing these tools and working with game teams which we present in this paper are only the beginning to what is capable within the domain of game analytics. As a domain that is still growing game analytics shows great potential when used to augment the game design process. It is now possible for game designers to support their design intuitions with quantitative measurments, allowing for quicker iterations during the development cycle. Analytic tools can monitor millions of players after a game is released giving developers data they never had the capabilities of accessing. Developers need to continue to discuss their analytic processes and pursue creating permanent live teams within their company to work as tool developers and gameplay analysts. ACKNOWLEDGMENTS The authors would like to thank Scott Probst, Simon Cooper and the entire Dead Space 2 multiplayer team for their support of this work. REFERENCES 1. Bateman, S., Mandryk, R. L., Gutwin, C., Genest, A., McDine, D. and Brooks, C. Useful Junk? The Effects of Visual Embellishment on Comprehension and Memorability of Charts. In Proc. CHI (2010). 2. Caoili, E. Zynga Chooses Tableau For Data Visualization, Analysis. Gamasutra. (2010) 3. Card, S., Mackinlay, J. and Shneiderman B. Readings in Information Visualization: Using Visualization to Think. Morgan Kaufmann (1999). 4. Carvin, A. Debate Continues Around Facebook Privacy Changes. NPR (2010) /npr-listeners-react-to-facebook-privacychanges 5. Chaudhuri, S. and Dayal, U. An overview of data warehousing and OLAP technology. ACM SIGMOD Record, 26, 1 (1997),

10 6. Few, S. Now You See It: Simple Visualization Techniques for Quantitative Analysis. Analytics Press (2009). 7. Freire, M., Plaisant, C., Shneiderman, B. and Golbeck, J. ManyNets: An Interface for Multiple Network Analysis and Visualization. In Proc. CHI (2010). 8. Fry, B. Visualizing data. O'Reilly Media (2008). 9. Haythornthwaite, C. Knowledge Flow in Interdisciplinary Teams. In Proc. System Sciences 2005, IEEE (2005). 10. Hecker, C. Achievements Considered Harmful?. GDC (2010). 11. Heer, J. and Bostock, M. Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In Proc. CHI (2010). 12. Herbsleb, J., Atkins, D., Boyer, D., Handel, M., and Finholt, T. Introducing instant messaging and chat in the workplace. In Proc. CHI 2002, ACM Press (2002), Jamison, P. FarmVillains. SF Weekly (2010). 08/news/farmvillains/. 14. jquery Kaner, C. and Bond, W. Software Engineering Metrics: What Do They Measure and How Do We Know? Software Metrics Symposium, IEEE (2004). 16. Kang, Y., Gorg, C. and Stasko, J. Evaluating Visual Analytics Systems for Investigative Analysis: Deriving Design Principles from a Case Study. In Proc. VAST (2009). 17. Kim, J., Gunn, D., Schuh, E., Phillips, B., Pagulayan, R. and Wixon, D. Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems. In Proc. CHI 2008, ACM Press (2008), Kim, S. Interdisciplinary Cooperation. In Laurel B. (Ed.) The Art of Human-Computer Interface Design (pp ). Addison-Wesley, Reading, MA, (1990). 19. Kincaid, J. This Is The Second Time A Google Engineer Has Been Fired For Accessing User Data. Techcrunch (2010) Kontagent Lazzaro, Nicole (2007). The 4 Most Important Emotions of Game Design. GDC (2007). 22. Liu, Z., Stasko, J. and Sullivan, T. SellTrend: Inter- Attribute Visual Analysis of Temporal Transaction Data.In Proc.Infovis (2009). 23. Ludwig, J. Flogging: Data collection on the high seas. Austin GDC (2007). 24. Medler, B. Generations of Game Analytics, Achievements and High Scores. Eludamos Journal for Computer Game Culture, 3, 2 (2009), Medler, B. and Magerko, B. Analytics of Play: Using Information Visualization and Gameplay Practices for Visualizing Video Game Data. Parsons Journal for Information Mapping, 3, 1 (2011). (in press). 26. Mochibot Nacke, L., Drachen, A., Kuikkaniemi, K., Niesenhaus, J., Korhonen, H., Hoogen, W., Poels, K., IJsselsteijn, W., and Kort, Y. Playability and Player Experience Research. In Proc. DiGRA (2009). 28. Nonoba Playtomic Pousman, Z., Stasko, J., and Mateas, M. Casual Information Visualization: Depictions of Data in Everyday Life. IEEE Transactions on Visualization and Computer Graphics, 13, 6 (2007), Protovis Rollings, A. and Morris, D. Game Architecture and Design: A New Edition. New Riders (2004). 33. Sharma, M., Ontañón, S., Mehta, M. and Ram, A. Drama Management and Player Modeling for Interactive Fiction Games. Computational Intelligence, 26, 2 (2010), Shneiderman, B. Tree visualization with tree-maps: 2-d space-filling approach. ACM Transactions on Graphics, 11, 1 (1992), Shneiderman, B. The eyes have it: A task by data type taxonomy for information visualizations. IEEE Visual Languages, (1996), Shneiderman, B. Universal usability. Communications of the ACM, 43, 5 (2000), Spotfire Professional Stasko, J. and Zhang, E. Focus plus context display and navigation techniques for enhancing radial, space-filling hierarchy visualizations. In IEEE InfoVis, (2000), Tableau Desktop Thomas, J. and Cook, K. Illuminating the Path: The Research and Development Agenda for Visual Analytics. IEEE Computer Society (2005). 41. Tufte, E. The Visual Display of Quantitative Information. Graphics Press (1983). 42. Valve Corporation. Steam & Game Stats Vassiliadis, P., Simitsis, A. and Skiadopoulos, S. Conceptual modeling for ETL processes. In Proc. 5th ACM Workshop on Data Warehousing and OLAP. (2002). 44. Visceral Games. Dead Space. [PC], USA: Electronic Arts. (2008). 45. Visceral Games. Dead Space 2. [Xbox360], USA: Electronic Arts. (2011). 46. Ware, C. Information Visualization: Perception for Design. Morgan Kaufmann (2000). 47. Weber, B. and Mateas, M. A Data Mining Approach to Strategy Prediction. IEEE CIG Symposium, (2009). 48. Williamson, C. and Schneiderman, B. The Dynamic HomeFinder: Evaluating Dynamic Queries in a Realestate Information Exploration System. In Proc. of R&D in Information Retrieval. (1992) Zoeller, G. Development Telemetry in Video Games Projects. GDC (2010).