High-throughput scientific research with multiplayer online games

Transcription

1 High-throughput scientific research with multiplayer online games Thesis proposal Jeehyung Lee Computer Science Department Carnegie Mellon University October 2013 Thesis Committee Adrien Treuille Niki Kittur Ariel Procaccia Rhiju Das, Stanford University

2 Table of Contents 1. Introduction Related work Early games with purposes Amazon Mechanical Turk Biotic games Scientific discovery games Eterna Problem definition Method Game design Experiments and results Formalizing player intuition Conclusion Future work: Cloud Lab Problem definition Method Evaluation Timeline Conclusion... 19

3 1. Introduction From the ancient study of planetary bodies to modern biochemistry, science has always been an endeavor fundamentally relying on intuition and creativity. As such, passionate non-experts could help making scientific advances. Prominent polymaths throughout the history had already demonstrated such potential. Nevertheless, modern science has walled off further public engagement with the long and difficult educational path required for a scientific career. Even though the public interest in science grows every day, no practical way has existed for non-experts to participate. There has been no channel to translate such massive human intelligence into scientific outcomes. Figure 1 Evolution of citizen science games. (a) ESP game attracted players to work on image labeling. (b) Foldit challenged players with one of the most complicated computational problems - protein folding. (c) Eterna, integrated a wet-lab pipeline into its gameplay to solve an experimental problem - RNA design. (d) Cloud lab dramatically increases expert involvement in Eterna by transforming the game into an open research framework that any expert scientist can use. Recent few years, however, have witnessed the emergence of a potentially powerful channel to connect science and non-experts: online gaming. People all over the world spend 3 billion hours playing extremely complicated games every week (McGonigal 2010). By reformatting scientific tasks into a game-like interface with a proper incentive scheme, early games such as ESP game (Figure 1a) and Galaxy Zoo (Ahn and Dabbish 2004; Land et al. 2008) had thousands of players voluntarily work on 1

4 image annotation tasks. The excitement was followed by a second generation of scientific discovery games that involved players in non-trivial computational problems that require skills beyond visual recognition. Foldit (Figure 1a) is the most prominent example. The game asked players to solve protein folding, an NP-complete optimization problem (Berger and Leighton 1998) on an energy function whose input is a 3D protein structure and output is the protein s free energy. Foldit converted this high dimensional search into visual puzzles which players could attempt to solve using simple click-and-drag interface. In a series of controlled experiments, many players successfully found better protein folds than state-of-the-art algorithms (Cooper et al. 2010) and proved gamers capabilities for scientific problem solving. Such successes marked the emergence of citizen science as a viable new approach to advance scientific problems. However, these successes also raised natural questions: can games change science? Games proved to be useful for annotating images or solving hard optimization problems, but can they perhaps help in wet-lab science as well? Only a small number of scientists have had a chance to utilize games, but can we allow more people to benefit from them? Most importantly, can games become a general research method for rapid scientific progress? Previous games have demonstrated substantial potential, but there are many different scientific problems the games have yet to address. This thesis specifically focuses on the two most fundamental limitations that confined previous games application domain and scientific outcome. Purely in silico (computational) approach. The first limitation is that the previous games operated in a purely in silico domain. Mature scientific problems such as protein folding enjoy accurate in silico simulation models that have been experimentally certified (Rohl et al. 2004). For such problems, solutions that players found within the simulations (i.e., games) serve as credible evidences for their in vitro (in test tubes) counterparts. However, many younger problems in experimental science such as RNA structure design do not enjoy reliable simulation models. Any solution found with the simulations in such fields requires thorough experimental verifications, thereby making in silico games infeasible. This limitation prevented application of games to experimental problems which constitute a substantial portion of scientific research. Limited scientific expert involvement. For players to contribute to science, they need interesting scientific problems distilled into playable puzzles. The second limitation is that scientific distillation process was limited to a small circle of experts directly involved in the game development (i.e., game developers). For example, Foldit required the developers to pick out an interesting protein relevant to their research and distill it into a well-defined in-game puzzle with proper constraints. This process essentially limited the game s scientific outcome to developers on-going research (Cooper et al. 2010; Eiben et al. 2012; Khatib, Cooper, et al. 2011; Khatib, DiMaio, et al. 2011). Despite the vast amount of player manpower, this bottleneck in the expert inputs considerably limited the scientific output of the games. This thesis focuses on exploring and designing a game framework that can (1) broaden the scope of scientific gaming to experimental science and (2) increase expert involvement to achieve a dramatically higher throughput. Two games were designed in series to achieve each goal: Eterna, an RNA design game with wet-lab integration applies scientific gaming to experimental science. Its sequel, 2

5 Cloud Lab, systemically invites external scientific experts to bring in their research projects and collaborate with players. Eterna (Figure 1b and Section 3) applies scientific gaming to experimental science by integrating wet-lab trial-and-error as its central gameplay. As a model problem we look at RNA design for which the goal is to find RNA sequences that fold into a given target structure. Because the in silico model of RNA is notoriously crude, a solution sequence must be verified by synthesizing it in the test tube and chemically inspecting its structure. As such, in Eterna, designing an RNA sequence in the game (in silico) only serves as the first step of its gameplay. Every week player designed sequences are synthesized in test tubes, analyzed and published back to players in gamer-friendly format. Players study the results and design another RNA sequence and repeat the process until they finally find solutions that work in vitro. The wet-lab integration saw a practical success. Players were able to design experimentally validated RNAs with novel structures every week, which is an unprecedented outcome for any research lab. The wet-lab integration also opened up opportunities beyond puzzle solving - players were able to formalize and experimentally validate rules for creating robust RNA designs. The game successfully broadened the application of scientific discovery games to experimental science. Cloud Lab (Figure 1c and Section 4) is an expansion of Eterna that aims to dramatically increase scientific expert involvement by transforming the game into an open research platform where any expert can propose his/her research projects for player participation. The game focuses on the same RNA design problem and uses the same gameplay with the integrated wet-lab. However, expert involvement is no longer limited to the game developers. In Cloud Lab, external scientists are motivated to run and share their research projects with players in exchange for built-in experimental resources and player manpower. Players on the other hand get chances to participate in an unprecedented number of state-of-the-art science projects. By making both ends of the game - puzzle creators (experts) and solvers (players) - public, Cloud Lab aims to be a truly highthroughput research channel. The game is still in the prototyping phase - however, initial tests over the past month have already generated great excitements in more than 10 different university research labs and players. This thesis is structured as follows: the Related Work section summarizes previous citizen science games and discusses their impacts on the field. The Eterna section describes the game design, experiments, and results in detail. The Cloud Lab section describes detailed plans and timeline for the game. Finally, the conclusion section summarizes the thesis and discusses the future of citizen science games. 3

6 2. Related work Citizen science has only emerged in the past few years. However, excitement about its potential has drawn considerable attention and led to creation of many games with various goals. This section reviews some of the important games and discusses their impact. 2.1 Early games with purposes Figure 2. Early image annotation games. (a) ESP game asks players to collectively label images. (b) Peekaboom asks players to annotate objects in images. (c) Galaxy Zoo challenges players to classify galaxies based on their shapes. Citizen science began with games with purposes. Most of these games took the simplest form possible. They leveraged people s innate ability of visual recognition and created image annotation games that could be played immediately without training. ESP game (Figure 2a) crowd-sources general image labeling (Ahn and Dabbish 2004). When users log in to the game, the game matches them into random pairs. The pairs are blind and cannot communicate with each other. The game then shows the pair a set of images and asks each of them to label the images with words. The pair is scored based on how many of the same words they assigned to the images. ESP game has generated more than 15 million labels. The same group later introduced a sequel, Peekaboom (Figure 2b) (Ahn, Liu, and Blum 2006) which again deploys pair-based gameplay to locate and label different objects in given images. Galaxy Zoo (Figure 2c) asks players to categorize galaxies based on their appearances. When users log in to the game, they are presented with a set of images and multiple choices to define the shapes of galaxies in the images. Players are scored based on the number of images they classify. Each image is presented to multiple users to ensure reliable classification. The game accumulated over 10 million classified galaxies and successfully yielded various projects based on the data (Land et al. 2008; Schawinski et al. 2010; Smith et al. 2011). These games demonstrated the capabilities of massive gamers for simple tasks. They proved that games can indeed attract crowd to work on useful tasks and laid the basis for the next generation scientific discovery games, which require significantly deeper scientific understanding and problem solving skills. 4

7 2.2 Amazon Mechanical Turk Amazon Mechanical Turk is an online marketplace where anyone can post programmable tasks for cheap manpower. The system is not a game but nevertheless demonstrated the power of online manpower along with the games with purposes. It works as follows. Requesters post jobs - from simple image/text recognitions to different types of surveys - with price per task and massive Workers take on the job to get paid. Workers can demonstrate their qualities through the number of jobs they successfully finished. The service generated excitement in many sectors from small businesses to research labs. With careful management from project admins, it proved to be a powerful solution for research that require human interactions on a massive scale (Kittur, Chi, and Suh 2008). Further study even proposed crowd-sourcing the project workflow management itself (Kulkarni, Can, and Hartmann 2012). The system inspired Cloud Lab (Section 4) which creates an open research platform for RNA researchers to share their projects in exchange for experimental resources and manpower. 2.3 Biotic games Figure 3. Biotic games. Biotic games manipulate living microorganisms with electrodes and provide unique user experiences. Biotic games (Riedel-Kruse et al. 2011) are unique games that manipulate living microorganisms, for example implementing Pac Man using paramecium as a substrate (Figure 3). They directly control paramecium with electrodes and capture their movement using a set of LEDs and a webcam. The games held different purposes from other citizen science games. Biotic games aimed to provide exciting educational experiences and also demonstrate the prospects of using such games in bioengineering. They are also one of the first games along with Eterna (Section 3) to connect games with a wet-lab. Biotic games, however, focus on providing players real-time hands-on wet-lab experiences whereas Eterna focuses on connecting players with a remote wet-lab for solving RNA design problems with rapid trialand-error. 5

8 2.4 Scientific discovery games Figure 4. Scientific discovery games. (a) Foldit challenged players with the protein folding problem which is considered a holy grail of computational biology. (b) Phylo challenged players to solve multiple sequence alignment problems. (c) EyeWire challenged players to find neurons in 3D maps of the retina. Citizen scientists eventually sought to test gamers potential for even more difficult problems. This led to emergence of second generation scientific discovery games. Previous games relied on innate human abilities and could be played without prior training. Scientific discovery games, however, focused on significantly more complicated problems that required skills beyond innate abilities and deeper scientific understandings. Foldit (Figure 4a) was one of the most successful pioneers in scientific gaming. The game challenged players with a protein folding problem, a holy grail in computational biology. The goal of the problem is to find a natural 3D structure of a given protein that minimizes its energy function. To engage players in this extremely complicated problem, Foldit deployed in-game tutorial puzzles, projected the high dimensional search into a visual problem with intuitive clickand-drag controls, and put players into a real time competition. In a series of controlled experiments, players overall performed comparably to state-of-the-algorithms and were exceptionally better on problems that required packing certain parts of a protein into the structure s core. While the algorithms couldn t stochastically sample the correct structure, players were able to visually reason the correct structures and fold proteins into them (Cooper et al. 2010). The players also subsequently contributed to discoveries of two novel protein structures (Eiben et al. 2012; Khatib, DiMaio, et al. 2011). Phylo (Figure 4b) challenged players with multiple sequence alignment problems (Kawrykow et al. 2012). Players were provided with tutorials and a click-and-drag interface to align sequences in multiple rows to maximize various alignment metrics. The game asked players to improve solutions computed by state-of-the-art algorithms. Phylo collected over 350,000 manual alignments to 739 puzzles. Players successfully improved the alignments for 70% of the puzzles. EyeWire (Figure 4c) challenged players to discover and connect specific neurons in the retina (Seung 2012). The game presents players with a stack of 2D microscopic images each of which is a slice of a 3D map of retinal neurons. Players browse through the images, detect, and connect pieces of the target neuron. These connected pieces in images are used to deduce a 3D structure of the neuron. The game collects at least 5 different player submissions for each neuron to ensure reliable detections. The game does not have an official report of its achievements yet, but shows great potential with its intuitive gameplay and massive user base (over 55,000 players). 6

9 Such success cases demonstrated that games can guide gamers with no scientific background to solve complicated scientific problems and dramatically heightened expectations for citizen science. With carefully designed in-game tutorials and intuitive visualizations, the games successfully guided players to outperform the most sophisticated algorithms. However, the games also had two limitations: 1) their application was limited to in silico problems and 2) their research throughput was essentially bottlenecked by the number of research problems that a small number of game developers could propose. The following sections present two new games that address each limitation respectively. Eterna (Section 3) extends the application domain of scientific discovery games to experimental problems and Cloud Lab (Section 4) systemically invites external scientific experts into the game for a significantly higher throughput. 7

10 3. Eterna Scientific discovery games such as Foldit successfully demonstrated the potential of gamers for solving difficult computational problems. However, many younger scientific domains do not enjoy a reliable and empirically validated computational model like protein folding. Such fields must rely on experimental feedback to make up for the absence of high-quality simulation models. Eterna solves this in principle by integrating wet-lab experiments into the game and scoring players based on these results. As a model problem we look at RNA design. Figure 5. Eterna gameflow. (a) Each week, players design sequences that fold into a target RNA structure in the sequence design interface. (b) Players review and vote for the best designs with the voting interface. (c) At the end of the week, the eight top-voted sequences are synthesized and verified by single-nucleotide-resolution chemical reactivity measurements. (d) The experimental results are published back in the game such that players can review them in the results viewer. Players repeat this cycle (round) until a successful in vitro solution is found. 3.1 Problem definition The goal of RNA design is to find sequences that fold into a given target structure under thermodynamic rules. The problem is not only computationally hard, but it is also impossible to solve the problem with a purely in silico approach. The computational energy model for RNA can only crudely approximate the 8

11 behavior of RNA. To design RNA that folds correctly in vitro, trial-and-error with experimental validation is essential. 3.2 Method Eterna proposes a game model with wet-lab integration at its core to address the problem. Eterna is a web browser game that combines an interactive interface for modeling biomolecules with a high-throughput RNA synthesis pipeline (Figure 5). A web-based interface challenges players to design (Figure 5a) and rank sequences (Figure 5b) that will fold into a target structure when synthesized in vitro. Highthroughput synthesis and structure mapping measurements (Figure 5c) assess 8 community-selected designs per week. Eterna returns these experimental results to players through visualization of the data (Figure 5d) as well as an overall synthesis score on a scale of 0 to 100 (experimental error ±5;). Since its public launch on January 2011, Eterna has gained over 110,000 players and successfully designed over 400 different novel RNA structures. 3.3 Game design Eterna is an on-line Adobe Flash application that can be directly accessed within any web browser. The main interface for sequence design (Figure 5a) is a game-like environment that allows players to mutate & design RNA sequences by clicking on them. The structure display updates in real-time predicted structures of sequences using ViennaRNA package (Hofacker 2004). The voting interface (Figure 5b) allows players to browse through each other s designs and vote on sequences that are most likely to succeed in vitro. The results viewer interface (Figure 5d) visualizes experimental data in a format similar to the sequence design interface. Every week, players submit designs, vote on top designs to be synthesized, and review experimental results using the three interfaces. The cycle which is called a round repeats until players find a sequence that correctly forms the target structure in vitro. To familiarize players with no prior scientific knowledge into the complicated experimental cycle, Eterna provides a step-by-step course. First, six tutorial puzzles introduce RNA design without assuming prior scientific knowledge. This entire series can be completed in 5-10 minutes. Second, challenge puzzles teach advanced RNA design concepts; each challenge can be solved in approximately 1 to 10 minutes. Both tutorials and challenges are in silico puzzles without experimental validation and are designed to familiarize participants with RNA design. Participants are rewarded with puzzle points for clearing each tutorial and challenge. A participant must then achieve 10,000 puzzle points to participate in real competitions - in vitro challenges. The entire course takes approximately 12 hours on average. 3.4 Experiments & Results Eterna ran approximately 1 year of controlled experiments to evaluate players abilities to solve in vitro problems. The initial six-month training period, called Phase I, saw the Eterna community engaging in six RNA design problems containing more complex topologies (Figure 6). 189 community-chosen sequences were synthesized, along with 65 sequences from RNAInverse (Hofacker) and NUPACK (Zadeh et al.) algorithms (state-of-the art in the field) for comparison. Initially, the community was inexperienced and their designs depended solely on computational predictions (Mathews et al. 1999). These designs fared 9

12 poorly: during the first lab competition (the three-helix Finger ; Figure 6a), many player designs gave synthesis scores lower than 70, compared to greater than 90 for all NUPACK designs (Table 3). However, as the community gained experience with empirical RNA design cycles, performance improved and community submissions converged to successful designs (above 90) in 2 to 3 rounds for this and all subsequent targets (Figure 6a-f). Figure 6. Phase I results. The top panel shows a summary of synthesis scores for designs from the RNAInverse (black) and NUPACK (gray) algorithms compared to Eterna players (blue). Squares and dots indicate sequences synthesized in two different chemical conditions - 1 M NaCl (circles) and 10 mm MgCl2 (square).the colored border lines connect designs within the same round. The bottom table shows the best scoring RNA designs and experimentally estimated structures from each design agent. 10

13 a. First half of Phase I (puzzle 1, 2 and 3) Participants RNAInverse NUPACK Participants b. Second half of Phase I (puzzle 4, 5 and 6) NUPACK RNAInverse Players NUPACK c. Phase II (puzzle 7-15) EternaBot NUPACK RNAInverse Players EternaBot NUPACK Table 2. p-value table. Pair-wise performance comparison of all tested design agents. Each table cell reports p value that the row agent performs better than the column agent in their 75% quantile designs. a. Max scores from different agents for Phase I and II puzzles Players RNAInverse NUPACK EternaBot b. Median scores from different agents for Phase I and II puzzles Players RNAInverse NUPACK EternaBot c. Mean scores from different agents for Phase I and II puzzles Players RNAInverse NUPACK EternaBot Table 3. Performance of all tested design agents. The max, median, and mean scores of all design agents. Puzzles 1-6 correspond to Phase I correspond to Phase II. Beyond this target-specific learning, synthesis scores from the first round of each new target increased over time, suggesting that the players were developing generalizable design rules. Over all six targets, these first-round scores increased continuously. By the third target (Figure 6c), players outperformed both RNAInverse and NUPACK in their first-round maximum score. By the fifth and sixth targets (Figure 6e and 6f), first-round median player scores exceeded the algorithms maximum scores, with top player 11

14 designs achieving scores indistinguishable from perfect designs, given experimental error (> 95). In contrast, the increasing structural complexity (measured in stems and junctions) led to declining performances for RNAInverse and NUPACK (Figure 6). First-round designs from Eterna players were significantly better than designs from RNAInverse and NUPACK in the last 3 puzzles, with p values of against both algorithms (Table 2b). In the subsequent testing period, called Phase II, nine new targets challenged Eterna players and prior algorithms (Figure 7). The targets (Figure 7a-i) were multi-junction structures distinct from each other and from the Phase I structures in topology. We evaluated only one round of player designs per target, thus testing whether community knowledge was generalizable across target structures. We again observed superior performance of the player designs compared to RNAInverse and NUPACK. Players first round designs significantly outperformed the automated designs in max, median and mean scores (p = and , respectively; see Table 2c and Table 3) in all puzzles. The results confirmed the players capabilities at in vitro RNA design with a remote wet-lab. 3.5 Formalizing player intuition During Phase I, we noticed that the players formulated and shared their own strategies for designing successful RNAs. We formalized this convention of strategy sharing as a design rule competition in which players were asked to formalize their strategies into design rules (Figure 8). By the end of the testing period, the collection of design rules grew to 40 contributions, most of which encoded novel insights into successful RNA design. We sought to evaluate the rules independently from Eterna players, via their integration into a new energy function, incorporation of this function into a new automated Monte Carlo algorithm, EternaBot, and rigorous experimental tests on Phase II puzzles. In four of nine cases (Figure 7b, d, e, and g), automated designs from the new EternaBot algorithm achieved maximum scores within ±1.5 of the player designs and median scores within ±5.5. In the remaining cases (Figure 7a, c, f, h, and i), EternaBot modestly underperformed players, a gap that may close as more experimental data and design rules are collected. Importantly, EternaBot outperformed RNAInverse and NUPACK (p = and , respectively; see Table 2c), with significantly higher maximum scores in eight of nine cases (Figure 7 and Table 3a). 12

15 Figure 7. Phase II results. The figure scheme is identical to Figure 6 but with an addition of EternaBot (Pink) created with players design rules. Only one round of player designs per target was evaluated to test whether the player skills were generalizable across different targets. 13

16 Figure 8. Top 5 design rules proposed by Eterna players. (a) The best designs from each design agent (Players, NUPACK and RNAInverse) for the last target shape of Phase I (Figure 6f); The designs are annotated with violations of the top five rules (i.e., rules that best explain the experimental results from Phase I) proposed by players. (b) Rule titles and statements. 3.6 Conclusion The controlled experiments in Phase II confirmed that players were capable of solving experimental scientific problem with a remote wet-lab. The gameplay successfully engaged players with trial-and-error based on iterative designs and experimental analysis. Players proved their abilities in the first year of the game via the experiments. Since then, players successfully solved over 400 design challenges by carefully analyzing more than 10,000 synthesized designs with approximately 1,200,000 data points - an experimental overhead that no single research lab could ever hope to achieve. The game also hinted at the possibility of formalizing player intuitions into re-usable principles (design rules). 14

17 Beyond its implications in RNA engineering, Eterna proposes a framework that opens up much broader domain of application in experimental science. The framework should be readily applicable to fields such as taxonomy, microbiology, and epidemiology, which are also being accelerated by rapidly growing experimental throughput and public interest. The game still faces the limitation that research ideas must come from a small number of developers (limited expert involvement), but the problem is addressed in its sequel, Cloud Lab. 15

18 4. Future work - Cloud Lab Eterna has successfully expanded the domain of scientific gaming to experimental science. However, one final problem remains. As in other scientific discovery games, players had to wait for the few game developers to find an interesting research project and distill it into a well-defined in-game puzzle to solve. Such waits implicated that the game was significantly underutilizing the players. Cloud Lab addresses this problem by transforming Eterna into an open research platform where any scientific expert can bring in his/her research project into the game to utilize its manpower and experimental resources. 4.1 Problem definition Unlike players, there are only few expert scientists in a game development team (or a research lab). As long as the proposal of research projects is confined to developers, a game is bound to underutilize the massive player manpower at its hands. The game must ensure scalable source of expert research projects to fully utilize its resources. One possible solution to this problem is to train players to read academic literatures and come up with meaningful projects themselves - creating an online graduate school. However, such an approach has 2 major problems: 1) it will take years of preparations and training for players to achieve a sufficient level of expertise. It is questionable whether a small group of experts (game developers) can sustain the game with the stream of interesting problems for such a long time. 2) In reality, only very small fraction of players will have the dedication to become an expert. The majority of them prefer to casually contribute to science while playing games. 4.2 Method Cloud Lab is a more immediate solution to this problem. Instead of having players create projects, Cloud Lab provides comprehensive tools and interfaces such that any research scientist in the field - experts outside the developer group - can easily post his/her project in the game for player participation. The game requires 2 key features: 1) an incentive scheme to attract expert scientists and 2) a generalizable method to transform experts research projects into playable puzzles. The following subsections describe detailed development plans for the two features. Attracting expert scientists The biggest challenge for Cloud Lab is attracting expert scientists to post their projects in the game. Distilling a research project into a puzzle that players can play is a time consuming task. Such a task cannot be expected to be performed voluntarily. Thus, Cloud Lab must provide tangible incentives to experts. Ideally, massive player manpower would be a sufficient incentive. However, it has been only few years since scientific discovery games started to produce publishable results. Because such results are still unknown to substantial portion of scientific community, it is generally very hard to convince experts that gamers can make meaningful contributions to their projects. 16

19 Figure 9. Cloud Lab preview. (a) Expert scientists propose their projects to Cloud Lab for free RNA synthesis. They need to provide easy and motivational descriptions of the projects to get players to vote for their projects. (b) Players vote on the projects that they want to participate in. Periodically, the top voted projects go active. Active projects open up for player participations and receive synthesis slots to experimentally test RNA sequences in the remote wet-lab. (c) Experts codes a scoring function to evaluate the sequences submitted to their projects. Instead, Cloud Lab uses Eterna s abundant experimental resources and online wet-lab pipeline as an attraction. Recent accelerations in the RNA synthesis pipeline now allow Eterna to synthesize and chemically analyze 1500 sequences a month. Cloud Lab offers experts free synthesis slots in the Eterna pipeline if players approve the experts research projects. Cloud Lab provides a special portal (Figure 9a) for experts to propose their on-going research projects and request synthesis slots for their RNA sequences. The proposed projects are evaluated by players in the form of voting (Figure 9b). Periodically, the highest rated projects go active and open up for player participation. Once the participation period ends, the experts requested sequences along with selected player sequences are synthesized and fully analyzed. The resulting data and all comments from players are organized into a single PDF formatted as a micropaper. The experts are asked to cite the micropaper when using the data in further publications. Although Cloud Lab is still in the prototyping phase, the incentive scheme and pipeline saw great initial results and excitement. 11 university research labs used Cloud Lab for projects ranging from testing novel RNA structures to verifying the structure of a flu virus. Project to puzzle translation Expert research projects have to be translated into playable puzzles to obtain participations from players. In RNA science, such process can be decomposed into 3 components: RNA structure definition, visualization, and creation of scoring function to evaluate player contributions. Whereas the first 2 components already have widely used formats, the scoring function often involves creative ideas from the researchers and cannot be defined with a specific format. Cloud Lab provides a browser based scripting interface (Figure 9c) through which experts can write any scoring function to evaluate given RNA sequences. Specifically, experts can plug in any function that accepts a sequence and all related experimental data as inputs and outputs a numerical score. When experimental results arrive 17

20 the scoring function is automatically run to evaluate all synthesized sequences in the project. The scripting interface is under development to provide an optimal coding experience and supports for a set of libraries widely used in RNA science. 4.3 Evaluation Cloud Lab will be evaluated based on the number of expert projects that incorporated at least one player proposed sequence in their experiments and the number of times micropapers from the game are cited in academic papers. 4.4 Timeline The approximate timeline for Cloud Lab is as follows. Milestone Deadline Build an expert workflow for project proposal and October, management. Build a player-expert collaboration workflow and November, 2013 an automatic project archiving (micropapers) Build a general scripting tool for project-puzzle December, 2013 translation Game live & evaluation March,

21 5. Conclusion Online gaming has emerged as a powerful channel to translate the massive manpower of gamers to scientific advances. From early image annotation games to the second generation scientific discovery games, citizen science games have demonstrated a radically new way to advance modern science. This thesis focused on two new games that addressed the fundamental limitations faced by previous games. Eterna integrated a wet-lab into its central gameplay, expanding the domain of games to experimental science. With a high-throughput RNA synthesis pipeline and trial-and-error based gameplay, players designed experimentally validated novel RNA structures on an unprecedented scale and formalized re-usable in vitro design rules. Eterna s sequel, Cloud Lab, is an open research platform where any expert scientist can propose his/her research projects in the game. Experts are provided with experimental resources and player manpower while players enjoy opportunity to take part in state-of-theart research. The game attracts massive scientific expert involvement and aims to solve significantly more research problems than any other previous game. Such successes motivate many future research venues that can further extend the use of games in science. One of them is games utilizing human dexterity instead of intelligence. Fields such as medical science often require extremely delicate experiments that can only be handled by human manipulations. Such experiments create a huge research bottleneck, as they often wait in line for limited technicians. But what if we setup remote controlled robot arms and high-definition web-cams allowing gamers to perform the experiments remotely? Many gamers are well-known for extremely skillful in-game maneuvers. Can they remotely help performing delicate experiments and further accelerate scientific progress? Another possible (but a longshot) venue is crowdsourcing the citizen science game development itself. Every game needs a team of dedicated game designers/developers/artists who can implement the game. It is extremely difficult for scientists to find such a team. But what if we design a platform where people from different fields can use their talents to build a game for the scientists? The platform could deploy a multi-stage development pipeline: a distillation stage in which the scientists translate their research into a well-defined problem; a gameplay design stage in which game designers translates the problem into a game; an implementation stage in which programmers and artists donate their talents; finally a testing stage in which thousands of gamers are invited to playtest and help improving the game. Such a platform could help even very small and underfunded research labs to deploy games for their research. Citizen science games ultimately offer a powerful vision in which games are fundamental infrastructures for scientific advances. Participating in scientific research could become a casual computer activity like playing Tetris. Gamers could discover their talents while playing games and find new career paths. Furthermore, games could easily bring education into contact with state-of-the-art research and promote early in-depth scientific experiences to a much broader audience. Citizen science games are still far from realizing this vision. However, continued efforts in the field will build more success cases and a larger scientific gamer basis. We believe such efforts will gradually, but surely mark games as an essential avenue for rapid scientific progress. 19

22 References Ahn, Luis von, and Laura Dabbish Labeling Images with a Computer Game. Proceedings of ACM CHI Ahn, Luis von, Ruoran Liu, and Manuel Blum Peekaboom: a game for locating objects in images. Proceedings of ACM CHI Berger, Bonnie, and Tom Leighton Protein folding in the hydrophobic-hydrophilic (HP) is NPcomplete. Proceedings of RECOMB '98 Proceedings of the second annual international conference on Computational molecular biology. Cooper, S., F. Khatib, A. Treuille, J. Barbero, J. Lee, M. Beenen, A. Leaver-Fay, D. Baker, Z. Popovic, and F. Players "Predicting protein structures with a multiplayer online game." Nature no. 466 (7307): Eiben, C. B., J. B. Siegel, J. B. Bale, S. Cooper, F. Khatib, B. W. Shen, F. Players, B. L. Stoddard, Z. Popovic, and D. Baker "Increased Diels-Alderase activity through backbone remodeling guided by Foldit players." Nature biotechnology no. 30 (2): Hofacker, I. L "RNA secondary structure analysis using the Vienna RNA package." Curr Protoc Bioinformatics no. Chapter 12:Unit Hofacker, Ivo. Vienna RNA Secondary Structure Package, Available from Kawrykow, A., G. Roumanis, A. Kam, D. Kwak, C. Leung, C. Wu, E. Zarour, players Phylo, L. Sarmenta, M. Blanchette, and J. Waldispuhl "Phylo: a citizen science approach for improving multiple sequence alignment." PLoS One no. 7 (3):e Khatib, F., S. Cooper, M. D. Tyka, K. F. Xu, I. Makedon, Z. Popovic, D. Baker, and Foldit Players "Algorithm discovery by protein folding game players." Proceedings of the National Academy of Sciences of the United States of America no. 108 (47): Khatib, F., F. DiMaio, S. Cooper, M. Kazmierczyk, M. Gilski, S. Krzywda, H. Zabranska, I. Pichova, J. Thompson, Z. Popovic, M. Jaskolski, D. Baker, Foldit Contenders Grp, and Foldit Void Crushers Grp "Crystal structure of a monomeric retroviral protease solved by protein folding game players." Nature Structural & Molecular Biology no. 18 (10): Kittur, Aniket, Ed H. Chi, and Bongwon Suh Crowdsourcing User Studies With Mechanical Turk Proceedings of ACM CHI Kulkarni, Anand, Matthew Can, and Bjorn Hartmann Collaborately Crowdsourcing Workflows with Turkomatic. Proceedings of ACM Conference on Computer-Supported Cooperative Work Land, K., A. Slosar, C. Lintott, D. Andreescu, S. Bamford, P. Murray, R. Nichol, M. J. Raddick, K. Schawinski, A. Szalay, D. Thomas, and J. Vandenberg "Galaxy Zoo: the large-scale spin statistics of spiral galaxies in the Sloan Digital Sky Survey." Monthly Notices of the Royal Astronomical Society no. 388 (4): Mathews, D. H., J. Sabina, M. Zuker, and D. H. Turner "Expanded sequence dependence of thermodynamic parameters improves prediction of RNA secondary structure." J Mol Biol no. 288 (5): McGonigal, Jane Gaming can make a better world. Riedel-Kruse, I. H., A. M. Chung, B. Dura, A. L. Hamilton, and B. C. Lee "Design, engineering and utility of biotic games." Lab Chip no. 11 (1): Rohl, Carol A., Charlie E. M. Strauss, Kira M. S. Misura, and David Baker "Protein structure prediction using Rosetta." Methods In Enzymology no. 383: Schawinski, K., M. Urry, S. Virani, P. Coppi, S. P. Bamford, E. Treister, C. J. Lintott, M. Sarzi, W. C. Keel, S. Kaviraj, C. N. Cardamone, K. L. Masters, N. P. Ross, D. Andreescu, P. Murray, R. C. Nichol, M. J. Raddick, A. Slosar, A. S. Szalay, D. Thomas, and J. Vandenberg "Galaxy 20

23 Zoo: The Fundamentally Different Co-Evolution of Supermassive Black Holes and Their Earlyand Late-Type Host Galaxies." Astrophysical Journal no. 711 (1): Seung, Sebastian. EyeWire Available from Smith, A. M., S. Lynn, M. Sullivan, C. J. Lintott, P. E. Nugent, J. Botyanszki, M. Kasliwal, R. Quimby, S. P. Bamford, L. F. Fortson, K. Schawinski, I. Hook, S. Blake, P. Podsiadlowski, J. Jonsson, A. Gal-Yam, I. Arcavi, D. A. Howell, J. S. Bloom, J. Jacobsen, S. R. Kulkarni, N. M. Law, E. O. Ofek, and R. Walters "Galaxy Zoo Supernovae star." Monthly Notices of the Royal Astronomical Society no. 412 (2): Zadeh, J. N., C. D. Steenberg, J. S. Bois, B. R. Wolfe, M. B. Pierce, A. R. Khan, R. M. Dirks, and N. A. Pierce "NUPACK: Analysis and design of nucleic acid systems." Journal of computational chemistry no. 32 (1):