AN INFORMATION SYSTEM AGNOSTIC ENVIRONMENT FOR REPRODUCIBLE EXPERIMENTS

Transcription

1 AN INFORMATION SYSTEM AGNOSTIC ENVIRONMENT FOR REPRODUCIBLE EXPERIMENTS Prototype Sillaber, Christian, University of Innsbruck, Austria, Graf, Alexander, University of Innsbruck, Austria, Breu, Ruth, University of Innsbruck, Austria, Abstract Designing reproducible experiments with information systems (IS) is a challenging task, mainly due to the complex nature of processes, vendor specific tools and the difficulty to find an adequate abstraction level. While technical challenges (e.g. finding performance bottlenecks) can be investigated in detail using simplified near-life settings or purposive research prototypes, these approaches are not feasible in experiments concerned with IS used in complex environments and problem domains (e.g. governance, risk and compliance management) as the software often cannot be simplified without considerable effort or losing rigor. On the other hand, software used in such domains has almost never been designed by the vendors with researchers goals and needs in mind which makes their usage in experiments burdensome. To address this, we present the research prototype of an experimentation environment that simplifies the creation of reproducible experiments around off-the-shelf information systems. Keywords: information system experiments, experimentation environment, reproducible research 1 Introduction Reproducible research is the foundation of judging the merit of scientific claims (Peng, 2011). While calls for more qualitative research in information systems research in general and domains such as information systems security research in particular have been made repeatedly (Puhakainen and Siponen, 2010; Siponen and Oinas-Kukkonen, 2007; Spears and Barki, 2010), new technologies, massive amounts of data and the complexity of the research questions at hand make reaching the goal of reproducible research results difficult (Jasny et al., 2011). When designing experiments with information systems, researchers have to decide between using preexisting (off the shelf) software and specifically developed software. Each approach comes with obvious trade-offs. For example, off-the-shelf information systems used in experiments typically do not keep track of users action in any form suitable for the researcher s needs. If, on the other hand custom made tools are used during experiments, often problems arise from dependency hell, code rot or imprecise documentation (Boettiger, 2014). Also, code is hardly published in academic publications and might be lost over time. For the sake of simplicity, when we use the term information system within the experimentation environment it refers to the software artifact itself. To address these problems, either existing information systems have to be adapted to the needs of the researchers, or researchers have to rely on other tools, that are better suited to produce reproducible results. Neither is likely to happen, as a) existing information systems are difficult to adapt (especially those used productively in organizations), b) impossible to adapt (e.g. closed source software), c) new information Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey,

2 systems can not be created due to a lack of resources or d) using specifically created information systems might influence the experiment. To address these gaps, we have developed an integrated experimentation environment that can be used to create reproducible experiments from already existing information systems (without having to modify them). The environment handles the creation of experiments, treatment, measurement and data extraction for evaluation of the experiment. To achieve this aim, first the theoretical foundation is discussed (Section 2), then the methodological approach is presented (Section 3). In Section 5, the prototype tool and its functions are presented. Finally, in Section 6, we discuss the theoretical implications of the prototype and conclude the paper. 2 Background Efforts such as the recomputation manifesto (Gent, 2013) and the Software Sustainability Institute (Crouch et al., 2013) aim to tackle the challenging task of evaluating and reproducing experimental results in applied computer science. While the reproducibility issue has recently received increased attention in science in general (Achenbach, 2015; Vasilevsky et al., 2013) and a variety of specialized domains (e.g. (Begley and Ioannidis, 2015; Collaboration, 2012)), Information Systems research is in this area is rather scarce. This lack of reproducibility makes it difficult, if not impossible to reproduce the findings of other scientists, extend experiments, test the conclusions or simply re-run the experiments with other participants. The advent of cloud computing and the success of virtualization technologies makes it easier to share code and data and collaboratively design and implement experiments. While there is still no established methodology to achieve this, recent research (Peng, 2011) discusses the need for a minimum standard of reproducibility in computer science, arguing that enough information about methods and code should be available for independent researchers to reach consistent conclusions using original raw data. As outlined in (Boettiger, 2014), current approaches to address these challenges are either wokflow solutions or virtual machines. While workflow systems have not yet been adopted widely (Dudley and Butte, 2010), virtualization garnered notable attention in recent years (Howe, 2012). 3 Design Science approach and Design Objectives The goal of this work is to develop a tool for creating reproducible experimentation environments. A design science methodology of information systems research (DSRM) approach was used as a research methodology (Peffers et al., 2012), with tool instantiation as an output (March and Smith, 1995). To achieve this, we analyzed the theoretical concepts for reproducible research and the available tools (Section 2). Based on the analysis, the prototype tool was conceptualized and developed (Section 4). The prototype was developed using an iterative development approach consisting of multiple build and evaluation cycles. The prototype has been developed and evaluated in close collaboration with the developers of the governance, risk and compliance management tool to be used in the (first) experiment as well as the persons responsible for designing the experiment (cf. the tool is called ADAMANT Sillaber, Brunner, and Breu, 2013 in Figure 4). The present prototype builds upon this research and focuses on the introduction of the core design objectives that emerged during the design cycles and were implemented in the presented prototype. The first design objective is to enable researchers to seamlessly design experiments with any information system, whether it was designed with experimentation in mind or not. The second design objective is to enable a concise definition of experiment settings and treatments. In addition, the need of seamless integration of the information system used for experimentation into the experimentation environment emerged as a core design objective. In order to find acceptance, the experimentation environment must provide a streamlined process to create new experimentation settings Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey,

3 around new information systems. Unwanted adaptions, restrictions to the user experience, as well as perceived breaks in the usage of the original information system would undermine any chances of creating successful experiments. 4 Description of the Prototype This section provides a demonstration of all experimentation activities supported within the prototype. This includes the preparation phase, experiment design, experiment execution and data collection. The prototype follows the recommended workflow approach by (Levy and Ellis, 2011; Wohlin et al., 2012): Preparation, Measurement, Treatment, Data collection. For each of these activities dedicated functionality was developed. Building on previous research by (Boettiger, 2014; Jimenez et al., 2015), that has identified Docker (Merkel, 2014) as the ideal tool to be used by researchers when building reproducible software environments. The prototype is implemented as a client-server application. The administrative client side is a web application, while the server side builds upon Docker, Postgresql and PHP 7. The users of the environment are the experiment administrators as well as the participants of the experiments. These two roles will be distinguished in continuation. 4.1 Preparation Phase First, the information system to be used during the experiment has to be prepared. For this, the application has to be containerized, i.e. separate containers for its core components have to be created. For example, one container has to be created for the application logic and one container has to be created for the database as shown in Figure 1. After this step, the new containers have to be added to a new experimentation environment. These two containers are added as Application Container 0 and Database Container 0 to the container manager. The container manager handles the creation and destruction of containers, links the application container to its database container, creates a url forward (e.g. and initializes the persistency layer. Furthermore, this component routes the incoming traffic from the participants to the appropriate container and is capable of capturing any data that goes through it (data in transit). The persistency layer has two tasks: to export the relevant contents of a container (data in storage) to a GIT repository and recreating a functioning container from a GIT repository. Furthermore, the persistency layer provides hooks that can be used to automatically generate reports, convert the database exports into human readable formats, etc. Each snapshot that is stored in the GIT repository contains all the data as created, modified or deleted by the user in the information system itself for later analysis. Additional data about the actions the user performed that did not manifest in the database can be gathered via hooks (e.g. the user changes between views, uses a search box, mouse events etc.). 4.2 Experiment Design After the containerization of the information system s components, the experiment can be designed. Depending on the required design of the experiment itself (e.g. number of participants, number of treatment groups, etc), appropriate branches for each of them can be created through the experiment manager. The experiment manager handles the design and execution of experiments and orchestrates the different containers through the container manager. Each experiment starts with an initial snapshot (S 0 ) that can be further prepared by entering data and creating appropriate snapshots (S 1...S n ). Each individual snapshot, as shown in Figure 2 is persisted to a GIT repository, allowing to use all features of a GIT repository, including branching and merging. For each participant, an individual, unique access code is generated and provided to them. With this unique access code (e.g. Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey,

4 Figure 1. Preparation: Turning a classic information system (left) into separated containers for the application and its database (right). Container 0 contains the unmodified database and application. Container 1...Container n are are modified instances of them. This is not limited to any architectural type and any n-tier architecture can be containerized. org/5e5351d7-e083-4ad7-beba-e55311a2cdf2/) both the user accessing the experimentation environment as well as the designated experimentation setting can be identified. From the point of view of the participant it is not possible to identify which treatment group their personal access code grants them access to. Figure 2. Example: Design progress of the experiment. Each individual state is persisted. Treatment groups are stored as GIT branches. 4.3 Experiment Execution The participants can now access the experimentation environment and perform the required tasks. To do this, they access the experimentation environment via their custom URL and the experimentation manager automatically redirects their requests to the appropriate container. To save resources, containers can be automatically destroyed (of course after persisting them). Using only a web browser, the participants can use the information system within the experimentation environment as they would use the original information system - with the difference that all interactions are automatically captured by the experimentation environment for further analysis. Listing 4.1. Git log partially showing the activities within the experimentation environment.... * a23483f User 2 data /* 48 b7f94 User 1 data cnt. * 2 a8040a User 1 data... / * cf Adapted treatment 1 Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey,

5 * 0 ffe20a Design of treatment 1 / * b Adapted treatment 0 * Initial design of treatment 0 / * 9 c24e2c Preparation of initial data * d Initial experimentation setup 4.4 Data Collection After the last participant has completed the assigned tasks, the collected data is ready for analysis. The experimentation environment has automatically captured all interactions between the user and the information system in a GIT repository as shown in Listing 4.1. The prototype supports extracting webserver logs (Apache + Tomcat) as well as exporting database contents (Postgresql) as shown in Figure 3. Additional exporters can be created as required. Currently, the prototype distinguishes between one-way and two-way synchronization. One way synchronization means, that the persistency layer only exports data from one of the containers to the GIT repository (e.g. application access or error logs). Two-way synchronization means, that the persistency layer is capable of exporting data from one of the containers and is also capable of populating an empty container with this data (e.g. populating a database). Figure 3. The persistency layer continuously extracts data from application and database containers. Any exported data, that is part of a snapshot is added as a new commit to the GIT Repository therefore ensuring that all modifications are persistently tracked. After conducting the experiment, the researchers can obtain the entire GIT repository (or selected commits) containing all modifications and data collected from the experiments for further analysis. If any need for clarification arises, each snapshot from each user can be reinstantiated and analyzed in the liveenvironment without by recreating the appropriate container from it. If the results of the experiments have to be validated or analysed in more detail, each snapshot provides all information necessary to reconstruct the state of the experimentation environment at the time of its creation. While data-minig and related techniques are necessary to further analyze the produced data (which out of scope for the scope of this paper), the availability to (re-)instantiate the state of the artifacts for further analysis certainly improves a researcher s capabilities. Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey,

6 4.5 Limitations and future work The main limitation of the proposed environment is that it relies on the ability of the information system s software to be containerized and that it exposes at least a rudimentary open web-interface. This currently limits the set of potential software being supported to web-based software. However, this limitation will be addressed in future work by increasing the number of supported interfaces, protocols and databases. As any software can be run in a virtual machine, we believe that ultimately the environment can be used for almost any piece of (modern) software - although additional overhead for virtualization might be required. Furthermore, additional effort is required to evaluate the viability and performance of the selected solutions as complexity of the environment increases with each instance. Future work will also focus on the reproducibility aspect itself and how to include the documentation of the experimentation setup that is external (e.g. selection of participants) into the snapshots and how researchers can be better supported in the analysis of the results. 5 Conclusion and Contribution The present paper describes a prototype that addresses a common practical problem, manifested in the challenge of creating reproducible experiments with information systems. The prototype combines state-of the art research from the field of virtualization and scientific computing to enable the creation of custom experimentation environments. Along with the creation of reproducible experiments, our prototype can be used to manage different treatment groups, variants of experiments and provides comprehensive information about the participants interactions with the information system. Whereas scientists can concentrate on the experiment it self as well as data collection, our prototype takes care of the low-level - activities required for the design, execution and evaluation of experiments. The researchers only have to containerize the information system under investigation. These containers can be initiated, modified, cloned, merged, etc. as required, while the experimentation environment tracks these instances and persists them in a reproducible manner. Different branches can be created for different treatment groups and different participants. Any activities performed by the participants inside the information system is transparently logged and available for later analysis by the researchers. With regard to the theoretical contribution, the present paper distinguishes between traffic data, data in storage and introduces the notion of experiment snapshots. The introduced prototype and the data it collects from experiments, enables researchers to further analyze how information systems are used by various stakeholders, and how efficient they are designed. Traffic data, i.e. the data containing all transactions between the user and the information system can be obtained for later analysis. The data in storage provides researchers with insights into all actions and activities that resulted in changes to the information systems persistency layer. As a whole, the paper and prototype form the basis for further research and practical support for long-term oriented information system experimentation by addressing the lack of generalized environments for the generation of reproducible experiments. The prototype itself is fully open sourced to ensure that the goal of true reproducibility can be achieved. This could have a significant impact, both with regard to improved research outcomes and overall quality of reproducible research in information systems research. Acknowledgment: This work was supported by the Austrian Federal Ministry of Economy (BMWFW), QE LaB - Living Models for Open Systems (FFG ). References Achenbach, J. (2015). The new scientific revolution: Reproducibility at last. Washington Post. Begley, C. G. and J. P. Ioannidis (2015). Reproducibility in science improving the standard for basic and preclinical research. Circulation research 116 (1), Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey,

7 Boettiger, C. (2014). An introduction to Docker for reproducible research, with examples from the R environment. CoRR abs/ Collaboration, O. S. et al. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science 7 (6), Crouch, S., N. C. Hong, S. Hettrick, M. Jackson, A. Pawlik, S. Sufi, L. Carr, D. De Roure, C. Goble, and M. Parsons (2013). The Software Sustainability Institute: Changing Research Software Attitudes and Practices. Computing in Science and Engineering 15 (6), Dudley, J. T. and A. J. Butte (2010). In silico research in the era of cloud computing. Nature biotechnology 28 (11), Gent, I. P. (2013). The Recomputation Manifesto. CoRR abs/ Howe, B. (2012). Virtual appliances, cloud computing, and reproducible research. Computing in Science & Engineering 14 (4), Jasny, B. R., G. Chin, L. Chong, and S. Vignieri (2011). Again, and again, and again... Science 334 (6060), Jimenez, I., C. Maltzahn, J. Lofstead, A. Moody, K. Mohror, R. Arpaci-Dusseau, and A. Arpaci-Dusseau (2015). Tackling the Reproducibility Problem in Storage Systems Research with Declarative Experiment Specifications. In: Proceedings of the 10th Parallel Data Storage Workshop. PDSW 15. Austin, Texas: ACM, pp Levy, Y. and T. J. Ellis (2011). A guide for novice researchers on experimental and quasi-experimental studies in information systems research. Interdisciplinary Journal of Information, Knowledge, and Management 6, March, S. T. and G. F. Smith (1995). Design and natural science research on information technology. Decision support systems 15 (4), Merkel, D. (2014). Docker: lightweight linux containers for consistent development and deployment. Linux Journal 2014 (239), 2. Peffers, K., M. Rothenberger, T. Tuunanen, and R. Vaezi (2012). Design science research evaluation. In: Design science research in information systems. Advances in theory and practice. Springer, pp Peng, R. D. (2011). Reproducible research in computational science. Science (New York, Ny) 334 (6060), Puhakainen, P. and M. Siponen (2010). Improving employees compliance through information systems security training: an action research study. Mis Quarterly 34 (4), Sillaber, C., M. Brunner, and R. Breu (2013). Towards an Architecture for Collaborative Cross Organizational Security Requirements Management. In: Business Information Systems. Springer Berlin Heidelberg, pp Siponen, M. and H. Oinas-Kukkonen (2007). A review of information security issues and respective research contributions. ACM Sigmis Database 38 (1), Spears, J. L. and H. Barki (2010). User participation in information systems security risk management. MIS quarterly 34 (3), Vasilevsky, N. A., M. H. Brush, H. Paddock, L. Ponting, S. J. Tripathy, G. M. LaRocca, and M. A. Haendel (2013). On the reproducibility of science: unique identification of research resources in the biomedical literature. PeerJ 1, e148. Wohlin, C., P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén (2012). Experimentation in software engineering. Springer Science & Business Media. Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey,

8 Presentation of the Prototype A fully functional demonstration of the prototype including the web application (cf. Figure 4) on a laptop and the experimentation environment on the server side will be shown. We will demonstrate, how an information system can be containerized and used in an experiment. Furthermore, we will demonstrate how experiments can be tailored and resulting data be collected. A sample experiment will be set up, executed and analyzed. Screens of both the experimentation administrator as well as the stakeholders partaking in the experiments can be mirrored to larger screens to show them to a larger audience. Figure 4. The management window for experiment instances Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey,