Natural experiments and correlational studies

Transcription

1 Natural experiments and correlational studies Contents 1 Introduction Non-experimental explanatory quantitative research designs Natural experiments Correlational studies Applications of non-experimental explanatory designs Strengths and weaknesses of non-experimental explanatory designs References Introduction This supplement expands the introduction to natural experiments and correlational studies in Chapter 6. Along with additional details about these types of study, it offers an overview of the design s strengths and weaknesses, along with additional references for further study. 2 Non-experimental explanatory quantitative research designs Given some of the challenges involved in conducting experiments in social science situations, it is not surprising that researchers have sought other ways to answer why-type research questions. Such non-experimental explanatory research designs do not involve manipulation of the independent variable by the researcher. For this reason they are also sometimes known as observational studies, even when they do not involve observational data collection techniques. In practice data sources can be primary and/or secondary and data collection can include questionnaires or the collection of company operating data as well as structured observation. Like experiments, non-experimental explanatory research designs take a deductive approach, pre-specifying the hypotheses to be tested and emphasising measurement and comparison. However, since experimental control is not possible, non-experimental explanatory designs typically rely on statistical control to deal with extraneous influences. Depending on the chosen design, non-experimental studies can be cross sectional or longitudinal. Longitudinal designs may be prospective, following developments over time, or Management Research: Applying the Principles 2015 Susan Rose, Nigel Spinks & Ana Isabel Canhoto 1

2 retrospective, researching what has already happened, sometimes in order to set a baseline against which the current situation can be compared. Terminology varies but following Shadish et al. (2002: 12) we can identify two main categories of explanatory non-experimental design: 2.1 Natural experiments Despite their name, natural experiments are not really experiments because the independent variable is not manipulated as part of the research itself. Instead the researcher looks for a naturally occurring comparison between a treatment and a comparison condition that is as close as possible to those that would have been created in an experiment. An example would be a study that compares a factory in which a new quality programme has been introduced (the treatment group) against one in which it has not (the comparison group) in terms of their respective quality levels (the outcome). The distinction between natural experiments and quasi-experiments is not always clear in the literature. Sometimes the term natural experiment is used to refer to conditions brought about by a natural event such as weather or earthquake. Much more broadly, however, Dunning (2012) argues that the key distinguishing feature of a natural experiment as opposed to a quasi-experiment is that the former are characterised by the assumption that the assignment of subjects to a treatment or control group is random or as good as random (Dunning 2012: 3). Such situations may arise because of genuine random assignment or because the process of allocation justifies the assumption that the process was as good as random. Dunning (2012) cites a number of studies which qualify as natural experiments on that basis, for instance, research on the impact of military service on lifetime earnings where allocation to military service is on the basis of a lottery (genuine random assignment) and investigations of the impact of minimum wage levels on employment comparing neighbouring states in the US (as good as random assignment). The extent to which the assumption of randomness holds is an important feature of the strength of natural experiments. In the fictional example of the impact of a quality management programme, for instance, such an assumption might break down for a number of reasons. If the decision to implement the programme was made following a year of very poor quality performance, for example, the intervention could not really be considered random. It Management Research: Applying the Principles 2015 Susan Rose, Nigel Spinks & Ana Isabel Canhoto 2

3 is quite possible that any improvement noticed may be due to regression to the mean following particularly bad performance (Dunning 2012). Natural experiments have attracted growing interest in economics but are not so widely encountered in business and management. Dunning s (2012) book-length treatment gives a good introduction. 2.2 Correlational studies In correlational studies the researcher measures two or more variables as they exist naturally for a set of individual cases (e.g. people) and then tests the relationship between them. (Note that, despite their name, the analysis procedures used in correlational studies are not limited to correlation but typically include regression and other statistical techniques as presented in Chapter 13.) Correlational studies can use primary or secondary data or a mix of the two as appropriate. An example of a correlational study using primary data would be an analytic survey study which measures the levels of satisfaction (the independent variable) and loyalty (the dependent variable) for an organisation s customers and tests whether or not they are related as expected. We illustrate the use of surveys in this way in the following research in practice and in Chapter 6 in the context of survey studies. Research in practice 1 Example correlational study using an analytic survey Analytic survey: Enterprise Resource Planning (ERP) systems and the manufacturing marketing interface Interdependence between marketing and manufacturing is seen as an important issue in operations management, especially under conditions of market or product uncertainty, but the role of information technology in facilitating interface has not received much attention. Enterprise Resource Planning (ERP) information systems intended, amongst other things, to integrate management information across the enterprise thereby improving coordination and, ultimately, performance. Gattiker (2007) set out to investigate the relationship between marketing manufacturing interdependence, improved coordination thanks to ERP and the overall business impact of ERP on the manufacturing plant. A conceptual model and hypotheses were developed which were tested in the research. The research design was a non-experimental explanatory study using a questionnaire-based analytic survey. A range of methods are reported by which potential respondents were identified including a professional association and ERP user groups; responses were screened to remove those not meeting appropriate criteria regarding, for example, ERP use, resulting in a total of 124 useable responses. Multivariate statistical analysis was used to establish the reliability of the scales and test the research hypotheses The author concludes that the findings support the general proposition that Management Research: Applying the Principles 2015 Susan Rose, Nigel Spinks & Ana Isabel Canhoto 3

4 interdependence is one factor that influences the degree to which organisations reap benefits from their ERP investments (Gattiker 2007: 2895). 3 Applications of non-experimental explanatory designs Non-experimental designs, particularly correlational studies, are widely used in both pure and applied research. Two areas stand out: Explanatory theory testing, in which non-experimental designs are used to test hypotheses developed from the theory under test; the theory is often depicted as a conceptual model of the variance type and the hypotheses are expressed in terms of relationships between independent and dependent variables as discussed in Chapter 5. The extent to which the findings allow conclusions to be drawn about cause and effect is discussed at the end of this section. Prediction of the value of a dependent variable from knowledge of the value of one or more independent variables. Correlational studies in particular are often used to measure the impact of one variable on another in order to determine the size of the effect and in many cases to compare it relative to other possible causes. In such cases determining that the relationship is causal may not be of interest even though one variable is identified as the predictor (i.e. independent) variable and the other as the outcome (i.e. dependent variable). From a practical point of view, a non-experimental design may be appropriate in situations where an experiment is not an option. Such situations can arise when you cannot, or do not want to, modify the independent variables the effects of which you are interested in studying. This might be the case if you wanted to investigate a situation that currently exists (e.g. consumer attitudes towards a product or service), where it has happened in the past (e.g. evaluating a change programme), where it would not be ethical to intervene (e.g. manipulating consumer exposure to alcohol or drugs) or where it is not feasible for you to change the conditions (e.g. employees place of work). These situations are commonly encountered in practice so it is not surprising that non-experimental designs are so widely used. Management Research: Applying the Principles 2015 Susan Rose, Nigel Spinks & Ana Isabel Canhoto 4

5 4 Strengths and weaknesses of non-experimental explanatory designs There are many potential attractions to using non-experimental designs, especially in the form of questionnaire-based correlational studies of the type just described which also benefit from the advantages of survey designs in general. Above all they avoid many of the practical challenges faced in running an experiment. Non-experimental designs typically gather their data in naturally occurring settings. As Field and Hole (2003: 10) put it the good thing about this kind of research is it provides us with a very natural view of the question we re researching. The bad thing is that non-experimental designs cannot employ the kinds or levels of control enjoyed by experiments. What are the implications of this for investigating causal questions? Recall the criteria for inferring causality that we outlined in Chapter 5. Non-experimental quantitative designs tend to be strongest on measuring co-variance, less strong on assessing whether cause comes before effect (unless they are longitudinal designs) and weakest on ruling out the influence of extraneous variables (Johnson 2001: 7). These problems lead some writers to deny any causal explanatory role for such designs. Field and Hole (2003: 26), for example, argue that correlational research does not allow causal statements to be made and many researchers avoid using words like cause when reporting their results. Our own position, as we argued in Chapter 5, is that non-experimental explanatory research designs do have a role to play in causal research. The identification of strong correlations, for example, can serve to stimulate further research, including experiments, to find whether or not the observed correlation is, in fact, causal. Development of causal theories can be supported by testing the associations implied by causal models (de Vaus 2001, Aneshensel 2002) although interpreting the results in causal terms remains contentious. A strong theoretical basis can, in turn, support causal analysis using non-experimental studies, for example by identifying relevant variables, including possible moderators and mediators, that can be included in testing (Rindfleisch et al. 2008: 275). Nevertheless, you should always be cautious when inferring causality on the basis of non-experimental studies. 5 References Aneshensel, C. S. (2002). Theory-based data analysis for the social sciences. Thousand Oaks, CA: Pine Forge. Management Research: Applying the Principles 2015 Susan Rose, Nigel Spinks & Ana Isabel Canhoto 5

6 de Vaus, D. (2001). Research design in social research. London: Sage. Dunning, T. (2012). Natural experiments in the social sciences. A design-based approach. Cambridge: Cambridge University Press. Field, A. and Hole, G. (2003). How to design and report experiments. London: Sage. Gattiker, T. F. (2007). Enterprise resource planning (ERP) systems and the manufacturing marketing interface: An information-processing theory view, International Journal of Production Research, 45(13), Johnson, B. (2001). Toward a new classification of nonexperimental quantitative research, Educational Researcher, 3(2), Rindfleisch, A., Malter, A. J., Ganesan, S. and Moorman, C. (2008). Cross-sectional versus longitudinal survey research: Concepts, findings, and guidelines, Journal of Marketing Research (JMR), 45(3), Shadish, W. R., Cook, T. D. and Campbell, D. T. (2002). Experimental and quasiexperimental designs for generalized causal inference. Belmont, CA: Wadsworth Cengage Learning. Management Research: Applying the Principles 2015 Susan Rose, Nigel Spinks & Ana Isabel Canhoto 6