The Good, the Bad, and the Ugly in Mixed Methods Research and Design

Transcription

1 Editorial The Good, the Bad, and the Ugly in Mixed Methods Research and Design Journal of Mixed Methods Research 5(4) Ó The Author(s) 2011 Reprints and permission: sagepub.com/journalspermissions.nav DOI: / Manfred Max Bergman 1 Mixed methods research and design (MMRD) deservedly continues its ascent in the social and related sciences. Its popularity has generated a critical mass of theoretical and empirical contributions, which has infiltrated and influenced many important research fields, particularly in education, health, and evaluation. Despite its wide recognition, also from government agencies and various national and private funding bodies, and despite its successes in academic and other types of research, many theorists, methodologists, and empirical researchers remain apprehensive, considering this approach insufficiently rigorous. They are often right. This text will identify some of which I consider the most notable weaknesses based on my observations as editor, reviewer, supervisor, project director, and consultant of various mixed methods related projects and programs. It is an idiosyncratic list, reflecting tastes, interpretations, and positions not necessarily shared by some of my esteemed colleagues. It is hoped that this text stimulates debates that improve the rigor and scope of MMRD. The first generation of mixed methods researchers, from the end of the 19th century, conducted their mixing, blending, combining, or meshing of different data collection and analysis methods informally and unencumbered by subsequent notions relating to the incompatibility thesis and the paradigm war, due in part to the absence of conventions, methodological sophistication, and orthodoxy (Tashakkori & Teddlie, 1998). Important works from the 1990s, 1 notably by Julia Brannen, Alan Bryman, Vicki L. Plano Clark, John W. Creswell, Abbas Tashakkori, Charles Teddlie, and others, form the core of the second generation. They provided mixed methods researchers with a vocabulary, taxonomy, and process description, which paved the way for its current success. Part of the secret of MMRD s success is based on the adoption of the division of labor between qualitative and quantitative methods as formulated by critical theory-, postmodernity-, and hermeneutics-inspired texts a decade earlier. Until today, an idiosyncratic and unsatisfactory interpretation of philosophical pragmatism was the pacifier in the paradigm war. A closer inspection reveals theoretical and conceptual shortcomings of this sleight of hand, which translate into either logical and procedural inconsistencies or diplomatic disregard of the rules and regulations outlined in the research methods literature (Bergman, 2008). The following are the inconsistencies, challenges, or unresolved problems, loosely grouped into two interrelated families: conceptualization/theorization and design. 1 University of Basel, Basel, Switzerland Corresponding Author: Manfred Max Bergman, Department of Social Sciences, University of Basel, Petersgraben 27, 4051 Basel, Switzerland max.bergman@unibas.ch

2 272 Journal of Mixed Methods Research 5(4) Conceptualization and Theorization Terminology Disputes about an appropriate terminology are part of any discipline. No one should be surprised that theorists and empirical researchers fail to agree on a common terminology. Some have argued that the term mixing is inappropriate as a label when combining research methods because qualitative (QL) and quantitative (QN) components are not really mixed; rather they are blended, meshed, combined, and so on. However, diverging from an established terminology also leads to confusion, rather than clarification, even though the initial aim may have been clarification. Until there are good reasons to abandon the now well-established term mixed, I suggest acquiescing to this terminology. A second dispute about terminology relates to the differences between triangulation research and MMRD. Three positions can be identified: Triangulation as a subset of mixed methods; mixed methods as a subset of triangulation; and mixed methods and triangulation as synonyms and, thus, interchangeable. Methods triangulation has been used mostly as a metaphor borrowed from navigation, referring to the identification of a location based on two separate reference points (e.g., Kelle, 2008; Smith, 1975), thus implying that mixed methods should be used such that the results of a qualitative component of a study converge with the results of the quantitative component. In this context, divergences can be used to qualify the research findings. Accordingly, it would be least confusing to use triangulation as a type of mixed methods research that aims at convergence. A third terminology dispute about MMRD relates to whether the combination of multiple QL or multiple QN research constitutes a mixed methods design. The current nomenclature for this type of design as proposed by Tashakkori and Teddlie (1998) is qualitative multimethod design or quantitative multimethod design. However, given the considerable variations within, and the considerable overlap between, the QL/QN methods groups, a move toward a more inclusive concept should be considered. It may be just as interesting and complex to integrate a qualitative thematic analysis with a narrative analysis, or a random controlled trial experiment with a questionnaire, for example. It may be best to subsume under the MMRD heading conventional mixed methods research, that is, QL and QN components within one design, as well as multimethod research, that is, multiple QL components in one study or multiple QN components in one study. Mixed Model Research A type of research design will emerge where the line of demarcation between the QL and QN components is no longer clearly identifiable at least that is what some authors augur. This affirmation can be interpreted in a strong and a weak sense. In a strong sense, qualitative and quantitative research belong to different paradigms, are underpinned by different philosophical positions (especially in relation to epistemology, ontology, and axiology), address different research themes and questions, differ in their data collection and analysis methods, demand different interpretations of their respective research results, and so forth. This position is highly problematic because many assertions related to these positions are demonstratively incorrect in both theory and application (e.g., Bergman, 2010, 2011). Blurring the lines from this perspective is difficult to conceive of because the question remains as to how we can combine a perspective that subscribes to objectivity, unbiased and value-free research, and the separation between the researcher and the researched, with a perspective that emphasizes subjectivity, researcher context, value-laden research, and the inseparability between the researcher and the researched. Accordingly, mixed model research may have to be abandoned or it should at least be elaborated or demonstrated. The weaker version invites us to reexamine the assumptions emphasized in the MMRD literature about the fundamental differences between QL and QN

3 Bergman 273 methods. Some of the decrees proposed or, rather, appropriated from other literature, may turn out to be attempts to segregate them for purposes associated with identity politics and resource competition. Should the dominant discourse move toward this, then researchers would find that we are indeed already engaged in mixed model research in that it is generally possible and often necessary to reflect on research questions, theoretical frameworks, sampling, and interpretations only weakly associated with whether or how we apply a statistical technique. In other words, a good mixed methods research project includes an epistemology and ontology, a research question and theoretical framework, sampling strategies, and interpretations that are conducive to both QL and QN methods. The tenets of mixed model research are a necessary condition of a good mixed methods project, and mixed model research is therefore not a separate research design awaiting discovery in some distant future. Bridging of Ontological and Epistemological Schisms From the arguments above, it must be concluded that MMRD is unable to bridge incompatible positions in relation to the nature of reality, the relation between the researcher and the researched, and whether or not all research is either value free or value laden. Neither is MMRD more democratic, participatory, or transformative, compared with monomethod research. Whether research aims to transform existing inequalities or power imbalances depends on the aims and objectives of the researcher, not on the particularities of data collection and analysis techniques. Participatory or transformative research can be pursued successfully from any research method. Design Research Focus and Question Often, heterogeneous research teams or programs fail to agree on a specific research focus, question, or epistemological approach. As a compromise, they may decide to engage in MMRD, where different subgroups or subprograms pursue their agenda from separate QL or QN perspectives. This rarely leads to a successful application of MMRD because the research questions, as a consequence of an uneasy compromise, are accordingly vague or overly ambitious. Instead, such research teams often produce thematically connected monomethod research outputs. Mixed Methods Research and Design Typologies Increasingly, researchers are seeking help with deciding on what research design would be appropriate for their particular purpose, partly because of taxonomies introduced in MMRD textbooks, which propose specific designs for specific purposes (e.g., Creswell, 2008; Morse & Niehaus, 2009). Researchers may be led to the simplistic assumption that inductive, deductive, transformative, and so on, research must each correspond to a specific type of mixed methods design. While an illustration of the association between research purpose and research design is most helpful, particularly for novice researchers, it may also prematurely constrain researchers. In that case, researchers may view the research process unduly formalistically. Sampling Probabilistic and nonprobabilistic sampling often is referred to as quantitative and qualitative sampling, leading to the incorrect assumption that the QL and QN components require data sampled within the correct family of sampling methods. However, it is conceivable to draw a stratified random or random cluster sample for small-scale QL research, or to draw a snowball

4 274 Journal of Mixed Methods Research 5(4) or atypical case sample for research associated with a QN (non-inferential) methods. A second issue relating to sampling and MMRD relates to inference. Possibly the most popular form of MMRD is a sequential design, in which an exploratory QL component (e.g., interviews) is followed by a QN component (e.g., questionnaire). In this case, the QL component tends to explore the dimensionality of a phenomenon under investigation before questionnaire items associated with the thus identified dimensions are included in a large-scale survey. The results are often presented as representative because of the sample characteristics of the QN component. However, this is only partially correct. The exploratory QL component usually is not suitable for population inference because dimensionality identified in a small sample drawn nonprobabilistically may not include all relevant dimensions. Although the large-scale survey may indeed allow for population inference, it is nevertheless limited to the constraints imposed by the dimensionality identified from the sample associated with the QL component. The same argument can be made for all other mixed methods designs aiming for population inference. Justification for Mixed Methods Research and Design There are many good reasons to prefer an MMRD to a monomethod design (e.g., Tashakkori & Teddlie, 2010). However some justifications are problematic. First, it is often argued that MMRD is better than monomethod research because it presents a supplementary perspective, that is, not just the QL or QN perspective, but both. However, it could be argued that any additional data set relevant to the research question, or any additional analysis of a given data set, would provide an additional perspective. It is not merely some additional perspective that makes MMRD so interesting; the additional perspective is a necessary but insufficient condition for the justification of MMRD. Second, the literature on MMRD often leads to the incorrect conclusion that some form of holism is possible in the sense that studying a phenomenon or research question from multiple perspectives leads to objective research results. Unfortunately, this is not the case. For example, the method effect introduced by the QL component is not cancelled out by the introduction of a QN component because the latter introduces its own method effect. In the end, no matter how many theoretical approaches, data sets, or analyses are part of a research project, it will never answer a research question in all its complexity. Third, MMRD is sometimes used to reveal the limitations of either QL or QN methods. For example, some authors aim to show how much richer or deeper data from a 1-hour interview is, compared with the responses of a few closed-ended questions from a questionnaire. Similarly, some authors aim at showing the shortcomings of interviews compared to normed and standardized scales. A good MMRD application deals with these limitations to improve on the limits of such findings, rather than misusing MMRD to illustrate the limitations of QL or QN per se. Fourth, it is often argued that MMRD is better than monomethod research in principle. However, there are many good reasons to limit a research project to a particular data set and a particular analysis, among which are parsimony, theoretical and methodological elegance, reduced complexity, time and cost considerations, and a greater ease in communicating in word or print the research findings. Of concern are an increasing number of doctoral students who, without sufficient training in QL and QN methods, embrace an MMRD in their dissertations, often for dubious reasons. Fifth, and associated with this point, some authors introduce a cross-table to summarize their findings from QL research, or they cite as background information official statistics as their sole QN component. Usually, this limited application of QN methods in conjunction with a dominant QL component does not constitute MMRD. Similarly, a literature review in preparation of a QN study does not constitute a QL component and, thus, would normally not lead to a MMRD. Despite these weaknesses, challenges, or unresolved problems, MMRD often offers considerable advantages compared to monomethod research. It can cross-validate or complement

5 Bergman 275 individual findings, and it may be able to combine different strands of knowledge, skills, and disciplines. Because of the multitude of choices within and between the QL and QN components, it brings the individual researcher into the centre of research activities. She or he may have to become more knowledgeable and more critical toward research in general and the different research steps and techniques in particular, compared with researchers engaged in monomethod research because MMRD often requires careful and critical assessments of sampling, data collection, and data analysis methods with regard to their assumptions, assertions, and widespread mantras. As such, MMRD may dislodge ossified positions primarily maintained by identity politics associated with QL and QN methods. Accordingly, an evolution in this regard may indeed lead to a revolution in social science research methods: a critical reassessment of the possibilities and limits of each research technique may actually lead to a reformulation of QL and QN methods. The third generation of MMRD will be marked by a moderation of the current optimism, especially with regard to objectivity and holism. By reframing its theoretical underpinnings, it will also make important contributions toward the reframing of the possibilities and limits of QL and QN. This third generation will build on the major contributions of the current generation MMRD literature, and it will continue to attract the deserved attention of doctoral students, professional researchers, funding bodies, and research programs. Author s Note The author is visiting professor at the Universities of Johannesburg and the Witwatersrand. Note 1. Avant-gardists who prepared the field include Denzin (1970), Smith (1975), and Jick (1979). References Bergman, M. M. (2008). Advances in mixed methods research: Theories and applications. Thousand Oaks, CA: SAGE. Bergman, M. M. (2010). On concepts and paradigms in mixed methods research. Journal of Mixed Methods Research, 4(3), Bergman, M. M. (2011). The politics, fashions, and conventions of research methods. Journal of Mixed Methods Research, 5(2), Creswell, J. W. (2008). Research design: Qualitative, quantitative, and mixed methods approaches. Thousand Oaks, CA: SAGE. Denzin, N. K. (1970). The research act: A theoretical introduction to sociological methods. Chicago: Aldine. Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24, Kelle, U. (2008). Die Integration qualitativer und quantitativer Methoden in der empirischen Sozialforschung: Theoretische Grundlagen und methodologische Konzepte [The integration of qualitative and quantitative methods in empirical social research: Theoretical principles and methodological concepts]. Wiesbaden, Germany: VS Verlag für Sozialwissenschaften. Morse, J. M., & Niehaus, L. (2009). Mixed method design: Principles and procedures. Walnut Creek, CA: Left Coast Press. Smith, H. W. (1975). Strategies of social research: The methodological imagination. Englewood Cliffs, NJ: Prentice Hall. Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: SAGE. Tashakkori, A., & Teddlie, C. (2010). SAGE handbook of mixed methods in social & behavioral research. Thousand Oaks, CA: SAGE.