The influence of primary research methodology on perceptions of good educational research Kelly D. Bradley, Jessica D. Cunningham and Jennifer Eli University of Kentucky Abstract This study further investigates the findings of Bradley, Royal, Cunningham, Weber, and Eli (), where survey items connected to methodology revealed great variation in perceptions of characteristics of good educational research. The purpose is to examine whether differential response patterns exists across the research methodology variable, which is the respondents selfidentified primary research methodology. Over half of methodology statements in the survey indicated differential response patterns across the research methodology variable. Responses for two methods items were found significantly different across respondents choosing entirely qualitative, mostly qualitative, and mostly quantitative as their primary research methodology. *** Preliminary paper; do not cite without permission of lead author Use Kelly D. Bradley, Ph.D. as author of contact. (P) 9-7-9 or (E) kdbrad@uky.edu
Methodology influence on research perceptions The influence of primary research methodology on perceptions of good educational research While innovative research in teacher education is a focal point in teacher education, examination of common characteristics of what constitutes good educational research is vital to maintaining high-quality research. This study utilizes results of a survey about perceptions of good educational research completed by College of Education faculty and graduate students at a southeastern university. Prior to distribution, it was assumed common ground and variation in perceptions of what constitutes good educational research could be identified through results coupled with existing literature. While commonality was found in responses to the ethics and theory survey sections, great variation was demonstrated in survey items connected to methodology (Bradley, Royal, Cunningham, Weber, & Eli, ). This finding spawned the question of whether differential responding to survey items existed depending on the variable classifying respondents by their self-reported primary research methodology. Theoretical Framework There is general sentiment that educational research should involve carefully constructed designs, while implementing the most appropriate methods for data collection and analysis with the purpose of answering research questions aimed at understanding the world of education (Bradley, Royal, Cunningham, Weber, & Eli,, p. 7). The quality of educational research is often evaluated on the merits of its methods (techniques or procedures for collecting and analyzing data) and methodology. As Teddlie and Tashakkori (9) posit: A research methodology is a broad approach to scientific inquiry specifying how research questions should be asked and answered. This includes worldview considerations, general preferences for designs, sampling logic, data collection and analytical strategies, guidelines for making inferences, and the criteria for assessing and improving quality. (p. )
Methodology influence on research perceptions Within the social and behavioral sciences, broadly speaking, educational researchers can be partitioned into three overarching categories based on methodological orientation specifically qualitatively leaning, quantitatively leaning, and mixed methodologists (Creswell, ; Creswell & Plano-Clark, 7; Tashakkori & Teddlie, 99; Teddlie & Tashakkori, 9). Different designs and methods are better for different purposes. No one design is the best or most desirable; multiple types of scientific inquiries and methods are required to generate the rich body of scientific knowledge needed to improve education (Borko & Whitcomb,, p. 66). Given this, attention must be given to differences among paradigms, methodologies and methods that guide researchers toward conducting good educational research. Paradigms According to Morgan (7) paradigms are a system of beliefs and practices that influence how researchers select both the questions they study and methods they use to study (p.9). Researchers beliefs, practices and philosophical assumptions on the nature of ontology, epistemology, and axiology play a vital role in shaping the methodology, methods and research questions. Is there a single reality or multiple constructed realities? Is the relationship between the researcher and the subject independent or dependent? Is inquiry value free or value bounded? Quantitative Methods The constructivist, positivist or post positivist, and pragmatic paradigms are often associated with the qualitative, quantitative and mixed methods research positions, respectively. Quantitative research methods are most often identified with the positivism or post positivism philosophical orientation as these methods propose measuring and analyzing causal relationships within a framework that may or may not be value-free. Positivists and post-positivists believe knowledge is objective; context-free generalizations are possible; causal links can be isolated and
Methodology influence on research perceptions identified. Furthermore, the positivist framework acknowledges personal bias but strives for separation between researcher and subject (Tashakkori & Teddlie, 99). A quantitative approach generally involves working with numerical data collected from a probability sample that will be analyzed using statistical techniques. Quantitative researchers typically employ deductive logic and reasoning, which is confirmatory in nature. Qualitative Methods On the other end of the spectrum, qualitative research methods are often identified with the constructivist philosophical orientation as these methods lend themselves to involve analyzing descriptive or narrative data. Frequently, an inductive approach is applied. The constructivist framework supports the idea of multiple realities, relationship between researcher and subject being dependent, knowledge as negotiated, value-bounded inquiry, and context-free generalizations as impossible (Tashakkori & Teddlie, 99, p. 6). The purpose of qualitative methods is often exploratory in nature. Mixed Methods Research Considered a middle between quantitative and qualitative research methods, mixed methods research is often identified with the pragmatic philosophical orientation, placing emphasis on a what works best approach, supporting both singular and multiple realities (Creswell & Plano-Clark, 7). The pragmatic approach has multiple stances (p. ) when it comes to axiology to include both biased and unbiased perspectives (p. ), allowing for the use of mixed methods in educational research. A mixed methods study involves the collection or analysis of both quantitative and/or qualitative data in a single study in which the data are collected concurrently or sequentially, are given a priority, and involve the integration of the data at one or more stages in the research process. (Creswell, Plano-Clark, Gutmann & Hanson,, p. )
Methodology influence on research perceptions Mixed method research involves the use of both narrative and numeric data; employment of both inductive and deductive logic; integration of categorical and contextual data analysis; and both confirmatory and exploratory in nature. Research Question As Feuer, Towne, & Shavelson () argue, No method is good, bad, scientific, or unscientific in itself: Rather, it is the appropriate application of method to a particular problem that enables judgments about scientific quality (p. ). Regardless of researchers philosophical orientation the choice of method(s) and methodology should be driven by the research questions which provide organization, focus, coherence, relevance and direction for conducting good educational research (Onwuegbuzie & Leech, 6). Questions help narrow the scope of the research objective and purpose. In addition, Onwuegbuzie & Leech make the case that research questions dictate the type of research design used, the sample size and sampling scheme employed, and the type of instruments administered as well as the data analysis techniques (p. 7). Teddlie and Tashakkori (9) describe the research question as a dual focal point that liaises between what is known about the topic before the study and what is learned about the topic during the study. Everything flows through and from the research questions. (p. 9). Thus, careful consideration for alignment of research questions to primary research methodology is vital for ensuring high quality education research. Method This study further investigates the findings of Bradley, Royal, Cunningham, Weber, and Eli (), where survey items connected to methodology revealed great variation in perceptions of characteristics of good educational research. The purpose of this study is to explore whether this variation existed across the research methodology variable respondents self-identified
Methodology influence on research perceptions primary research methodology. Specifically, is differential item functioning occurring within the category of primary research methodology? Response Frame The target population consisted of faculty and approximately graduate students. The response frame consisted of 6 faculty (n=; 9.7%) and graduate students (n=76; 6.%) from the COE. The university s COE includes programs departmentalized in six areas of educational research and practice, including: curriculum and instruction; educational leadership studies; educational and counseling psychology; educational policy and evaluation; kinesiology and health promotion; and special education and rehabilitation counseling. A demographic description is presented below for both faculty (see Table ) and graduate students (see Table ). Table Demographic Characteristics of Faculty Respondents (n = ) Characteristic n % Faculty Rank Instructor.9 Lecturer Assistant Professor 9. Associate Professor. Professor. Visiting Professor.9 Adjunct Professor. Part-time Professor/Instructor.9 Other.9 Years as a faculty member (at any higher education institution) Less than year 6. - years. -6. 7-9 6. - 7. 6-. Greater than years 6.
Methodology influence on research perceptions 6 Table (continued) Characteristic n % Tenure Status Tenured 7.9 On tenure track, but not tenured. Not on tenure track.6 Age 7 or older. 6-7 6. -6. - 7.9 -.9 Less than.6 Table Demographic Characteristics of Graduate Student Respondents (n = 76) Status Characteristic n % Graduate Assistant. Research Assistant 9.7 Teaching Assistant.9 Doctoral Student 9.7 Master s Student. Specialist Degree Student. Other 6. Years as a graduate student (at any institution) Less than a year. Between - years 9. Between - years. Between -6 years.7 7 or more years 6. Age Under - 6. -9.7 -. -9. - 6.7-9 6.7-6. or older.
Methodology influence on research perceptions 7 The majority of the faculty sample consists of full-time professors distributed across the categorizations of assistant, associate, and professor. Half of the faculty members have been a faculty member at a higher education institute for either - years or greater than years, with age ranges of either - or -6, and a majority with tenured status. The graduate student sample consists of a majority of doctoral students that are either research or teaching assistants. The majority of the students have been a graduate student for more than three years with ages ranging from -9. With regard to the collective sample, 9% (n = ) of respondents reported being of White/Caucasian ethnicity. Asian and Pacific Islanders comprised.9% and African American or Black respondents accounted for.%. Data Collection Using a list of emails provided by the dean s office for all College of Education faculty and graduate students, an invitation to participate in the study was sent to the entire population. The email invitation included the link to the survey and statements regarding purpose and participant rights. Three reminders were subsequently sent, including final notice of a closing date, which was two and a half weeks following the opening date. Responses were collected and stored on a secure web server provided by the survey software program SurveyMonkey (7). All data were downloaded onto the primary investigator s computer in aggregate form to ensure respondents were unidentifiable. Instrumentation The survey instrument, in its entirety, consisted of 6 items, including demographics. Research statements (see Table 6, appendix) were partitioned into three domains: methodological (7 items), ethical ( items) and theoretical ( items). These statements were created from guidelines for good or quality research set forth by research organizations in educational
Methodology influence on research perceptions disciplines, such as the American Educational Research Association (AERA) and the American Psychological Association (APA). Items utilized a four-point rating scale to endorse level of agreement. In the demographic section, respondents were asked to select the response category best describing their primary research methodology (entirely qualitative, mostly qualitative, mixed methods, mostly quantitative, entirely quantitative). Table Primary Academic Interest for Faculty and Graduate Student Respondents (n = 7) Demographic Variable n % Primary Research Methodology Entirely qualitative % Mostly qualitative % Mixed Methods (/) % Mostly Quantitative 6% Entirely Quantitative 7 % Out of the 7 respondents completing the survey, the only category resulting in less than ten respondents for primary research methodology was entirely qualitative. Analysis Crucial to determining common characteristics of good educational research is the quality of the instrument used to examine participant responses. In an effort to ensure the quality of the measure, a rating scale model was applied with persons and items to test the overall fit of the data to the model using Winsteps version.6. software (Linacre, 6). Missing data were treated as missing, with no values being imputed. The stability of the measurement instrument follows the procedures outlined in Bradley, et al. ().
Methodology influence on research perceptions 9 One of the more straightforward approaches to assessing construct bias, particularly for rating scales is the Rasch model (Wright & Masters, 9; Selner-O Hagan, Kindlon, Buka, Raudenbush, & Earls, 99). As with all IRT models, the Rasch model assumes that an additive structure underlies the observed data, that both participants and items can be arrayed on a continuum, and that the items have equal discriminative power (Kan, Breteler, Van der Ven, & Zitman, 99). Rasch measurement allows for analyses of individual differences in response tendencies, as well as an item s discrimination (i.e., how well the item is able to discriminate between examinees holding different levels of a latent construct) and an item s difficulty. Rasch scaling procedures are used to determine equivalence at the item level, and if differences are obtained, to determine the pattern of responding across groups of respondents. Each person is accompanied with a person label indicating their reported primary research methodology. Relative frequency tables illustrate a general picture of the characteristics the respondents associated with good educational research separated by primary research methodology. Differential item analysis within WINSTEPS version.6. was used to determine if there are statistically significant differences in characteristics of good research across the three domains by self-identified primary research methodology. A separate calibration t-test approach (Wright and Stone, 979) was used to determine differential response across the subpopulations. While there is not a consensus on statistical significant differences, Smith () suggested using t-statistic estimates greater than or less than - for detecting differential item responding. Results and Discussion The response categories provided on the survey for respondents to classify their primary research methodology included entirely qualitative, mostly qualitative, mixed methods, mostly quantitative, and entirely quantitative. Participants were asked to select only one. The forced
Methodology influence on research perceptions choice response was difficult for some respondents, as they did not feel comfortable selfclassifying. Even so, these categories were selected to construct the boundaries often presented in the literature, the typical quantitative/qualitative split, with mixed methods serving as the balancing point. To begin the discussion, frequency tables were produced to illustrate the overall picture of item responses by rating scale category as well as separated by primary research methodology. Seven out of ten ethics items, five out of twelve theory items, and nine out of seventeen methods items had less than % disagreeing with the statement. Table Counts and percents of each rating scale category for methods survey items Item SD D A SA M (%) (%) 9 (%) 7 (%) M (%) (%) 9 (%) (%) M (%) ( %) (9%) 6 (%) M (%) 9 (%) 9 (69%) M (%) 9 (%) 66 (9%) 6 (6%) M6 (%) 9 (%) 6 (77%) M7 (%) 6 (%) (6%) M (%) (%) (%) 9 (66%) M9 (%) 9 (%) 96 (7%) M 6 (%) (6%) (%) 7 (7%) M (%) (%) (%) 76 (7%) M 7 (7%) 7 (%) (6%) 7 (%) M 7 (%) 6 (%) (%) 7 (%) M (%) 6 (7%) (%) 7 (%) M 9 (%) 9 (6%) (%) (%) M6 (%) (%) (9%) (%) M7 (%) (9%) (%) 6 (7%)
Methodology influence on research perceptions Table Counts and percents of each rating scale category for ethics survey items Item SD D A SA E (%) (%) (%) E (%) (%) (6%) E (%) 6 (%) E (%) (9%) 6 (7%) (%) E (%) 9 (7%) (%) 69 (%) E6 (%) (%) (%) 7 (7%) E7 (%) 6 (7%) (%) E 7 (%) 6 (%) (%) E9 (%) (6%) (%) E 6 (%) 7 (7%) Table 6 Counts and percents of each rating scale category for theory survey items Item SD D A SA T (%) 67 (%) (%) T (%) (%) 6 (%) 6 (9%) T (%) (6%) 6 (%) (%) T (%) (%) 6 (%) (%) T (%) (7%) 7 (6%) (%) T6 (%) (6%) 7 (7%) (%) T7 7 (%) 79 (6%) 9 (%) T (%) 9 (%) 6 (%) 7 (%) T9 (%) 7 (7%) (%) T (%) 7 (%) (%) T (%) (%) 79 (6%) (6%) T (%) (7%) (%)
Methodology influence on research perceptions All respondents either agreed or strongly agreed to ethics items and. In contrast, methods item resulted in less than % endorsing the statement. For a list of the full statement associated with each item, please see appendix. Discussing the fit of the data to the Rasch rating scale model is essential prior to ensure meaningful interpretations of differential item functioning across primary research methodological classifications. The person separation (.) and reliability (.6) estimates were deemed acceptable for the analysis to continue; likewise, the items resulted in acceptable item separation (7.6) and reliability (.9) estimates. According to Wright and Stone (), fit statistics above one standard deviation of the infit and outfit mean are labeled as item misfit. Nine of the ten items resulting in high fit statistics were methods items, indicating unexpected responses are being produced by these items. Items,,,,, 7, and ethics item resulted in a high outfit statistic while items,,, 6,,,, and 7 resulted in high infit statistics. Methods item, ethics item, and theory items and resulted in a decrease in the average person measure over the rating scale categories, but this was a result of one or two respondents disagreeing with this item unexpectedly. More problematic decreases in average person measures over the rating scale occurred for items and 6 in ethics, in theory, and, and in methods. Finally, items are inspected for differential item functioning across the variable classifying respondents by their reported primary research methodology. The second variable of primary research methodology resulted in several significant classes differentially responding across items. Respondents not reporting their primary research methodology were found significantly different for methods items 6,,,,, 6 and ethics items and 9. Respondents choosing entirely qualitative and mostly qualitative were found significantly different for methods items and. Respondents choosing entirely quantitative were
Methodology influence on research perceptions significantly different for methods items,, and. Mostly quantitative respondents were significantly different for methods item. A frequency table was produced for the items resulting in differential item functioning to depict the distribution of responses across the rating scale by self-identified primary research methodology. Table 7 Counts of rating scale category by primary research methodology for DIF methods items Item Method SD D A SA M 9 6 M 6 6 9 M6 9 6 M 7 6 7 7 M 6 6 7 6
Methodology influence on research perceptions Table 7 (continued) M 9 9 M 9 9 9 M 9 6 7 M6 9 Table Counts of rating scale category by primary research methodology for DIF ethics items Item Method SD D A SA E 7 E9 9
Methodology influence on research perceptions Conclusion The intent of educational research is to advance knowledge in the field through inquiry into the world of education. With much agreement within the domain of ethics and theory and considerable variability within the methodology domain, it seemed reasonable to examine the differential item functioning within each domain across the respondents reported primary research methodology. Evidence from this study supports the lack of consensus among educational researchers of what constitutes good research. Sadly, it also demonstrates that the great divide of methodological framework is evident a qualitative/quantitative divide of sorts. Ultimately, the goal is for this research to improve the overall quality of educational research by encouraging researchers to be mindful of characteristics of their own research and hopefully, to be more receptive to the varying perceptions of their peers. References Bradley, K.D., Royal, K.D., Cunningham, J.D., Weber, J., & Eli, J.A. (). What constitutes good educational research? A consideration of ethics, methods and theory. Mid-Western Educational Researcher, (), 6-. Borko, H., & Whitcomb, J.A. (). Teachers, teaching, and teacher education: Comments on the national mathematics advisory panel s report. Educational Researcher, 7(9), 6-7. Creswell, J.W. (). Research design: Qualitative, quantitative, and mixed methods approaches ( nd Ed.). Thousand Oaks, CA: SAGE Publications, Inc. Creswell, J.W., & Plano-Clark, V.L. (7). Designing and conducting mixed methods research. Thousand Oaks, CA: SAGE Publications, Inc.
Methodology influence on research perceptions 6 Creswell, J.W., Plano-Clark, V.L., Gutmann, M., & Hanson, W.E. (). Advanced mixed methods research designs. In A. Tashakkori, & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 9-). Thousand Oaks, CA: SAGE Publications, Inc. Feuer, M.J., Towne, L., & Shavelson, R.J. (). Scientific culture and educational research. Educational Researcher, (), -. Morgan, D. (7). Paradigms lost and pragmatisms regained: Methodological implication of combining qualitative and quantitative methods. Journal of Mixed Methods Research,, -76. Onwuegbuzie, A.J., & Leech, N.L. (6). Linking research questions to mixed methods data analysis procedures. The Qualitative Report, (), 7-9. Tashakkori, A., & Teddlie, C. (99). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: SAGE Publications, Inc. Teddlie, C., & Tashakkori, A. (9). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks, CA: SAGE Publication, Inc.
Methodology influence on research perceptions 7 Appendix of Items Ethics E E E E E E6 E7 E E9 E High-quality research abides by ethical standards. High-quality research informs participants about the consent to research. High-quality research should protect the safety and welfare of participants. High-quality research minimizes the use of techniques or methodologies that have negative social consequences. High-quality research adheres to established institutional policies for conducting research. High-quality research recognizes all researchers who have contributed substantively to the study. High-quality research should only be conducted by investigators who have completed ethics training. High-quality research findings must be disseminated to the professionals within the discipline. Results from high-quality research should be disseminated to the public. High-quality research should abide by the ethical guidelines recognized by the related professional organizations in that field. Theory T T T T T T6 T7 T T9 T T T High-quality research reflects the researchers' awareness of both their own and competing paradigms. High-quality research should provide the rationale for the conceptual orientation of the study. High-quality research should provide the rationale for the theoretical orientation of the study. Educational researchers should adhere to the standards of their own theoretical perspectives to achieve high-quality research. Reliability, validity and trustworthiness are the most important considerations in high-quality research. High-quality research provides objective answers to research questions. High-quality research assists in developing theories to explain phenomena. High-quality research aims to develop generalizations based on findings. High-quality research adds to our understanding of the issues in education. High-quality research should be useful to other professionals within the discipline. High-quality research connects the work to its impact on human well-being. High-quality research merges reason and value.
Methodology influence on research perceptions Methods M High-quality research should be evidence-based. M High-quality research should be original. M High-quality research should be reproducible. M High-quality research should be attentive to detail. M High-quality research should consider efficiency in choosing research methodology. M6 High-quality research should be methodologically sound. M7 High-quality research should follow a clear logic of inquiry. M High-quality research should be mindful of differences within the research population (e.g. cultural, religious, gender, etc.). M9 High-quality research should demonstrate awareness that different types of research call for different data collection techniques. M The appropriate methods ensure the high-quality of research data. M High-quality research requires research methods and techniques based on the nature of the research questions. M High-quality research requires random sampling. M High-quality research requires quantifiable measures of results. M High-quality research consists of experimental studies that yield prescriptions for action. M High-quality research can be determined solely by examining the research methodology. M6 High-quality research should provide the rationale for the methodological orientation of the study. M7 Educational researchers should adhere to the standards of their own methodological perspectives to achieve high-quality research.