Advantages and Limitations of a Mixed Method, Technology Based Approach to Evaluating a National, University Level STEM Education Program

Transcription

1 Advantages and Limitations of a Mixed Method, Technology Based Approach to Evaluating a National, University Level STEM Education Program Frances Lawrenz University of Minnesota, USA lawrenz@umn.edu This paper discusses the strengths and limitations of a technology based mixed method evaluation of a US national education program for plant breeders. The education program also uses technology to deliver instruction and create community. Mixed methods included participant observation, surveys, case studies, focus groups, social network analysis and interviews. The results of the evaluation of the program are presented and the implications surrounding the use of mixed methods and technology to evaluate such programs are discussed. Introduction Mixed method research and evaluation are tools commonly used by STEM researchers and evaluators to investigate program or policy merit and worth (Creswell, Trout, & Barbuto, 2002; Teddlie & Tashakkori, 2008). In many situations, a combination of different types of information provides multiple stakeholders with information they can use with confidence (Chelimsky, 2007). Greene (2007) described that mixed method studies may be generative, as paradox and contradiction are engaged and fresh insights, new perspectives, and original understandings emerge (2007, p. 103). Other mixed method authors share this belief in the promise of mixed methods. For example, Tashakkori and Teddlie (2003) used the term gestalt to indicate how inferences from mixed methods may be greater than the single method components. Barbour (1999) described mixed methods as a whole greater than the sum of its parts. By using multiple, diverse methods, researchers may corroborate findings to increase confidence in the inferences drawn from them. The advent of the internet has allowed communication among formerly isolated or widely dispersed individuals. In the case of STEM, this enhances the educational and research capacity by connecting students and researchers at distant locations. STEM education programs that take advantage of these new opportunities are different from traditional programs and hence require rethinking of traditional approaches to STEM program evaluation. These new programs need to be evaluated using technology based mixed methods approaches (Green, 2007; Cresswell & Plano Clark, 2010). This paper discusses the implementation of such an evaluation and its advantages and limitations to provide guidelines for future evaluations. The project evaluated, the Triticeae Coordinated Agricultural Project (TCAP), is a complex, University-level educational program being implemented at different institutions across the United States. The TCAP is funded by the United States Department of Agriculture. It is an effort to improve the quality of wheat and barley breeding and increase the numbers, diversity and skills of plant breeders. The educational component of TCAP includes researchers and faculty from over 20 universities across the United States. This component consists of providing education and research opportunities for graduate students in plant breeding programs and partnering with faculty from eight minority serving institutions (MSIs) to develop interest in plant breeding careers among MSI undergraduate students. The project is relying heavily on developing a networked learning community. This complex project has five major clusters of activities including: relationship building with MSIs, fostering of social networking between and among all groups involved in the project, online course development and presentation, incorporation of inquiry based learning into courses, and development and use of motivational videos/curriculum in courses at a variety of levels. The belief is that these activities will produce more, better prepared and connected plant breeders who come from diverse communities, as well as improved content and teaching procedures in plant breeding courses. The objective of the evaluation was to document baseline perceptions of the participants. This would provide information upon which to base future programming as well as an initial data point from which to measure future change as the TCAP continues. The objective of this presentation is to document the advantages and limitations of a technology based mixed method approach to evaluation of a complex, University-level STEM education initiative.

2 256 Proceedings of episteme 5, India Initiation of the evaluation The first step in the process of designing the technology based, mixed methods evaluation was to develop a logic model for the educational program. Logic modeling is a fundamental process to help program staff to map out their program s theory of action and inform evaluation activities and priorities. Modeling is usually conducted face-to-face in a group process over a period of several days (see Mayeske & Lambur, 2001; Taylor-Powell & Henert, 2008; W.K. Kellogg Foundation, 2004). However a face-to-face process was impossible in this setting and telephone and web based conferencing was used. The evaluation team played a neutral role helping members of the TCAP educational committee come to agreement on what activities and goals were important. The process included four meetings over a period of one month, with each meeting lasting approximately two hours. Adobe Connect, a web conferencing tool, allowed for everyone to see a shared screen where documents were presented and edited simultaneously. Telephone conferencing also supplemented the calls when there were audio issues with Adobe Connect. Discussion took place primarily through the audio and written chat features. The logic model that was developed is presented on the following page. Based on the logic model the mixed methods evaluation was designed to assess outcomes through monitoring, a focus group, social networking, surveys, interviews, participant observation, and case studies. A combination of web and phone conferencing was used, as well as in-person activities to foster relationships. The evaluation also made use of the available audio recordings of the communications that took place among participants via the web community hub. This allowed the evaluation to examine the type and depth of communication in terms of how passive or active the members were in connecting with each other. The evaluation will continue for the five years of the project. What is reported on here are the results from the first year. Methods used in the evaluation Monitoring data were collected by the project personnel and included demographic information as well as school background and participation in project activities. A 90-minute focus group with nine MSI faculty members who were not part of TCAP was held early in the project to obtain their perceptions of the proposed TCAP small grant program where funds from TCAP would be awarded to MSI institutions to support students and research at the MSI. The discussion explored the potential for MSI participation and how best to structure that participation. Three online surveys were conducted, one of the principal investigators (PIs) in the project with 42 of the 54 PIs responding (78%), one of the graduate students with all 12 of the students responding and one of the principal investigators from the MSIs with all eight responding. All surveys were designed collaboratively with project staff members and think alouds were conducted to verify understanding. The graduate student survey assessed perceptions of plant breeding education, interest and motivation in the plant breeding field, perceptions of the TCAP educational programming, and collaborative networks with other students, faculty, and researchers within and outside of the TCAP. Similarly the TCAP PI and MSI PI surveys also assessed perceptions of plant breeding education and collaborative networks, but additionally assessed perceptions of TCAP educational components and relationships with PIs from TCAP and MSI institutions. Telephone interviews were conducted with TCAP students (n = 6) and PIs (n = 7). Students and PIs were selected for the interviews based on their background and university characteristics. Students were asked about their perceptions of the educational component of TCAP, the online learning community, and their relationships with other TCAP students and faculty. PIs were asked about their relationships and collaborations with other TCAP and MSI institutions, their involvement in and perceptions of the educational component of TCAP and their beliefs about recruitment efforts. The educational components of the TCAP included online course meetings as well as webinar broadcasts. One member of the evaluation team acted as a participant observer of these online activities and also perused all of the related archived materials. Case studies were conducted at three MSIs, selected based on their diverse sizes, locations, and types of institutions. The site visits lasted one to two days and included observations and interviews with members of the MSIs, as well as participating faculty members and students.

3 Advantages and Limitations of a Mixed Method, Technology Based Approach to Evaluating a National, University Level STEM Education Program 257 Results from the evaluation Figure 1. TCAP Logic Model Overall the MSI faculty participants in the focus group were optimistic and supportive of the potential that participation in the TCAP could provide for them and their students. Participants wanted more details about programmatic elements. The comments about how to structure the participation in the TCAP could be summarized into one overarching theme with several sub themes. The overarching theme was equality. The subthemes included respectful collaboration, mutual understanding, articulation, continuing relationships, and faculty and student support. Survey data indicated a range of findings. For instance, TCAP graduate students felt very confident in working collaboratively (83%) while almost half of the students surveyed (42%) felt very confident about genetics when asked to rate plant breeding knowledge areas. With respect to TCAP faculty, the majority of PIs (81%) reported research to be the most important educational component of TCAP. About one-third of the PIs identified lack of interest and/or awareness of plant breeding as the top barrier to increasing the number of underrepresented minorities in plant breeding. A cross-comparison study of TCAP faculty and graduate students was conducted by comparing survey results of similar sections. Findings indicate that students seem to have less confidence in areas that PIs consider to be very valuable and some areas students report having confidence in are areas was considered by the PIs to be less valuable Both students and PIs agreed about the importance of educational processes in plant breeding. Students felt the most important aspect of TCAP was the opportunity to network with faculty and students from across the nation. They appreciated having a network of shared resources and expertise. They invariably mentioned how they felt TCAP would help them develop collaborative networking opportunities. The faculty interviews show that they are excited about the TCAP and the opportunities it presents for working together about important research topics. The PIs interviewed also felt that the TCAP has expanded the types of researchers that are involved in the discussions about TCAP research. The webinars exposed TCAP graduate students to each other s research as well as content pertinent to the student s research. Additionally, the webinars provided opportunities for students to troubleshoot research

4 258 Proceedings of episteme 5, India projects. While the online course meetings achieved the goal of exposing students to discussions of content with each other and in a whole class discussion, the discussion board itself did not appear to serve its intended goal of spearheading group discussions. The case studies of the MSI institutions provided strong contextual information. Students at MSI institutions expressed a genuine excitement to be a part of the TCAP. Students hoped for content knowledge and expanded networks and were excited to belong to a TCAP community and to interact with TCAP graduate and undergraduate students. All students interviewed indicate that they are looking at this experience as an opportunity to improve their skills and help them prepare for graduate school. How best to recruit students was an issue brought up by both the students and the faculty members. All also mentioned the potential for development of strong, collaborative relationships between the MSIs and TCAP institutions. While faculty and students appeared enthusiastic about the educational component, they did offer suggestions for improvement. Advantages and limitations of the evaluation approach The use of mixed methods in this evaluation had both advantages and limitations. The advantages included a variety of information, different perspectives on the project, and a balance of measurement biases. The limitations included need for a variety of expertise, time for data collection and report development, and synthesis of a large amount of information. Evaluators turn to mixed method methodology to address the practical challenges and resultant uncertainty of using any single method (O Cathain, Murphy,& Nicholl, 2007). Both post-positivist and interpretive methods have limitations. For example, Weiss (1995) described the challenges to conducting randomized controlled studies, and the mechanisms to identify causal mechanisms in interpretive research have been considered to require further development (Johnson & Onwuegbuzie, 2004). The methods used in this evaluation represented a range of methodological approaches which provided a balance of the biases inherent in each individual approach while providing diverse information that produced a comprehensive picture of the project. Surveys, focus groups and interviews share the bias of self-report but interviews allow for more probing and more emergence of the ideas of the interviewees as opposed to the preconceived ideas of the survey developers. Focus groups allow for group speak and the development of group consensus. Participant observation and coding of recorded interaction provides an independent look at the conduct of the project. Case studies allow the development of situated and contextualized understanding of the project operation. On the other hand, implementation of these different techniques required the use of a team of evaluators with a range of expertise in the different methodologies as well as a substantial amount of time and expense to collect and analyze all of the data. It also was difficult to synthesize the different types of data into a coherent picture of the project. How and when to best to mix methods is a subject of debate (Greene, 2007; Creswell & Plano Clark, 2010). Using mixed methods is recommended when convergence of findings is desired for triangulation, when diverse perspectives are desired to expand one s understanding of a project, and/or if multiple stakeholders have differing information needs. The technological aspects of this evaluation also had advantages and limitations. Advantages included archiving of the online interactions, reduction of time and travel costs, convenience, and potential to involve everyone from everywhere. Limitations included misunderstanding, technological difficulties, and lower rates of interaction and awareness. The presence of the internet allows quick communication and written documentation of conversations. It also allows for the digital capturing of video presentations or other types of communication that can then be used as data for evaluations. This was a real advantage in the assessment of the communication and social networking components of the project. Because of the networked nature of the community, the evaluation was able to understand the real time vibe of the network. However, how to optimize communication via the network is an ongoing issue. In conducting the evaluation discussion via internet and telephone did not always result in clear communication. Audio issues were a re-occurring problem and because the webcam feature was used only once during the process, it was impossible to gain understanding through body language and facial expressions. Sometimes some of the members met together face-to-face, while others logged in from their own computers and phones. This resulted in differential discussion and feelings of involvement. The use of the internet lessened geographical barriers and provided the opportunity for everyone to participate while saving travel time and cost. The convenience of the meetings also allowed the scheduling of more frequent meetings. To optimize the use of technology in a STEM evaluation, the technology needs to be carefully tested, technological failures need to be anticipated and plans for how to deal with any developed in advance.

5 Advantages and Limitations of a Mixed Method, Technology Based Approach to Evaluating a National, University Level STEM Education Program References Barbour, R. S. (1999). The case for combining qualitative and quantitative approaches in health services research. Journal of Health Services Research and Policy, 4(1), Chelimsky, E. (2007). Factors influencing the choice of methods in federal evaluation practice. New Directions for Evaluation, 2007(13), Creswell, J. W. & Plano Clark, V. L. (2010). Thousand Oaks, CA: SAGE Publications. 259 Designing and Conducting Mixed Methods Research (2nd ed.). Creswell, J. W., Trout, S., & Barbuto, J. E. (2002). A Decade of Mixed Methods Writings: A Retrospective. Research Methods Forum. pp Published by the Academy of Management. ARD # Greene, J. C. (2007). Mixed methods in social inquiry. San Francisco: Jossey-Bass. Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), Kellogg Foundation. (2004). W.K. Kellogg Foundation Logic Model Development Guide. Retrieved from: Mayeske, G.W. & Lambur, M.T. (2001). How to design better programs: A staff-centered approach to program logic modeling. Journal of Extension, 39(3). Retrieved from: O Cathain, A., Murphey, E., & Nicholl, J. (2007). Integration and publications as indicators of yield from mixed methods studies. Journal of Mixed Methods Research, 1(2), Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research. Thousand Oaks: SAGE Publications. Taylor-Powell, E. & Henert, E. (2008). Developing a logic model: Teaching and training guide. Madison, WI: University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation. Retrieved from: Teddlie, C. B. & Tashakkori, A. (2008). Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. Thousand Oaks, CA: SAGE Publications. Weiss, C. H. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In J. Connell, A. Kubisch, L. Schorr, & C. Weiss, (Eds.), New approaches to evaluating community initiatives: Concepts, methods, and contexts, Washington, DC: Aspen Institute.