Measuring the Effectiveness of STEM Talent Initiatives for Middle and High School Students

Transcription

1 Measuring the Effectiveness of STEM Talent Initiatives for Middle and High School Students Linda Brody Johns Hopkins University America has long been concerned about providing an adequate talent pool to stay on the forefront of technology and scientific problem-solving. This concern has increased during periods when world events have suggested we may not be competitive in these areas compared to other nations. Notably, the launching of Sputnik led to numerous educational initiatives designed to enhance the preparation of our nation s youth in scientific areas. The 1970s brought a focus on barriers that might be holding females back from aspiring to careers in these fields, and concern about the under-representation of females and minority groups in scientific fields continues today, along with a fear that the overall pipeline to these careers, regardless of gender or ethnic background, is too small to meet American needs in the face of global competition. As a consequence, numerous intervention programs have been developed specifically for middle and high school students to encourage their interest and learning in scientific disciplines in the hope of increasing the number of students in the pipeline to STEM careers. Educators whose focus is gifted students have also contributed to the development of programmatic initiatives for middle and high school students. Their efforts to provide challenging opportunities for advanced students have led to the establishment of magnet middle and high schools for gifted students in many school systems, as well as supplemental educational opportunities such as academic summer programs, distance education courses, competitions, internships, and mentoring programs. Many of these resources provide participants with increased exposure to STEM content and career opportunities. The scope and quality of research and evaluation of the effectiveness of the different STEM talent initiatives has varied enormously. The focus on barriers to females pursuing math and science careers produced some of the most systematic research that has been done, and the findings may be transferable beyond gender (e.g., see Chipman, Brush & Wilson, 1985). Similarly, insights resulting from investigations into the factors that affect the participation of minority groups in STEM careers can have broader implications for program development (e.g., see Boothe & Stanley, 2004). Many talent initiatives for middle and high school students, however, lack systematic evaluative procedures or are evaluated by their own program administrators or principal investigators rather than by outside experts. This practice can produce biased results, and rarely is a longitudinal component included. In spite of a lack of really definitive research about what works, the research that has been done, along with retrospective observations by practicing scientists, suggests that the following elements seem to be important in encouraging students to aim for mathematical and scientific careers, and many talent initiatives try to address these needs. Solid preparation from an early age in math and science content

2 2 Experience with hands-on content Awareness of the utility of school-based learning in the workplace Exposure to role models and mentors who work in these fields Access to peers who share these interests In addition, researchers have found family variables (education of parents, careers of parents, support for student interests, etc.) and certain affective and personality traits (investigative and theoretical interests, assertiveness, motivation, etc.) to be important components of success in these fields. These factors, of course, are harder to influence than the environmental ones. It seems to be the interaction of these personal and family traits with experiential factors over time that determines the outcome, making it difficult to measure the impact of one specific program. For more information see also Benbow, Lubinski, & Sanjani (1999); Bloom (1985); Brandwein & Passow (1988); Fennema & Leder (1990); Fox, Brody, & Tobin (1980); Olszewski-Kubilius & Yasumoto (1995); Piirto ( 1991); Pyryt, Masharov, & Feng (1993); Simonton (1994); Subotnik & Steiner (1994); Wieczerkowski & Prado (1993); Zuckerman (1977). Discussion Questions What participant outcomes of middle school and high school STEM talent development initiatives should be used as measures of success? In order to evaluate the effectiveness of talent development initiatives, goals need to be established, and these are likely to vary depending on the nature of the program. Certainly, one might have different goals for a short intervention than for a magnet high school. In general, one might expect the following short-term outcomes to result from most initiatives designed to increase the pipeline into STEM careers. Pre- and Post-test measures using questionnaires, tests of knowledge, and other assessment tools should evaluate the following outcomes of the intervention. Learning of content. At some level, knowledge of scientific principles, enhancement of technical or problem solving skills, or other preparation to go on to more advanced work should be expected. Enhanced positive attitudes even excitement -- toward the discipline Increased self-confidence in one s ability to excel in this field Increased knowledge of the value of math and science in the workplace and increased awareness of math and science careers Removal of any barriers to possible advancement in math and science that existed prior to the intervention Long-term goals for intervention programs are likely to focus primarily on whether participants enter STEM careers at rates higher than would be likely without having had the

3 3 intervention or use their increased knowledge and appreciation of math and science in other careers (e.g., science journalism). Of course, other life experiences are likely to positively or negatively influence decisions over time, making it somewhat difficult to distinguish the impact of the a single intervention, complicating longitudinal research. To assess whether students maintain momentum following the intervention and stay on a path toward a STEM career, the following should be assessed at key points in follow-up studies of the population: Participation in subsequent optional math and science opportunities at the precollege level and beyond High achievement in rigorous math- and science-related courses at the precollege level and beyond Choice of a STEM discipline as a college major Honors and awards for math and science achievement Positive attitudes toward math and science content Confidence in one s ability to excel in scientific fields. Interest in and continued progress toward achieving entry requirements for a STEM career Entry into a STEM career If the goal is to impact the level of achievement or eminence attained in the career, not just entry into a STEM career, further evaluation of career achievements will be needed. To assess this, one might evaluate: Status of position at entry Rapidity and level of promotions Honors, awards, and recognitions Publications and patents Evaluations by peers What are the strengths and weaknesses of such outcome measures as: Going on to be STEM college majors? For most students, college major is a strong indicator of future career goals, though it is not a guarantee. It certainly can be considered an indication of strong interest in the area at the time the major is selected. As an outcome measure to evaluate precollege intervention programs, it seems to be a critical component. If students do not go on and major in a STEM field, they are unlikely to return to the pipeline later, and, if they do, it will likely be the result of something other than the intervention being investigated. However, one must be cautious about attributing causation to the intervention. Students who select STEM majors may always have had this interest and thus the intervention might not have been a critical component in influencing the choice of major. It s also possible that another experience was a more important influence on the decision than the one being evaluated. This argues against simplistic evaluations. The researcher must be aware of all the experiences the

4 4 student has had in order to evaluate the role that was played by the one under investigation and also consider such factors as interest and motivation. Going on to successful careers in the sciences? This is the goal of most initiatives aimed at increasing the pipeline to STEM careers so it clearly is a vital component of studying the impact of the intervention. As with college major, however, more factors must be assessed before attributing causation of the selection of this pathway to the middle or high school intervention program that is being evaluated. It is likely that ability, interests, a wide variety of educational and occupational experiences, and a certain amount of luck will have played a role in determining the career one ultimately chooses to pursue. This does not negate the possibility that the middle or high school initiative might have played a role, but the size and impact of the role this program played can only be determined as part of a more comprehensive study. High likelihood of becoming future scientific leaders and innovators? If the ultimate goal of the intervention is to produce scientific leaders and innovators, one would try to identify candidates with the most potential, provide experiences to help them succeed, and then assess over time the degree to which they achieve this goal. This was Julian Stanley s goal when he established the Study of Mathematically Precocious Youth he hoped to find and nurture future research mathematicians and physicists who would make groundbreaking discoveries. Studies of this group must continue into adulthood to determine if the prediction becomes a reality. In studies of students with promise, however, one must be realistic about how rare true leadership, innovation, and eminence are in any field and recognize that this level of achievement is not likely to be a common outcome in any group under investigation. Terman s (1925) work has been criticized because there weren t more Nobel Prize winners in the group, and yet large numbers achieving this distinction should not be expected from any one group of individuals. Still, if a goal of the intervention is to produce scientific leaders and innovators, this outcome is important to measure. Once again, though, we must be cautious about attributing causation from the intervention that took place in middle or high school. It is likely that many factors will have influenced the individual s achievement by the time eminence is achieved, and it is difficult to assess the role the particular intervention under investigation played. Careful monitoring of influences along the individual s educational and career path, as well as retrospective data, is critical to drawing any conclusions about this target goal. Having a good enough grasp of STEM to apply it creatively and appropriately to any discipline or career path?

5 5 This is an important component that is often overlooked. Typical evaluations of math and science interventions seek affirmation of positive results by assessing how many students directly enter STEM career fields. Yet our society benefits when our political leaders, journalists, poets, teachers, patent lawyers, and philosophers have a solid grounding in understanding the physical, biological, and technical world in which we live. More studies should include this aspect and evaluate this influence. But it is challenging to do so. First, it definitely requires funds to carry the research well into the individual s career. Another issue is that the subjects may not even realize how relevant STEM knowledge is to their careers, making measurement difficult. What other definitions of success could be or should be used? We might consider the possibility that the main goal of a STEM middle or high school program should be to excite students enough for them to seek and take advantage of the next related opportunity, rather than to expect long-term effects. Thus, the intervention being evaluated could be considered successful if students move on to subsequent opportunities, then those opportunities become responsible for keeping them in the pipeline beyond that point. For example, if a student enjoys a middle school summer program in archaeology and subsequently volunteers on a dig, the summer archaeology class was successful in achieving the goal of enhancing interest in this particular STEM content area. After working on the dig, the student works with a mentor in a museum, takes college courses in archaeology, and volunteers on more digs. While the middle school summer program played an important role in getting this pathway started, the subsequent experiences become more important in determining whether the student stays on this path. If she doesn t because of poor mentoring or because she doesn t enjoy fieldwork, it doesn t mean the middle school program failed; it did its job by spurring the student to investigate the next opportunity. On the other hand, if the student becomes a career archaeologist, credit to the middle school program should not be overplayed when so many other critical experiences came into play. The subsequent opportunities need to be evaluated separately for their effectiveness in keeping the pipeline going. Related to this is the possibility that it is useful for students to find out what they don t like, as well as what they do like when they participate in special programs. In most programs for gifted students, we seek to help students maximize their potential in whatever fields they excel, whether it is science, or the arts, or humanities. Those with multiple talents may need to explore a variety of areas before finding their niche. And, if students work with a mathematician and find the experience too solitary for their social needs and this results in their not taking up a coveted space in a graduate program that could go to someone more suited for this work, that might be considered a useful and successful outcome, even though the stated goal of keeping someone in the pipeline was not achieved. What are the implications for research, policy, and practice? We know from existing research that interventions aimed at increasing participation in STEM fields do work for some students some of the time; in fact, for many of those for whom they work, the interventions have played a defining role in their decision-making about the direction their lives have taken. However, we also know that those same interventions may have

6 6 had little impact on other students. And still other students for whom the impact at the time was positive may lose interest in pursuing a STEM career later, and we don t always know why. The bottom line is that it is complicated. The interaction of personality, interests, opportunity, knowledge, zeitgeist, and a certain amount of luck seems to be required to help a future scientist or mathematician excel, so teasing out the role of one specific intervention can be challenging. We also rarely do controlled experiments. If we believe an intervention is likely to be effective, it can seem unethical to hold students back from participating for the sake of research. It also can be difficult to control enough variables to have a good comparison group; certainly willingness to participate cannot be the criteria for those selected while the uninterested students become the comparison group. So we must be cautious about interpreting results. The implications of all of this are as follows: We need more well-funded research studies that measures both the short-term and longterm effects of programs. Thus, studies must be longitudinal. This research must include assessment of personality, interest, motivation, and family and school variables, in addition to measuring the direct effects of numerous educational opportunities over time. The goal of research should be to learn how to maximize matching programs to the particular students who would most likely benefit from the particular type of intervention. Program development in the short term should not wait for these studies, as studies done to date and anecdotal comments by participants are evidence that they do make a difference for many of the students. We need more publicly funded programmatic opportunities as many that exist today are in the private sector, and fees can be a barrier to program participation. Intervention in middle and high school can cease to have an impact if additional interventions are not available for college and graduate students. A long-term solution may be needed with ongoing support and mentoring, particularly for at-risk populations. References Benbow, C. P., Lubinski, D., & Sanjani, H. E. (1999). Our future leaders in science; Who are they? Can we identify them early? In N. Colangelo & S. G. Assouline (Eds.), Talent Development III (pp ). Scottsdale, AZ: Great Potential Press. Bloom, B. S. (Ed.). (1985). Developing talent in young people. New York: Ballantine Books. Boothe, D. & Stanley, J. C. (2004). In the eyes of the beholder: Critical issues for diversity in gifted education. Waco, TX: Prufrock Press. Brandwein, P. F., & Passow, A. H. (Eds.) (1988). Gifted young in science: Potential through performance. Washington DC: National Science Teachers Association.

7 7 Chipman, S. F., Brush, L. R. & Wilson, D. M. (Eds.) (1985). Women and mathematics: Balancing the equation. Hillsdale, NJ: Erlbaum. Fennema, E. & Leder, G. C. (Eds.) (1990). Mathematics and gender. New York: Teachers College Press. Fox, L. H., Brody, L., & Tobin, D. (Eds.) (1980). Women and the mathematical mystique. Baltimore: Johns Hopkins University Press. Olszewski-Kubilius, P., & Yasumoto, J. (1995). Factors affecting the academic choices of academically talented middle school students. Journal for the Education of the Gifted, 18 (3), Piirto, J. (2004). Understanding creativity. Scottsdale, AZ: Great Potential Press. Pyryt, M. C., Masharov, Y. P., & Feng, C. (1993). Programs and strategies for nurturing talents/fits in science and technology. In K. A. Heller, F. J. Monks, & A. H. Passow (Eds.), International handbook of research and development of giftedness and talent (pp ). New York: Pergamon. Simonton, D. K. (1994). Greatness; Who makes history and why. New York: Guildford Press. Subotnik, R. F. & Steiner, C. L. (1994). Adult manifestations of adolescent talent in science: A longitudinal study of 1983 westinghouse science talent search winners. In R. F. Subotnik & K. D. Arnold (Eds.), Beyond Terman: Contemporary longitudinal studies of giftedness and talent (pp ). Norwood, NJ: Ablex Publishing Co. Terman, L. M. (1925). Mental and physical traits of a thousand gifted children. Stanford, CA: Stanford University Press. Wieczerkowski, W., & Prado, T. M. (1993). Programs and strategies for nurturing talents/gifts in mathematics. In K. A. Heller, F. J. Monks, & A. H. Passow (Eds.), International handbook of research and development of giftedness and talent (pp ). New York: Pergamon. Zuckerman, H. (1977). Scientific elite: Nobel laureates in the United States. New York: Free Press. Reprinted by Transaction Publishers, New Brunswick, NJ, 1996.