Summary of Evaluation Plan for the Illinois Math Science Partnership. Measuring State Level Goals

Transcription

1 Summary of Evaluation Plan for the Illinois Math Science Partnership Measuring State Level Goals Overview of the Illinois MSP grant program and the evaluation framework for state-level initiatives. Elizabeth Oyer Originally Released: 3/07/2008 Revised: 4/02/2008 Revised: 5/4/2009 Revised: 5/5/2009

2 Overview of Illinois MSP Program Evaluation Framework Participants There are 10 universities with 23 grants. Some university partners have multiple grants. Grants cover elementary programs, biology, earth and space science, environmental science, secondary math, physics, chemistry, IT/pre-engineering, and health science tech. Here is a breakdown of the grants by university: Institution IIT ISU NIU SIU-C SIU-E UIUC Loyola Aurora Bradley DePaul Grand Total Biology Chemistry Earth/Space Elementary Environment IT/pre-eng Physics Sec Math Grand Total Core program components Each of the programs has the following core elements: New Graduate level program granting master s degree for participants they are working with Illinois Board of Higher Ed to get these approved (exception: health program participants don t have teaching certificate already since many are nurses etc.) Partnerships with STEM organization or business, university, and local school district emphasis is on strong collaboration between partners Teacher Action Research each project has to incorporate teacher action research. The only current requirements for action research are that all research problems be aligned to some project goal. Illinois MSP Evaluation Framework 2

3 3

4 Project Outcomes MSP outcomes are framed around the CCSSO matrix of PD outcomes. There are basically five categories of outcomes: 1. Quality of PD Activities 2. Change in Teacher Content Knowledge 3. Change in Instructional Practice (including strategies, resources, and content knowledge) 4. Change in Student Achievement 5. Other: Organizational Support and Change Overview of Meta-Analytic Model for Evaluating Illinois MSP Outcomes Based on past research, policies, and legislative focus, there are several important themes to an effective MSP evaluation. The framework should incorporate mixed methods to address questions of generalizeability as well as serious influences of individual contexts. In addition, the framework should reflect an understanding of the role and importance of diversity for the STEM initiatives' success, for achieving and sustaining successful working partnerships, as well as for evaluating and generating realistic and comprehensive recommendations. The heart of the Illinois state-level evaluation is the synthesis of local evaluation results to evaluate the overall effect of program intervention components on key program outcomes. Local evaluations examine specific contextual factors in each funded partnership that affect implementation and outcomes. The Illinois cross-site evaluation framework uses these local evaluation results in a systematic way as an indicator of the effectiveness of the Illinois MSP project overall. By applying meta-analyses to this multisite evaluation, a global statistical summary of program effectiveness for the individual sites as well as across sites is made more meaningful and accurate (Kalaian, 2003).Because multiple site and cluster evaluations represent an opportunity to increase the generalizability of evaluation findings, the proposed statistical methodology improves knowledge about the effectiveness of the range of math and science partnership initiatives funded by the Illinois MSP program. Meta-analyses will be applied as appropriate to model the rate and scale of change of students in intervention classrooms compared with students in control classrooms. Separate meta-analysis models will examine the magnitude of effectiveness across local partnerships in terms of math and nice, elementary, middle, and secondary programs in addition to other classifications as appropriate. Moderating variables include levels of teachers participation in quality PD, teachers levels of integration of key intervention components (e.g., STEM technologies, appropriate teaching methods, new content), general measures of teachers instructional practices (based on SEC and implementation measures), achievement covariates, and partnership model characteristics. In addition to meta-analyses, regression modeling and/or inferential statistical techniques (as appropriate to specific questions), results from qualitative analyses and analysis of artifacts will be triangulated with quantitative results to provide a more complete picture of Illinois' progress toward MSP goals. Specifically, triangulation of changes in quantitative IMSP Evaluation Framework 4

5 outcomes (e.g., teacher content knowledge test scores) with action research artifacts, field notes from interviews and observations, and rubric summaries will be completed as appropriate. Figure 1. MSP Evaluation Model IMSP Evaluation Framework 5

6 Background/Context for local MSP grants Benchmarks: Methodology 100% of grants provide relevant background/context data Data: Demographic and background information about participating teachers and Program Profiles (Required) This is individual data. Much data will be exported from APR (assuming functionality completed by fed programmer). Otherwise, grants will re-enter relevant data in Portal. Individual teacher data related to teachers relevant background data (e.g., current degrees, endorsements, and highly qualified status) will be reported. Narrative of important context considerations (Required) Narratives of important contextual information not captured in descriptive program statistics will be provided by each grant. Analyses: Descriptive summary and content analyses of program elements, participants, and partners across MSP grants. CCSSO Matrix Outcome A (Quality of PD Activity) APR Part VIII A (1, 4) APR Part IV A,B,C,D APR Part V A,,B,C APR Part VI A,B Benchmarks: At least 95% of participants complete course evaluations/pd logs for each grant Data: Course evaluations (Required) This is aggregate data. Evaluations should incorporate feedback specific to quality PD, including ratings of course design, implementation, content, instructional materials. IMSP Evaluation Framework 6

7 PD logs (Required) This is aggregate data. Logs documenting teacher contact hours in PD outside MSP activities completed annually. Administrator Academy data (As Appropriate) This is aggregate data. Summaries of Administrator Academy courses/credits and participation by local partners. Templates for summaries of Administrator Academy data will be developed based on common reporting elements as well as local needs. ERC Partner Interviews, Surveys, and Artifacts (Required) See Appendix A This is individual data. Analyses: Descriptive summary of program quality across grants. Elements of PD quality variable will be used as moderator variables in meta-analyses. CCSSO Matrix Outcome B (Changes in Teacher Content Knowledge) APR Part VIII A (2,3,5,6) APR Part VII F APR Part VII G Benchmarks: 80% of intervention and control group teachers complete annual test 100% of grants upload data to MSP Portal by deadline Cumulative increases in subject matter knowledge (or mastery) by 50% each year (until mastery is reached) 100% completion of requirements for degrees and highly qualified status 80% completion rate across IHEs Data: Pre/Posttests of Content Knowledge (Required) This is aggregate data. Local evaluators will administer tests of content knowledge to all teachers participating in their partnership (and an appropriately matched sample of teachers not participating in their partnership -control teachers- if they have an experimental design with a control group) beginning in Phase II of the grant. These data points will provide an indicator of changes in teacher content knowledge. Course Grades (Required) This is aggregate data. Teacher participant grades for MSP courses. IMSP Evaluation Framework 7

8 ERC Partner Interviews, Surveys, and Artifacts (Required) See Appendix A This is individual data. Teacher action research (Required-No time frame specified) See Appendix B. This is individual data. Classroom Observation Data (Optional) This is aggregate data. Where applicable, data from classroom observations will be used to triangulate and describe changes in levels of teacher knowledge. See Appendix C for Classroom Observation requirements related to implementation levels. Portfolios with student artifacts (Optional) This is aggregate data. Where applicable, summaries of rubric ratings will be used to triangulate and describe changes in levels of teacher knowledge. Administrator Academy data (As Appropriate) This is aggregate data. Summaries of Administrator Academy data as relevant to teacher content knowledge change. Templates for summaries of Administrator Academy data will be developed based on common reporting elements as well as local needs. Analyses: Descriptive summary of progress toward benchmarks. Pre/post tests will be analyzed using meta-analyses. Each local grant will be required to submit "quality of the instrument" data (like reliability measures) as well as summary data required for meta-analyses across sites (e.g., mean, standard deviation, effect sizes, n). Partnership survey results from teacher participants will be analyzed using descriptive statistics to evaluate teachers perceptions of knowledge change. Two reports will be generated based on teacher action research projects. The first report on teacher action research reports will be a descriptive report detailing the number of studies, general content focus, and methodologies of the action research projects completed each year. The second report on teacher action research will address the goal of constructing larger narratives or general theories about the problems that the teacher-researchers investigated or the outcomes that they achieved. Since the only requirement for the teacher action research projects is that they align with a goal of their local IMSP, the specific focus of the meta-synthesis cannot be predetermined.. IMSP Evaluation Framework 8

9 CCSSO Matrix Outcome C (Instructional Strategies, Resources, and Content) APR Part VII E APR Part VII I APR Part IX A Benchmarks: 90% of intervention and control group teachers submit measures of practice Increases in integration of information, methods or other resources each year by at least 50% of teachers; integration by 75% of teachers by end of Year 4 50% cumulative increase in integration of technologies (general and STEM) 50% cumulative increase in integration of SBRTM Data Comprehensive self-reported skill/ practice (Required) This is individual data. Teachers will annually complete a survey of practices in the relevant content areas (e.g., SEC). These data will be used to identify changes in practices from year to year among participating teachers. It is expected that the participating teachers practices will include increased evidence of effective instructional practices. Composite variables that align (on an individual teacher level) data from the content portions and instructional strategies portions of the SEC will be generated using latent trait analyses. These variables will be used in meta-analyses and linear models as appropriate measures of teacher practice ERC Partner Interviews, Surveys, and Artifacts (Required) This is individual data. See Appendix A. Measures of Program Implementation, including classroom observation (Required) This is aggregate data. See Appendix C. Teacher action research (Required-No time frame specified) This is individual data. See Appendix B. Portfolios with student artifacts (Optional) This is aggregate data. Where applicable, summaries of rubric ratings will be used to triangulate and describe changes in levels of teacher knowledge. Analyses Data from implementation measures, ERC Partner site visits and surveys, and student artifacts will be analyzed with descriptive statistics, content analyses, and narrative summary as appropriate. IMSP Evaluation Framework 9

10 CCSSO Matrix Outcome D (Student Achievement) APR Part VIII B (1,2,3,4,5,6,7,8) APR Part IV E APR Part VII H Benchmarks: 30% increase in ISAT "Meets Expectations," "Exceeds Expectations," and 30% increase in effect sizes from synthesized classroom based assessments. Data: Student achievement data (Required) This is aggregate data. Collected from multiple sources, including: SIS demographic and assessment data (ISAT and Prairie State scores as available) Student enrollment and grades in advanced math or science courses Student enrollment and grades in AP / IB programs Student participation in supplemental extra-curricular math/science programs (not sponsored by the Partnership) Classroom-based, action-research data on student performance Student interest in math/science Teacher action research (Required-No time frame specified) This is individual data. See Appendix B. Portfolios with student artifacts (Optional) This is aggregate data. Where applicable, summaries of rubric ratings will be used to triangulate and describe changes in levels of teacher knowledge. Analyses: Pre/post tests will be analyzed using meta-analyses. Each local grant will be required to submit "quality of the instrument" data (like reliability measures) as well as summary data required for meta-analyses across sites (e.g., mean, standard deviation, effect sizes, n). Moderating variables include levels of teachers participation in quality PD, teachers levels of integration of key intervention components (e.g., STEM technologies, appropriate teaching methods, new content), general measures of teachers instructional practices, achievement covariates, and partnership model characteristics. Data from ERC Partner site visits and surveys will be analyzed with descriptive statistics, content analyses, and narrative summary as appropriate. IMSP Evaluation Framework 10

11 Two reports will be generated based on teacher action research projects. The first report on teacher action research reports will be a descriptive report detailing the number of studies, general content focus, and methodologies of the action research projects completed each year. The second report on teacher action research will address the goal of constructing larger narratives or general theories about the problems that the teacher-researchers investigated or the outcomes that they achieved. Since the only requirement for the teacher action research projects is that they align with a goal of their local IMSP, the specific focus of the meta-synthesis cannot be predetermined. CCSSO Matrix Outcome E (Other: Sustained Administrative Support) APR Part VII I APR Part VII J Benchmarks: 50% cumulative increase in ratings of quality, confidence, effectiveness, collaborative decision-making of partnership (NOTE: Data collection for can be the baseline). 50% cumulative increase in administrative program support (NOTE: Ratings by partners as well as the ERC site evaluators) 100% of LEAs employ Highly Qualified teachers (NOTE: Data collection for can be the baseline.) 100% of IHEs commit to sustaining programs Data: Demographic and background information about participating teachers and Program Profiles (Required) This is individual data. Much data will be exported from APR (assuming functionality completed by fed programmer). Otherwise, grants will re-enter relevant data in Portal. Individual teacher data related to teachers relevant background data (e.g., current degrees, endorsements, and highly qualified status) will be reported. Narrative of important context considerations (Required) Narratives of important contextual information not captured in descriptive program statistics will be provided by each grant. ERC Partner Interviews, Surveys, and Artifacts (Required) See Appendix A. This is individual data. IMSP Evaluation Framework 11

12 Analyses: Descriptive summary and content analyses of program elements, participants, and partners across MSP grants. Data from ERC Partner site visits and surveys will be analyzed with descriptive statistics, content analyses, and narrative summary as appropriate. IMSP Evaluation Framework 12

13 Appendix A Illinois Mathematics and Science Partnership (IMSP) Site Visit Evaluation Framework Partnership Process Focus Analysis Question Data Source Analyses Strategies Respondents / Participants 1) Partnership Composition 2) Organizational Structure Size and diversity of partnership decision-makers and stakeholders MSPs will categorize their organizational structure based on HUD s Office of University Partnerships (HUD, 2002, pp ). Who are the partners across Illinois MSP grants? How diverse are the stakeholders in positions of power? What are the contributions of the partners? What is the geographic dispersion of the partnership? How are the IMSPs organized? Where is the IMSP located? Who are the decision-makers? Partnership Background Survey (based on Scherer, 2004). Interview and Site Visit Protocols adapted from Brinkerhoff (2002), HUD (2002), Lewis (2000), Mattessich (2001), Borden and Perkins (1999). Partnership Background Survey Artifact Analyses Interview and Site Visit Protocols Descriptive statistics, content analyses, and narrative summary Descriptive statistics, content analyses, and narrative summary Grant director or grant evaluator Grant director or grant evaluator IMSP Evaluation Framework 13

14 Partnership Process Focus Analysis Question Data Source Analyses Strategies Respondents / Participants 3) Action Plan and Operational Guidelines Artifact Analyses Grant director or evaluator 4) Qualities of the Partnering Relationship and Maintaining the Partnership Review of the IMSP program and articulation of formal commitments and understandings between all partners. Characteristics are associated with quality partnerships: Mutuality & trust Leadership Resources Collaboration and mechanisms of communication. What is the scale of the IMSP project? What formal agreements are in place to define, establish, and support communication and collaboration between partners? To what extent is there a mutual need, trust, equality in decisionmaking, resource exchange, transparency, respect, representation, enthusiasm, and sustained understanding between partners and stakeholders across MSP grants? To what extent is leadership collaborative and transformational? Interview and Site Visit Protocols Artifact Analyses Interview and Site Visit Protocols Interview and Site Teacher Satisfaction Survey (Adapted from Wolf, 2003). Partner Satisfaction Survey (Adapted from Wolf, 2003). Descriptive statistics, content analyses, and narrative summary Descriptive statistics, content analyses, and narrative summary Grant director, grant evaluator, key representatives from all partners (e.g., industry, LEAs, non-profit organizations), teachers. IMSP Evaluation Framework 14

15 Partnership Process Focus Analysis Question Data Source Analyses Strategies Respondents / Participants 5) Project Outcomes Evidence of implementation and project outcomes 6) Sustainability Evidence of established infrastructure for IMSP goals. How did the IMSP perform and what were the observable outcomes? Does the IMSP show evidence that the university-communityschool partnership can be sustained in the future? Interview and Site Visit Protocols Interview and Site Teacher Satisfaction Survey Partner Satisfaction Survey Interview and Site Visit Protocols Interview and Site Teacher Satisfaction Survey Partner Satisfaction Survey Descriptive statistics, content analyses, and narrative summary Descriptive statistics, content analyses, and narrative summary Grant director, grant evaluator, key representatives from all partners (e.g., industry, LEAs, non-profit organizations), teachers. Grant director, grant evaluator, key representatives from all partners (e.g., industry, LEAs, non-profit organizations), teachers. IMSP Evaluation Framework 15

16 1. Partnership Composition. For site visits in Year 1 Planning Phase History: What is history of the university in the community or with the partners? Did the university (or parts of it) have experience with or a record of engagement in community outreach, community service or applied research in the past? [Were these efforts coordinated? Was there a pre-existing partnership/program within the University that preceded the IMSP? If so, what role does that office have on the work of the IMSP? What is the relation between the IMSP and the program? Is there a University unit that oversees the work of this center? What was the relationship between the university and the community partners in the IMSP prior to the ISBE application?] For collaboration between colleges within IHE: What was the relationship among the colleges prior to the IMSP? Were their prior relationships with each other similar or different? In what way? Process. What was the process for creating the IMSP? [How did the IMSP partners develop the application to ISBE? Did community or school partners contribute to the application, review the draft, etc.? How did the IMSP partners refine the partnership relationships after receiving the grant? Are there any groups that should have been included that were not part of the IMSP? ] For collaboration between colleges within IHE:: Did both/all schools participate in developing the IMSP proposal? How were the roles defined? How were responsibilities assigned? Staffing. How is the IMSP staffed? [Have new staff been hired to conduct the work of the IMSP? What positions were filled? Where did the candidates come from? How many staff members work (will work) for the IMSP? What policies are in place for the replacement of staff as needed?] For collaboration between colleges within IHE: Are IMSP staff drawn from both/all institutions? Are faculty and students from both/all institutions involved in IMSP? Context. What is the school environment for IMSP reform? [What are the major educational initiatives in the city/region/state? How has the IMSP related to these efforts? Can the IMSP have improved coordination with other programs to achieve greater outcomes? Are there resources for and attention to these issues? What is the context for university funding? What other programs are competing for university resources and attention?] For collaboration between colleges within IHE: How does the institutional context for the IMSP differ among the schools? IMSP Evaluation Framework 16

17 2. Organizational Structure of Partnership. Structure. What is the structure of this IMSP? Does the IMSP have an advisory board(s) and what is its role? Is there a sense of equity among the partners? [Who are the board members and what are their respective affiliations? What is the governance of the IMSP? How are decisions made? By whom? Are community / school perspectives valued and respected? What are the roles of the university, community/ school in the IMSP? To what degree have universitycommunity/school relationships constituted a partnership? (Not at all, somewhat, to a moderate degree, to a great degree)] For collaboration between colleges within IHE: What are the respective roles of the colleges in the IMSP? Do all schools participate equally in governance and decisionmaking? How is accountability by each school to the partnership determined? How are imbalances in institutional resources compensated for? Is the IMSP seen as an opportunity for faculty and student collaboration among the schools, or as individual efforts under a single banner? Location within the University. Is there a specific space designated for the IMSP within the university? What parts of the university are involved with the IMSP? What structures, policies and/or practices of the university support community outreach or hinder outreach activities? [Where is the IMSP physically housed? What was the rationale for its placement? Is the IMSP embraced by the leadership of the university? If so, how?] For collaboration between colleges within IHE: Where is the IMSP located in the consortium? Why? Artifacts: IMSP Membership list, IMSP/ IHE organizational chart 3. Action Plan and Operational Guidelines IMSP Program Areas. What is the nature of the IMSP program and how ambitious is it? [What program areas does the IMSP address? What is the scope and sequence of the new program?] For collaboration between colleges within IHE: Are program areas divided by schools? If so how? Or do the schools work jointly on the same project areas? Operational Guidelines. What formal agreements are in place to define, establish, and support communication and collaboration between partners? Who established these guidelines? Artifacts: Logic Model, Evaluation Framework, Data Analysis Plans, IBHE proposal IMSP Evaluation Framework 17

18 For site visits in Years 1-4 Planning and Implementation Phases 4. Quality of Partnerships Mutuality & Trust. Do the goals and objectives of the IMSP address mutual needs across partners? What are the perceptions of trust across partners? Is there a sense of safety for sharing of information and resources? What steps have partners taken to build trust? What is the nature of most interactions between partners? Face-to-face? ? What was the nature of relationships between partners before the IMSP? How respectful is the IMSP to differences in cultural and organizational norms, values, and beliefs? How transparent are the IMSP operations? Is their equality in decision-making? Is there reciprocal accountability? Is there a balance in the representation of all partners in the IMSP? Does leadership across partners work closely together? Is there enthusiasm surrounding IMSP goals and activities? For collaboration between colleges within IHE: What is the nature of relationships between colleges? Is there a sense of equality in decision-making and resources? Is there a respect for differences in cultures? Is there shared enthusiasm for the IMSP? Artifacts: Meeting agendas, minutes Leadership. Who are the leaders of the IMSP? [Who led the development of the IMSP application? Are there one or more persons taking leadership? What is their role in the institution? What is their continuing role in the IMSP? Was there participation from the top levels of the institution?] For collaboration between colleges within IHE: Is leadership for the IMSP shared among the colleges? Is there a key person at each school leading the IMSP? Is there participation from top levels at both/all schools? Resources. Has the IMSP received matching funds? [From what sources? How does this compare with the initial proposal? Are there adequate resources to accomplish IMSP goals? Are resources sufficient for all partners?] limited not just to financial resources but extending to managerial and technical skills, contacts, information and the like; For collaboration between colleges within IHE: How will resources be divided among the institutions? Did all/both schools provide matching funds? Artifacts: Budget summary/narrative Communication. What are the guiding principles for your IMSP? Is there shared decisionmaking between partners? What are the primary vehicles for communication? Is there a formal management and communication plan? How are conflicts resolved in the partnership? IMSP Evaluation Framework 18

19 Artifacts: Meeting agendas, meeting minutes, newsletters, websites, other forms/policy statements 5. Performance and Outcomes For site visits in Years 2 4 Implementation Phase Performance. How has the IMSP performed? [What areas did the IMSP address most successfully? How? What areas was the IMSP not successful in addressing? Why? Did the IMSP areas addressed serve community/school priorities? Has the IMSP pursued the major strategies it originally planned? If not, why not? ] Did the IMSP program produce benefits to the community/school and to the University? What were the major outcomes from the IMSP activities? For collaboration between colleges within IHE: What was the performance of the collaboration between colleges? In addition to discussing the performance of each of the participants, explain how well the IMSP performed as a collaborative. Capacity Building. Were there capacity building outcomes from the IMSP activities? [Did the IMSP assist community organizations or schools? How? Which ones? What were the results of that assistance? Did the capacity of the organizations increase? In what ways? Did the IMSP efforts lead to changes in the skills of any of the members of the participating organizations? Were there capacity building outcomes for the university as a result of the IMSP?] Community / School Outcomes. Were there changes in the physical, institutional or economic conditions of the community/school in the areas that the IMSP focussed on? [ What were they? How much did the IMSP contribute to those results?] Community / School Change. Was there any change in the level of social capital (e.g. civic engagement, health of community institutions, trust between community and the institution) in the community? [How much did the IMSP contribute to those changes?] Information and Knowledge. What were the outcomes in information and knowledge? [Who used this information and, and how? Did IMSP efforts yield reports, articles, etc. derived from IMSP data, information bases, or experiences? If so, were these reports accessible to the community? How were they distributed, or made available? Did community members contribute to them? If new information bases were developed were they useful to the community? To the university? To the Department? To the city or other agencies? Was the information used? How? Did the IMSP reports contribute to applied research? Basic theory?] Institutional Change. Did the IMSP lead to changes in attitudes, values or behaviors of the participating organizations? [Did the IMSP lead to changes in attitudes, values, policies, practices or behaviors of the university? E.g., in hiring, tenure and IMSP Evaluation Framework 19

20 promotions, procurement, etc. How did the IMSP lead to changes in the content of university courses? To new courses? Changes in degree programs? To new programs? Did the IMSP lead to changes in the means or methods of instruction and the dissemination of information in the university?] IMSP Evaluation Framework 20

21 For site visits in Year 4 Implementation Phase 3. Sustainability Profitability of the Partnership for Community/Schools. Did the IMSP provide benefits to partner organizations or projects? Examples. Were there costs (time, effort, funding, etc.) or obstacles to the community partners working with the IMSP? What were they? [Did the community partners find the benefits of the IMSP outweigh their costs? Could the IMSP have increased the benefits to the partners? How? Could the IMSP have reduced the costs to the partner organizations? How?] Profitability of the Partnership for the University. Did the University benefit from participating in the IMSP? [How? What costs or obstacles did the University incur by participating in the IMSP? Did the benefits to the University outweigh the costs it incurred? Were any of the costs or benefits unanticipated]? For collaboration between colleges within IHE: Did each of the participating schools benefit from the partnership with the community/schools? With each other? Did the consortium present more costs or obstacles to the participating institutions? Fairness. Were the benefits derived from the partnership fair and commensurate with the contributions made by the partners? [Were the benefits to the community/school partners greater than the benefits to the university? About the same? Or less than what the university received?] For collaboration between colleges within IHE: Did the participating institutions derive benefits commensurate with their contributions? Did one school gain more or incur greater costs than the others? Future of the Partnership. Do the partners believe that the IMSP should be continued? [Do they believe it will be? Why or why not? If so, will they continue to participate? Should the composition of the IMSP partnership in the future be the same? Should new partners be added? Should some partners be replaced? Examples. Should the activities of the IMSP in the future be the same? Should new activities be added? Should some activities be dropped? Examples.] For collaboration between colleges within IHE: Will the institutional partnership continue? Has it led to other collaborative efforts among the participating institutions? Institutionalization. Is the IMSP institutionalized in the university or on a trajectory toward institutionalization? [Have the functions of the IMSP been integrated into the university? How? Did the IMSP efforts lead to changes in the structure and policies of the participating organizations? E.g., creation of new units, reorganization or change in mission of old units, IMSP Evaluation Framework 21

22 changes in budgeting practices, hiring practices, purchasing, creation of new coordinating agencies or collaborations, or changes in the mission of the university?] For collaboration between colleges within IHE: Has each school integrated the functions of the IMSP? Have policies and structures of all/both institutions been influenced by the IMSP? Resources to Sustain the Partnership. Are there resources now available to sustain the IMSP? [If not, are the partners taking reasonable and timely steps toward securing such resources? What other sources of funding have supported the IMSP? Was the IMSP successful in finding private funding for the partnership? Have resources been identified to continue the partnership after IMSP funding expires? If yes,from what sources? In what amounts? Are there any new sources of funding? If not,what is the strategy to respond to the expiration of IMSP funding?] For collaboration between colleges within IHE: Will all/both institutions continue to support the partnership? Documenting the Site Visit 1. SUBMIT A LIST OF ALL INDIVIDUALS INTERVIEWED IN PREPARING THIS REPORT, WITH THEIR ORGANIZATIONAL AFFILIATION, TITLE AND CONTACT INFORMATION. 2. SUBMIT A LIST OF ALL ARTIFACTS USED IN PREPARING THIS REPORT AND INDICATE WHICH ONES YOU HAVE COPIES OF. 3. SUBMIT INTERVIEW NOTES AND CODES THROUGH IMSP DATA PORTAL. 4. TO BE ANSWERED AFTER THE SITE VISIT AND THE REPORTING FOR HAVE BEEN COMPLETED: On reflection, what lessons, if any, do you think we learn from this site? What interesting and broadly relevant issues does it raise? What common problems have been encountered by local stakeholders that have been addressed in an interesting way? Rate the Partnership Quality across the 6 dimensions. IMSP Evaluation Framework 22

23 Partnership Composition Organizational Structure Action Plan & Operational Guidelines Partnership Quality Performance & Outcomes Sustainability Beginning Emerging Developing Transformative IMSP Evaluation Framework 23

24 IMSP Teacher Satisfaction Survey 1 Please indicate your level of satisfaction with each aspect of your MSP participation. (Likert scale: Very Satisfied Very Dissatisfied) Vision and Mutuality 1. Clarity of the vision for IMSP goals and objectives 2. Planning process used to prepare the IMSP objectives 3. Follow-through on IMSP activities 4. Efforts to promote collaborative action with other educators 5. Efforts to promote collaborative action with STEM professionals outside the university 6. Processes used to assess teachers needs 7. Processes used to assess my students' needs 8. Participation of influential people in the IMSP that represent teachers interests 9. Diversity of partners and participants 10. Respect, acceptance and recognition of my contributions to reaching the IMSP goals 11. Resources provided by my district and/or school to support my commitment to the IMSP grant Comments: Leadership 12. Strength and competence of IMSP leadership 13. Sensitivity to cultural issues 14. Opportunities for me to take leadership roles 15. Trust that partners and participants afford each other Comments: Communication 16. Use of the media to promote awareness of the IMSP goals, actions, and accomplishments 17. Communication among members of the partnership 18. Communication between the IMSP and the broader community 19. Extent to which IMSP participants are listened to and heard 20. Working relationships established with school officials 21. Information provided on issues and available resources Comments: Technical Assistance: 22. Strength and competence of IMSP faculty and staff 23. Training and technical assistance provided by faculty and staff 24. Help given the participants in meeting IMSP requirements 25. Help given the participants to become better able to address and resolve their concerns Comments: 1 Adapted from Annual Satisfaction Survey for Community Coalitions. Wolff,T (2003).. A practical approach to evaluating coalitions. In T.Backer(Ed.) Evaluating Community Collaborations. Springer Publishing IMSP Teacher Survey Adapted from Wolf Satisfaction Survey of Coalitions Page 24

25 Progress and Outcomes: 26. My progress in learning new content through the IMSP grant. 27. My progress in using new instructional resources through the IMSP grant. 28. My progress in using new STEM technologies through the IMSP grant. 29. My progress toward meeting endorsement or certification requirements. 30. My access to STEM industry experts through the IMSP grant. 31. My access to mentors because of the IMSP grant. 32. Fairness with which resources and opportunities are distributed 33. Capacity of IMSP teachers to give support to each other 34. IMSP grant's contribution to improving science and/or mathematics instruction in my school. Comments: Please indicate how much you agree or disagree with the following statements. (Likert scale: Strongly Agree Strongly Disagree) Job Satisfaction 35. In most ways, being a STEM teacher is close to my ideal. 36. My conditions of being a STEM teacher are excellent. 37. I am satisfied with being a STEM teacher. 38. So far I have gotten the important things I want to be a STEM teacher. 39. If I could choose my career over, I would change almost nothing. Comments: Sustainability 40. I received important professional benefits from my participation in the IMSP. 41. The benefits I received were worth the time, effort, and cost I invested in the IMSP. 42. The benefits I received were commensurate with the contributions I made to the IMSP. 43. I strongly believe the IMSP should be continued. 44. I will participate fully in IMSP activities in the future. 45. The IMSP activities need to be dramatically improved to make it worth my investment. 46. I will continue to integrate IMSP strategies and materials into my classroom instruction. 47. I have access to the resources I need to continue to integrate IMSP strategies and materials into my classroom instruction. 48. My district will support my continued integration of IMSP strategies and materials into my classroom instruction. Comments: IMSP Teacher Survey Adapted from Wolf Satisfaction Survey of Coalitions Page 25

26 IMSP School Partner Satisfaction Survey 2 Please indicate your level of satisfaction with each aspect of your IMSP partnership. (Likert scale: Very Satisfied Very Dissatisfied) Vision and Mutuality 1. Clarity of the vision for the IMSP goals and objectives 2. Planning process used to prepare the IMSP objectives 3. Follow-through on IMSP activities 4. Efforts to promote collaborative action 5. Efforts to promote collaborative action between STEM professionals and teachers 6. Processes used to assess teachers needs 7. Processes used to assess students' needs 8. Participation of influential people in the IMSP that represent a variety of interests 9. Diversity of partners and participants 10. Respect, acceptance and recognition of my contributions to reaching the IMSP goals 11. Resources provided by the partner districts and/or school to support the IMSP grant Comments: Leadership 12. Strength and competence of IMSP leadership 13. Sensitivity to cultural issues 14. Opportunities for me to take a leadership role 15. Trust that partners and participants afford each other 16. Transparency of decision-making. Comments: Communication 17. Use of the media to promote awareness of the IMSP goals, actions, and accomplishments 18. Communication among members of the partnership 19. Communication between the IMSP and the broader community 20. Extent to which IMSP participants are listened to and heard 21. Working relationships established with school officials 22. Information provided on issues and available resources Comments: 2 Adapted from Annual Satisfaction Survey for Community Coalitions. Wolff,T. (2003). A practical approach to evaluating coalitions. In T.Backer(Ed.) Evaluating Community Collaborations. Springer Publishing IMSP Teacher Survey Adapted from Wolf Satisfaction Survey of Coalitions Page 26

27 Technical Assistance: 23. Strength and competence of IMSP faculty and staff 24. Training and technical assistance provided by faculty and staff 25. Help given the participants in meeting IMSP requirements 26. Help given the participants to become better able to address and resolve their concerns Comments: Progress and Outcomes: 27. Progress in improving teachers content knowledge through the IMSP grant 28. Progress in teachers access and use of new instructional resources through the IMSP grant 29. Progress in teachers access and use of new STEM technologies through the IMSP grant 30. Teachers progress toward meeting endorsement or certification requirements 31. Effective collaboration between STEM industry experts and teachers through the IMSP grant 32. Teachers access to mentors through the IMSP grant 33. Fairness with which resources and opportunities are distributed 34. Capacity of IMSP teachers to give support to each other 35. IMSP grant's contribution to improving science and/or mathematics instruction in schools Comments: Please indicate how much you agree or disagree with the following statements. (Likert scale: Strongly Agree Strongly Disagree) Sustainability 36. My district received important professional benefits from participation in the IMSP. 37. The benefits my district received were worth the time, effort, and cost invested in the IMSP. 38. The benefits my district received were commensurate with the contributions made to the IMSP. 39. I strongly believe the IMSP should be continued. 40. I will participate fully in IMSP activities in the future. 41. The IMSP activities need to be dramatically improved to make it worth my district s investment. 42. The composition of the IMSP needs to be expanded or changed to be more effective. 43. My district has changed the structure, policies, or functions to institutionalize the IMSP goals and activities. 44. My district intends to sustain IMSP activities after the expiration of grant funds. IMSP Teacher Survey Adapted from Wolf Satisfaction Survey of Coalitions Page 27

28 45. My district is actively seeking alternative funds to sustain IMSP activities after the expiration of grant funds. Comments: IMSP Teacher Survey Adapted from Wolf Satisfaction Survey of Coalitions Page 28

29 IMSP Industry Partner Satisfaction Survey 3 Please indicate your level of satisfaction with each aspect of your IMSP partnership. (Likert scale: Very Satisfied Very Dissatisfied) Vision and Mutuality 1. Clarity of the vision for the IMSP goals and objectives 2. Planning process used to prepare the IMSP objectives 3. Follow-through on IMSP activities 4. Efforts to promote collaborative action between partners 5. Efforts to promote collaborative action between STEM professionals and teachers 6. Participation of influential people in the IMSP that represent a variety of interests 7. Diversity of partners and participants 8. Respect, acceptance and recognition of my contributions to reaching the IMSP goals 9. Resources provided by the partner organizations to support the IMSP grant Comments: Leadership 10. Strength and competence of IMSP leadership 11. Sensitivity to cultural issues 12. Opportunities for me to take a leadership role 13. Trust that partners and participants afford each other 14. Transparency of decision-making. Comments: Communication 15. Use of the media to promote awareness of the IMSP goals, actions, and accomplishments 16. Communication among members of the partnership 17. Communication between the IMSP and the broader community 18. Extent to which IMSP participants are listened to and heard 19. Working relationships established with school officials 20. Information provided on issues and available resources Comments: Technical Assistance: 21. Strength and competence of IMSP faculty and staff 22. Training and technical assistance provided by faculty and staff 23. Help given the participants in meeting IMSP requirements 24. Help given the participants to become better able to address and resolve their concerns Comments: 3 Adapted from Annual Satisfaction Survey for Community Coalitions. Wolff,T. (2003). A practical approach to evaluating coalitions. In T.Backer(Ed.) Evaluating Community Collaborations. Springer Publishing IMSP Teacher Survey Adapted from Wolf Satisfaction Survey of Coalitions Page 29

30 Progress and Outcomes: 25. Progress in improving teachers content knowledge through the IMSP grant 26. Progress in teachers access and use of new instructional resources through the IMSP grant 27. Progress in teachers access and use of new STEM technologies through the IMSP grant 28. Teachers progress toward meeting endorsement or certification requirements 29. Effective collaboration between STEM industry experts and teachers through the IMSP grant 30. Teachers access to mentors through the IMSP grant 31. Fairness with which resources and opportunities are distributed 32. Capacity of IMSP teachers to give support to each other 33. IMSP grant's contribution to improving science and/or mathematics instruction in schools Comments: Please indicate how much you agree or disagree with the following statements. (Likert scale: Strongly Agree Strongly Disagree) Sustainability 34. My organization received important professional benefits from participation in the IMSP. 35. The benefits my organization received were worth the time, effort, and cost invested in the IMSP. 36. The benefits my organization received were commensurate with the contributions made to the IMSP. 37. I strongly believe the IMSP should be continued. 38. I will participate fully in IMSP activities in the future. 39. The IMSP activities need to be dramatically improved to make it worth my organization s investment. 40. The composition of the IMSP needs to be expanded or changed to be more effective. 41. My organization has changed the structure, policies, or functions to institutionalize the IMSP goals and activities. 42. My organization intends to sustain IMSP activities after the expiration of grant funds. 43. My organization is actively seeking alternative funds to sustain IMSP activities after the expiration of grant funds. Comments: IMSP Teacher Survey Adapted from Wolf Satisfaction Survey of Coalitions Page 30

31 Appendix B Synthesis of IMSP Teacher Action Research Developed by Elizabeth Oyer for the IMSP Evaluation Methodological Framework Overview The proposed methodology for synthesizing the teacher action research projects is grounded in qualitative meta-study and meta-ethnography. These methods attempt to integrate or compare findings from qualitative studies in order to construct a larger narrative or general theory about a subject (Sandelowski, Docherty, & Emden, 1997). The goal of the meta-synthesis is to maintain the uniqueness of the individual interpretations, reveal differences between studies, and identify ways in which the studies or findings are similar to each other to advance knowledge and produce a comprehensive view (Saiu & Long, 2005). The goal is not only to aggregate findings but to facilitate knowledge development (Thorne, Jensen, Kearney, Noblit, & Sandelowski, 2004). Noblit and Hare (1988) address the complex task of combining individual ethnographies to attain general conclusions. Their approach includes identifying an area of interest, selecting studies relevant to the area, conducting repeated reading of individual studies to identify metaphors, determining relationships between studies, translating studies into one another, and synthesizing the translations. This meta-ethnographic approach incorporates reciprocal translations as well as lines-of-argument syntheses that not only translate studies into one another but also frames similarities and differences in a new context. Paterson, Thorne, Canam, & Jillings (2001) build on these meta-study, metaethnography, and meta-theorizing practices by providing a summary of procedures and processes that incorporate theory, research methods, and data. Their process incorporates meta-data-analysis, meta-method, meta-theory, and meta-synthesis in a cyclical analytic study. As such, the process includes the aggregative and interpretive stage n the examination of multiple accounts of a phenomenon, the study of the rigor and sounds of the research methods, the underlying theory framing the individual studies, and bringing together the deconstructed ideas to form a new interpretation of the body of work. Meta-Synthesis Examples Meta-synthesis has been used to study questions in a variety of areas including e- government processes (Saiu & Long, 2005), patients medicine taking behavior and communication with health professionals (Britten et al., 2002), and research on diabetes care (Campbell et al., 2003) for example. Greenaway, Milne, Henwood, Asiasiga, & Witten (2004), funded by the Ministry of Research, Science and Technology, analyzed case studies with a meta-analytic grounded theory approach to present an IMSP Teacher Survey Adapted from Wolf Satisfaction Survey of Coalitions Page 31