Pretest Results April 11, 2011

Transcription

1 Self-Assessment Rubric for the Institutionalization of Service-Learning in University STEM Departments An Assessment Tool of the (STEM) 2 Initiative Pretest Results April 11, 2011 Submitted by : Rebecca M. Eddy, Ph.D. Todd Ruitman, M.A. & Nancy Hankel COBBLESTONE APPLIED RESEARCH & EVALUATION, INC Foothill Blvd., Ste. 202 La Verne, CA (800)

2 The California State University (CSU) was awarded a grant from Learn and Serve America (LSA) in July The goal of the grant is to increase student success and graduation rates in Science, Technology, Engineering and Mathematics (STEM) through service-learning initiatives. In an effort to show progress toward increasing the institutionalization of service learning as a result of grant activities, each campus completed a Self-Assessment Rubric for the Institutionalization of Service-Learning in University STEM Departments to act as a pretest measurement. Each of the 23 CSU campuses will complete the rubric again at the end of the three-year grant to determine growth as a result of grant activities. Description of the Self-Assessment Rubric The rubric was adapted from the work of Andrew Furco, University of California, Berkeley, 1999 (revised in 2002, 2003, 2006) and based on the Kecskes/Muyllaert Continuums of Service Benchmark Worksheet. Each campus self-rated twenty-two aspects divided into five dimensions of their servicelearning program specifically related to STEM departments (see Table 1). Dimension 1 measures the development of a definition, philosophy, and mission of service learning on the campus. Dimension 2 measures STEM faculty support of, and involvement in, service learning. Dimension 3 measures STEM student support for, and involvement in, service learning. Dimension 4 measures community participation and partnerships, and finally, Dimension 5 measures departmental support for service learning. Table 1. Components of the Five Dimensions Dimensions Dimension 1: Philosophy and Mission of Service-Learning Dimension 2: STEM Faculty Support for and Involvement in Service-Learning Dimension 3: STEM Student Support for and Involvement in Service-Learning Dimension 4: Community Participation and Partnerships Dimension 5: Departmental Support for Service-Learning Dimension Components 1.1 Definition of Service-Learning in STEM Departments 1.2 Strategic Planning 1.3 Alignment with Departmental Mission 1.4 Alignment with Educational Reform Efforts 2.1 STEM Faculty Knowledge and Awareness 2.2 STEM Faculty Involvement and Support 2.3 STEM Faculty Leadership 2.4 STEM Faculty Incentives and Rewards 3.1 STEM Student Awareness 3.2 STEM Student Opportunities 3.3 STEM Student Leadership 3.4 STEM Student Incentives and Rewards 4.1 Community Partner Awareness 4.2 Mutual Understanding 4.3 Community Partner Voice and Leadership 5.1 Coordinating Entity 5.2 Policy-Making Entity 5.3 Staffing 5.4 Funding 5.5 Administrative Support 5.6 Engaged STEM Department Status 5.7 Evaluation and Assessment Cobblestone Applied Research & Evaluation, Inc. Page 2

3 Rating For each dimension component, respondents rated their current status on a scale of 1 to 9, with 1 representing the lowest possible rating and 9 representing the highest possible rating. The numerical ratings were accompanied by a statement to be used by the respondent as a guide for their rating. For example, one of the components contained the following statement: Few service-learning opportunities exist for STEM students; only a handful of service-learning courses are available. If the respondents felt that this statement best reflected their STEM departments then they would select one of three numerical responses that corresponded to this statement. A rating of 1 to 3 represented that a campus was in the stage of critical mass building meaning that the STEM departments were in the beginning stages of service-learning institutionalization; a rating of 4 to 6 represented the area of quality building which suggested that the STEM departments were aware of their service-learning needs and were working toward service-learning institutionalization; and a rating of 7 to 9 represented the area of sustained institutionalization meaning that service-learning was fully integrated into the STEM departments. Following each of the 22 ratings, a space was provided for explanation should the respondent feel an explanation of the rating was necessary. Figure 1 displays the three stages of the rubric. Figure 1. Three Stages of the Self-Assessment Rubric Critical Mass Quality Sustained Building Building Institutionalization Method As a pretest measurement, each of the 23 CSU campuses was provided with a self-assessment rubric to measure their current institutionalization of STEM service learning on their campus. The rubric was ed during the last quarter of 2010 and was to be completed by January 31. Most campuses were able to return their completed rubrics by this date; however, not all rubrics were completed until mid-march. Each rubric contained detailed instructions on how to complete the assessment in addition to a brief introduction. The rubric was primarily completed by each campus Director of Service Learning. Cobblestone Applied Research & Evaluation, Inc. Page 3

4 Average rating out of 9 Many directors included other qualified individuals to assist in the completion of the rubric, such as STEM faculty or service-learning staff. Overall Results Results Figure 2 below shows the average scores among all campuses for each dimension. The highestrated dimension, Dimension 1 (Philosophy and Mission of Service-Learning), had an average score of 4.04 out of 9 while the lowest-rated dimension, Dimension 2 (STEM Faculty Support for and Involvement in Service-Learning), had an average score of 3.07 out of 9. Overall the scores show that most campuses are, on average, between the critical mass building and quality building stages in each of the five dimensions. Specifically, Dimension 1 is the only dimension where the average rating was in the quality building stage. The other four dimensions all scored above a three but below a four which suggests that the individual components of each dimension were generally rated in the critical mass building stage with the exception of some individual components rated in the quality building stage. With such variance in rating components within dimensions, it was important to analyze the average results of the individual components. Figure 2. Average Campus Ratings across Dimensions Dimension While most components within each dimension were fairly consistent, Dimension 3 showed a surprising result whereas it contained both the highest rated component in addition to one of the lowest rated components (see Figure 3). This finding illustrates the need to pay particular attention to individual components in addition to the overall dimensions. The results showed that the highest rated components, irrespective of dimension, were: 3.4 STEM Student Incentives and Rewards (m = 5.0), 1.2 Strategic Planning (m = 4.3), 1.4 Alignment with Educational Reform Efforts (m = 4.3), 1.3 Alignment with Departmental Mission (m = 4.2), and 3.1 STEM Student Awareness (m = 4.0). These five components all were rated on average in the quality building stage of the rubric. All other components had an average Cobblestone Applied Research & Evaluation, Inc. Page 4

5 Campus 1 Campus 2 Campus 3 Campus 4 Campus 5 Campus 6 Campus 7 Campus 8 Campus 9 Campus 10 Campus 11 Campus 12 Campus 13 Campus 14 Campus 15 Campus 16 Campus 17 Campus 18 Campus 19 Campus 20 Campus 21 Campus 22 Campus 23 Average rating out of 9 rating below a four on the rubric with the lowest components being 5.3 Staffing (m = 2.7), 3.3 STEM Student Leadership (m = 2.8), 2.1 STEM Faculty Knowledge and Awareness (m = 2.9), 2.4 STEM Faculty Incentives and Rewards (m = 2.9), and 5.4 Funding (m = 2.9). Figure 3. Average Score by Component Individual Campus Results Each campus also received a total score for their self-assessment. The average for each dimension was added together to obtain the total score. We used the average of each dimension thereby making each dimension of equal weight in the total score. Out of a total 45 points possible, campuses averaged a total score of 17.6, with a range of 6.3 to 37.2 points. Figure 4 shows the distribution of scores across all campuses. Figure 4. Total Score of Self-Assessment Rubric for all Campuses Critical Mass Campus 1 Building Campus Quality Building 19 Campus Sustained 23 Institutionalization Cobblestone Applied Research & Evaluation, Inc. Page 5

6 Figure 4 above also shows the stage (i.e., critical mass building, quality building, sustained institutionalization) of each campus after averaging out all ratings on the self-assessment rubric. The total score for the self-assessment rubric shows that eleven campuses are in the critical mass building stage, eight are in the quality building stage, and four are in the sustained institutionalization stage overall. While this information is helpful as an overall view of each campus, this information fails to show the strengths and weaknesses of each individual campus. For example, after averaging each of the five dimensions, the overall rating of the campus was in the sustained institutionalization stage on the rubric; however, a closer look at the individual campus results show a few components in Dimension 2 are rated in the critical mass building stage. While individual campus results are not available in this report, each of the 23 CSU campuses has either full dimensions or individual components that can be improved. Discussion The self-assessment rubric revealed that most schools are either in the critical mass building stage or quality building stage. Given the grant activities planned over the next three years, we expect that participating campuses will see gains in most areas of the self-assessment rubric. Campuses are expected to participate in varying levels of program activities. For example, some campuses were awarded funds for Innovation subgrants, while others will have faculty and administration participate in Engaged Department Institutes. Given the varying levels of program participation, we can investigate how these activities will impact institutionalization of service learning on all CSU campuses. While the self-assessment rubric is being used ultimately as a pretest-posttest measure, it has provided important and useful information about the state of the CSU system as grant activities are just beginning. A copy of the self-assessment rubric completed by campuses can be found in Appendix A. Cobblestone Applied Research & Evaluation, Inc. Page 6

7 Appendix A. Self-Assessment Rubric for the Institutionalization of Service-Learning in University STEM Departments DIMENSION I: PHILOSOPHY AND MISSION OF SERVICE-LEARNING A primary component of service-learning institutionalization in your Science, Technology, Engineering, and Mathematics (STEM) departments on your campus is the development of a definition for service-learning that provides meaning, focus, and emphasis for the service-learning effort in your STEM departments. How narrowly or broadly service-learning is defined on your campus will effect which campus constituents participate/do not participate, which campus units will provide financial resources and other support, and the degree to which service-learning will become part of the campus institutional fabric. DIRECTIONS: For each of the four categories (rows), highlight or circle the number in the continuum that best represents the CURRENT status of the development of a definition, philosophy, and mission of service-learning in your STEM departments. Critical Mass Building Quality Building Sustained Institutionalization Explanation DEFINITION OF SERVICE-LEARNING IN STEM DEPARTMENTS There is no definition for servicelearning across STEM departments. The term "service-learning" is used inconsistently to describe a variety of service and outreach activities. There is an operationalized definition for service-learning in the STEM departments, but there is some variance and inconsistency in the application of the term. The STEM departments have a formal, universally accepted definition for high quality servicelearning that is used consistently to operationalize many or most aspects of service-learning. STRATEGIC PLANNING The campus does not have an official strategic plan for advancing service-learning.. Although certain short-range and long-range goals for servicelearning have been defined for the campus, these goals have not been formalized into an official strategic plan for STEM departments that will guide the implementation of these goals. The campus has developed an official strategic plan for advancing service-learning, which includes viable short-range and long-range institutionalization goals for STEM departments.

8 ALIGNMENT WITH DEPARTMENTAL MISSION While service-learning complements many aspects of the campus s mission, it remains on the periphery. Service-learning is rarely included in larger efforts that focus on the core missions of the campus, particularly within STEM departments. Service-learning is often mentioned as a primary or important part of the campus s mission, but servicelearning is not included in the campus; official mission or strategic plans, particularly within STEM departments. Service-learning is part of the primary concern of the campus. Service-learning is included in the campus s official missions and/or strategic plans, particularly in STEM departments. ALIGNMENT WITH EDUCATIONAL REFORM EFFORTS Service-learning stands alone and is not tied to other important, high profile efforts on campus (e.g., STEM departments/ community partnership efforts, establishment of learning communities, improvement of undergraduate teaching, faculty research). Service-learning is tied loosely or informally to other important, high profile efforts on campus (e.g., STEM departments/community partnership efforts, establishment of learning communities, improvement of undergraduate teaching, faculty research). Service-learning is tied formally and purposefully to other important, high profile efforts on campus (e.g., STEM departments/community partnership efforts, establishment of learning communities, improvement of undergraduate teaching, faculty research). Please provide any overall comments about your ratings from above. Cobblestone Applied Research & Evaluation, Inc. Page 8

9 DIMENSION II: STEM FACULTY SUPPORT FOR AND INVOLVEMENT IN SERVICE-LEARNING One of the essential factors for institutionalizing service-learning in higher education is the degree to which faculty members are involved in implementation and advancement of service-learning on a campus (Bell, Furco, Ammon, Sorgen, & Muller, 2000). DIRECTIONS: For each of the four categories (rows), highlight or circle the number in the continuum that best represents the CURRENT status of STEM faculty involvement in and support for service-learning in STEM departments. Critical Mass Building Quality Building Sustained Institutionalization Explanation STEM FACULTY KNOWLEDGE AND AWARENESS Very few STEM faculty fully know what service-learning is or understand how service-learning is different from community service, internships, or other experiential learning activities. An adequate number of STEM faculty members know what service-learning is and understand how service-learning is different from community service, internships, and other experiential learning activities. A substantial number of STEM faculty members know what servicelearning is and can articulate how service-learning is different from community service, internships, or other experiential learning activities. STEM FACULTY INVOLVEMENT AND SUPPORT Very few STEM faculty members are instructors, supporters, or advocates of service-learning. Few support the strong infusion of service-learning into the academic programs or into their own scholarly and professional work. Service-learning activities in STEM are sustained by a few faculty members on campus. While a satisfactory number of STEM faculty members are supportive of service-learning, few of them are advocates for infusing service-learning into the academic programs or into their own scholarly and professional work. An inadequate or unsatisfactory number of KEY STEM faculty members are involved in servicelearning. A substantial number of influential STEM faculty members participate as instructors, supporters, and advocates of service-learning and support the infusion of servicelearning into the academic programs or into their own scholarly and professional work. STEM FACULTY LEADERSHIP None of the most influential STEM faculty members serve as leaders for advancing service-learning in the STEM departments. There are only one or two influential STEM faculty members who provide leadership to the STEM departments service- A highly respected, influential group of STEM faculty members serves as the STEM departments servicelearning leaders and/or advocates. Cobblestone Applied Research & Evaluation, Inc. Page 9

10 learning effort. STEM FACULTY INCENTIVES AND REWARDS In general, STEM faculty members are not encouraged to be involved in service-learning activities; few if any incentives are provided (e.g., minigrants, sabbaticals, funds for conferences) to pursue servicelearning activities; STEM faculty members' work in service-learning is not usually recognized during their review, tenure, and promotion process. Although STEM faculty members are encouraged and are provided various incentives (e.g., minigrants, sabbaticals, funds for servicelearning conferences) to pursue service-learning activities, their work in service-learning is not always recognized during their review, tenure, and promotion process. STEM faculty who are involved in service-learning receive recognition for it during the department's review, tenure, and promotion process; STEM faculty are encouraged and are provided various incentives (e.g., minigrants, sabbaticals, funds for servicelearning conferences) to pursue service-learning activities. Please provide any overall comments about your ratings from above. Cobblestone Applied Research & Evaluation, Inc. Page 10

11 DIMENSION III: STEM STUDENT SUPPORT FOR AND INVOLVEMENT IN SERVICE-LEARNING An important element of service-learning institutionalization is the degree to which students are aware of service-learning opportunities in the STEM departments and are provided opportunities to play a leadership role in the development of service-learning in the STEM departments. DIRECTIONS: For each of the four categories (rows), highlight or circle the number in the continuum that best represents the CURRENT status of student support for and involvement in service-learning in your campus STEM departments. Critical Mass Building Quality Building Sustained Institutionalization Explanation STEM STUDENT AWARENESS For STEM students, there is no mechanism for informing students about service-learning courses, resources, and opportunities that are available to them. While there are some mechanisms for informing STEM students about service-learning courses, resources, and opportunities that are available to them, the mechanisms are sporadic and concentrated in only a few programs (e.g., course flyers). There are coordinated mechanisms (e.g., service-learning listings in the schedule of classes, course catalogs) that help STEM students become aware of the various service-learning courses, resources, and opportunities that are available to them. STEM STUDENT OPPORTUNITIES Few service-learning opportunities exist for STEM students; only a handful of service-learning courses are available. Service-learning options (in which service is integrated in core academic courses) are limited to only a certain groups of students in the STEM departments (e.g., students in certain majors, honors students, seniors). Service-learning options and opportunities (in which service in integrated in core academic courses) are available to STEM students in many areas throughout the STEM departments,. STEM STUDENT LEADERSHIP Few, if any, opportunities in the STEM departments exist for students to take on leadership roles in advancing service-learning in their department. There are a limited number of opportunities available for STEM students to take on leadership roles in advancing service-learning in their department. STEM students are welcomed and encouraged to serve as advocates and ambassadors for institutionalizing service-learning in their department. Cobblestone Applied Research & Evaluation, Inc. Page 11

12 STEM STUDENT INCENTIVES AND REWARDS The campus has neither formal mechanisms (e.g., catalogued list of service-learning courses, special notation on students transcripts) nor informal mechanisms (e.g., news stories in paper, unofficial student certificates of achievement) that encourage or reward STEM students to participate in servicelearning activities. While the campus offers some informal incentives and rewards (e.g., news stories in paper, unofficial student certificates of achievement) that encourage and/or reward STEM students to participate in service-learning activities, the campus offers few or no formal incentives and rewards (e.g., catalogued list of servicelearning courses, special notation on students transcripts). The campus has one or more formal mechanisms in place (e.g., catalogued list of service-learning courses, special notation on students transcripts) that encourage and/or reward STEM students to participate in servicelearning activities. Please provide any overall comments about your ratings from above. Cobblestone Applied Research & Evaluation, Inc. Page 12

13 DIMENSION IV: COMMUNITY PARTICIPATION AND PARTNERSHIPS An important element for service-learning institutionalization is the degree to which the STEM departments nurture community partnerships and encourage community agency representatives to play a role in implementing and advancing service-learning in the STEM departments. DIRECTIONS: For each of the three categories (rows), highlight or circle the number in the continuum that best represents the CURRENT status of community participation and partnerships in your STEM departments. Critical Mass Building Quality Building Sustained Institutionalization Explanation COMMUNITY PARTNER AWARENESS Few, if any, community agencies that partner with the campus are aware of the campus s goals for service-learning and the full range of service-learning opportunities that are available to STEM students. Some, but not the majority of community agencies that partner with the campus are aware of the campus s goals for service-learning and the full range of servicelearning opportunities that are available to STEM students. Most community agencies that partner with the campus are aware of the campus s goals for servicelearning and the full range of service-learning opportunities that are available to STEM students. MUTUAL UNDERSTANDING There is little or no understanding between the campus and community representatives regarding each other's needs, timelines, goals, resources, and capacity for developing and implementing service-learning activities in STEM. There is some understanding between the campus and community representatives regarding each other's needs, timelines, goals, resources, and capacity for developing and implementing service-learning activities, but there are some disparities between community and STEM departments goals for service-learning. Both the campus and community representatives are aware of and sensitive to each other's needs, timelines, goals, resources, and capacity for developing and implementing service-learning activities. There is generally broad agreement between the STEM departments and community on the goals for service-learning. COMMUNITY PARTNER VOICE & Few, if any, opportunities exist for community agency There are a limited number of opportunities available for Appropriate community agency representatives are formally Cobblestone Applied Research & Evaluation, Inc. Page 13

14 LEADERSHIP representatives to take on leadership roles in advancing service-learning in STEM departments; community agency representatives are not consistently invited or encouraged to express their particular agency needs or recruit student and faculty participation in servicelearning. community agency representatives to take on leadership roles in advancing service-learning in STEM departments; community agency representatives are provided limited opportunities to express their particular agency needs or recruit student and faculty participation in service-learning. welcomed and consistently encouraged to serve as advocates and ambassadors for institutionalizing service-learning in STEM departments; community agency representatives are provided substantial opportunities to express their particular agency needs or recruit student and faculty participation in servicelearning. Please provide any overall comments about your ratings from above. Cobblestone Applied Research & Evaluation, Inc. Page 14

15 DIMENSION V: DEPARTMENTAL SUPPORT FOR SERVICE-LEARNING In order for service-learning to become institutionalized, the STEM departments must provide substantial resources, support, and muscle toward the effort. DIRECTIONS: For each of the seven categories (rows), highlight or circle the number in the continuum that best represents the CURRENT status of STEM departmental support for STEM service-learning. Critical Mass Building Quality Building Sustained Institutionalization Explanation COORDINATING ENTITY There are no STEM faculty and/or STEM department leaders that are devoted to assisting the various campus constituencies in the implementation, advancement, and institutionalization of servicelearning. There are STEM faculty and/or STEM department leaders who utilize service-learning activities, but the entity either does not coordinate them exclusively or provides services only to a certain constituency (e.g., students, faculty) or limited number of STEM departments or programs. There are STEM faculty and/or STEM department leaders who maintain a coordinating entity (e.g., committee, center or clearinghouse) that are devoted primarily to assisting the various constituencies in the implementation, advancement, and institutionalization of servicelearning. POLICY-MAKING ENTITY The campus s official and influential policy-making board(s)/committee(s) do not recognize service-learning as an essential educational goal for the STEM departments. The campus s official and influential policy-making board(s)/committee(s) recognize service-learning as an essential educational goal for the STEM departments, but no formal policies have been developed. The campus s policy-making board(s)/committee(s) recognize service-learning as an essential educational goal for the STEM departments and formal policies have been developed or implemented. STAFFING There are no staff and/or faculty members on campus whose primary paid responsibility is to advance and institutionalize service-learning in STEM. There is an appropriate number of staff and/or faculty members on campus who understand servicelearning fully and/or who hold appropriate titles that can support The campus houses and funds an appropriate number of permanent staff and/or faculty members who understand service-learning and who hold appropriate titles that Cobblestone Applied Research & Evaluation, Inc. Page 15

16 the advancement and institutionalization of servicelearning throughout the STEM departments; however their appointments are temporary or paid from soft money or external grant funds. can support the advancement and institutionalization of servicelearning in the STEM departments. FUNDING The campus s service-learning activities in STEM are supported primarily by soft money (e.g., short-term grants) from sources outside the institution. The campus s service-learning activities in STEM are supported by both soft money (e.g., short-term grants) from sources outside the campus as well as hard money from within the campus.. The campus s service-learning activities in STEM are supported primarily by hard funding from the campus or directly from STEM departments. ADMINISTRATIVE SUPPORT The campus s administrative leaders have little or no understanding of service-learning in STEM, often confusing it with other campus outreach efforts, such as community service or internship programs. The campus s administrative leaders have a clear understanding of service-learning in STEM, but they do little to make servicelearning a visible and important part of the campus s work. The campus s administrative leaders understand and support service-learning in STEM, and actively cooperate to make servicelearning a visible and important part of the campus s work. ENGAGED STEM DEPARTMENT STATUS The STEM departments are not considered by the broader university community to be engaged departments. The STEM departments are considered to be emerging engaged departments by the broader university community. The STEM departments are considered to be advanced engaged departments by the broader university community. EVALUATION & There is no organized effort An initiative to account for the An ongoing, systematic effort is in Cobblestone Applied Research & Evaluation, Inc. Page 16

17 ASSESSMENT underway for STEM department to account for the number, quality, and impact of servicelearning activities taking place. number, quality, and impact of service-learning activities taking place throughout in the STEM departments has been proposed. place to account for the number, quality, and impact of servicelearning activities that are taking place throughout the STEM departments. Please provide any overall comments about your ratings from above. Cobblestone Applied Research & Evaluation, Inc. Page 17