Teagle Foundation Value-Added Assessment Projects

Transcription

1 Teagle Foundation Value-Added Assessment Projects The Benefits and Challenges of Inter-Institutional Collaboration Lee Cuba, Professor of Sociology, Wellesley College Joseph Swingle, Assistant Professor of Sociology, Wellesley College Steven Weisler, Dean of Academic Development, Hampshire College Hamilton Assessment Conference April 13-14, 2007

2 Teagle s Emphasis on Assessment The Teagle Foundation is giving high priority to systematic assessment of the outcomes achieved in courses, institutions, and whole sectors of higher education. While colleges already expend great effort in assessing what they do, through such means as teaching evaluations and program reviews, the systematic measurement of the value added in various settings of higher education is an area in which more can be done. Nothing, we believe, has more potential to affect students' educational experience as much as sustained and appropriate assessment of what they learn.

3 Teagle Grants for Faculty Driven Value Added Assessment Consortia 1. Augustana College, Alma College, Gustavus Adolphus College, Illinois Wesleyan University, Luther College, and Wittenberg University a research project that seeks to detail the relationship of students' intellectual growth measured in terms of writing and criticalthinking and students' growth in civic engagement. 2. Carleton College, Grinnell College, Macalester College, and St. Olaf College investigate four learning outcomes that have been identified as central to the institutional values and missions of each of the colleges: effective writing (Carleton), analytical reasoning (Macalester), critical thinking (St. Olaf), and global understanding (Grinnell).

4 Teagle Grants for Faculty Driven Value Added Assessment Consortia 3. Furman University, Austin College, Juniata College, and Washington and Lee University conduct a project that seeks to develop assessments of student outcomes that will provide a rich source of insight into the mechanics of student instruction and motivation at the level of the small liberal arts college. 4. Kalamazoo College, Colorado College, and Earlham College use quantitative and qualitative means to gather data and stories that will (1) provide the evidence for its assertions about student outcomes and (2) help to improve and sustain an interrogative approach to shaping curricula through which students become educated.

5 Teagle Grants for Faculty Driven Value Added Assessment Consortia 5. Bates College,Bowdoin College, Colby College, Middlebury College, Smith College, Trinity College, Wellesley College, and the New England Association of Schools and Colleges undertake a collaborative project that investigates students' learning at points when they make choices that influence the scope and quality of their education, as well as prepare them for what they do when they graduate. 6. Hampshire College, Allegheny College, Bard College, Center of Inquiry in the Liberal Arts at Wabash College, Hamilton College, Hobart and William Smith Colleges, Hope College, and Vassar College study the assessment of student writing and other foundational skills, create a "common data set" for the collaborative, and convene consultancies and meetings designed to integrate assessment efforts and practices at all levels of each institution.

6 Common Properties of Teagle Assessment Initiatives 1. Faculty driven 2. Value added 3. Improvement oriented 4. Consortially based

7 The Wellesley Consortium General goals for the project Explore student learning in relation to institutional practices Better understand how students make important decisions to inform teaching and build cultures of inquiry Involve faculty more directly in assessment Share institutional research findings

8 The Wellesley Consortium Major questions addressed by the project How do students make the transition from high school to college? How do students learn? How do students make important decisions about their academic program (e.g., major program, study abroad, capstone experiences?) How do students engage the curriculum? How do our institutional practices and policies affect student learning?

9 Longitudinal Panel Study of the Class of students at each of six colleges (for a total sample of 216 first year students) Race-stratified random sample (African American, Asian American, Latino/a, International, and domestic white students) Equal numbers of women and men at coed schools Three interviews in first year, two in subsequent years Student interviewers at five colleges, faculty interviewers at one

10 Strengths of the Panel Study Partnerships among senior administrators, IR staffs, faculty and students in assessment Direct involvement of faculty and students in assessment of student learning Qualitative, longitudinal design Attention to experiences of students of color and international students

11 The Benefits of Collaboration Different institutional contexts, practices and policies ( best practices ) Shared faculty and staff expertise Comparative institutional data analysis Larger sample size Regular, ongoing conversation with peers Corporate identity

12 Interviewer Training Workshop 2 days in late August 2006 Participants = students and faculty from participating schools + NEASC staff Sessions included: Interviewing techniques Training with digital voice recorders Videotaped mock interviews (students interviewing students) Discussion of research ethics and how to handle sensitive and/or urgent information

13 Student Interviewee - Incentives No money up front; unannounced gift certificate to be provided at end of first year Made to feel part of an important research project Opportunity to express their opinions and experiences Response rate at Wellesley College: 35 out of the initial 36 students agreed to participate

14 Student Interviewer - Incentives Majors: Psychology, Sociology, Anthropology + others Interviewers (IRs) paid per interview IRs establish closer working relationships with supervising faculty member IRs report academically rewarding doing actual social science research IRs report personally enriching inspired to reflect on their own experiences

15 Faculty Supervisor - Incentives Disciplines: Psychology, Sociology, Anthropology, Economics Annual stipend Workload light and episodic three rounds of interview per year Academically rewarding giving students hands-on research experience; resonates with own research interests Service to college community

16 The Hampshire Consortium Three projects: 1. Build a cross-institution database to support assessment of student learning 2. Administer an adapted version of the Hamilton writing assessment across institutions 3. Exploit complementary institutional strengths to encourage consortial members to bring missions, departmental practices, programs, and assessment into alignment

17 The Common Data Project: 1.0 Collect common survey data from all participating institutions (CIRP, NSSE, CSEQ/XQ, etc.) Merge with institutional data Develop a cross-institutional comparative database

18 The Result Institutions administer instruments on different schedules Institutions hold common data in incommensurable formats Policies for merging institutional data with survey data vary Across survey organizations Across campuses

19 The Common Data Project: 2.0 The Wabash National Study of Liberal Arts Experiences Students - 19 institutions (all types) - Longitudinal analysis - Tightly controlled research protocols and data handling

20 The College Impact Model College access - Opportunity to attend - Choice of college - Financial aid Student background (pre -college) characteristics - Demographics - Socioeconomics - Academic characteristics - High school involvement - Interests - Aspirations & motivations Structural/Organizational characteristics of the institution - Enrollment - Faculty -student resources - Academic profile - Assets - Etc. The College Experience - Institutional environment Activities - Academics - Extracurricular - Employment Interactions - Faculty / Student - Peer groups - Employer / Coworker Quality of The College Experience -Quality of students efforts -Quality of students Learning & experiences -Academic & social interactions educational outcomes (I) - Cognitive (II) - Psycho -social - Educational aspirations - Educational attainment - Career orientations Note. Model adapted from Astin (1993), Pascarella (1985), Tinto (1975), and Weidman (1989)

21 Wabash Learning Outcomes Effective reasoning and problem solving Inclination to inquire and lifelong learning Integration of learning Intercultural effectiveness Leadership Moral reasoning Well-being

22 Conclusions Collecting and organizing data in a consortium presents special challenges in terms of data control, institutional policies, and commensurability Consortia offer the potential for powerful comparative analyses that can help to identify outstanding characteristics of an institutional profile