Quality Assurance Implications of High- Impact Practices and Related Improvement Efforts George D. Kuh

Transcription

1 Quality Assurance Implications of High- Impact Practices and Related Improvement Efforts George D. Kuh NMHEAR Preconference Workshop February 28, 2013

2 Overview A word about our times, student success, and quality assurance NILOA who we are, what we do Degree Qualifications Profile Introduction to High-Impact Practices Activity Discussion

3 US Economy Defined by Greater Workplace Challenges and Dynamism More than 1/3 of the entire US labor force changes jobs ANNUALLY. Today's students will have jobs by age 38. Half of workers have been with their company less than 5 years. Every year, more than 30 million Americans are working in jobs that did not exist in the previous year. DOL-BLS

4 The World is Demanding More More college educated workers. Higher levels of learning and knowledge for all college graduates.

5 Employer expectations of employees have increased % who agree with each statement Our company is asking employees to take on more responsibilities and to use a broader set of skills than in the past Employees are expected to work harder to coordinate with other departments than in the past The challenges employees face within our company are more complex today than they were in the past To succeed in our company, employees need higher levels of learning and knowledge today than they did in the past 91% 90% 88% 88% 5 Raising The Bar October/November 2009 Hart Research for

6 Narrow Learning is Not Enough: The Essential Learning Outcomes Knowledge of Human Cultures and the Physical & Natural World Intellectual and Practical Skills Personal and Social Responsibility Deep Integrative Learning

7 Deep, Integrative Learning Attend to the underlying meaning of information as well as content Integrate and synthesize different ideas, sources of information Discern patterns in evidence or phenomena Apply knowledge in different situations View issues from multiple perspectives

8 Key Capabilities Open the Door for Career Success and Earnings Irrespective of college major or institutional selectivity, what matters to career success is students development of a broad set of cross-cutting capacities Anthony Carnevale, Georgetown U. Center on Education and the Workforce

9 The World is Demanding More More college educated workers. Higher levels of learning and knowledge for all college graduates. Assurance of quality

10

11 Student Success in College Academic achievement, engagement in educationally purposeful activities, satisfaction, acquisition of desired knowledge, skills and competencies, persistence, attainment of educational objectives, and postuniversity performance

12

13 What Are the Critical Quality Assurance Questions?

14 1. To what degree do students benefit from attending your university in terms of: Academic achievement, content knowledge, and cognitive/intellectual skills; Practical competencies and personal development; Degree attainment; Employment; and Post baccalaureate study?

15 2. What enriching educational opportunities are available and how many students take advantage of them? What distinctive opportunities are provided to students? 3. What evidence suggests students are able to integrate and apply what they learned?

16 4. What are the net costs for the typical resident and nonresident student to attend this institution? What factors influence the cost for students? 5. What internal institutional processes assure the quality of student learning? How are the results of these processes used to enhance student learning?

17 NILOA NILOA s mission is to discover and disseminate effective use of assessment data to strengthen undergraduate education and support institutions in their assessment efforts. SURVEYS WEB SCANS CASE STUDIES FOCUS GROUPS OCCASIONAL PAPERS WEBSITE RESOURCES NEWSLETTER LISTSERV PRESENTATIONS TRANSPARENCY FRAMEWORK FEATURED WEBSITES ACCREDITATION RESOURCES ASSESSMENT EVENT CALENDAR ASSESSMENT NEWS MEASURING QUALITY INVENTORY POLICY ANALYSIS ENVIRONMENTAL SCAN

18

19 NILOA resources include reports, tools, briefs, papers, and updates that support institutions in their outcomes assessment efforts

20 TRANSPARENCY

21 Transparency Reports

22

23 NILOA NILOA s mission is to discover and disseminate effective use of assessment data to strengthen undergraduate education and support institutions in their assessment efforts. SURVEYS WEB SCANS CASE STUDIES FOCUS GROUPS OCCASIONAL PAPERS WEBSITE RESOURCES NEWSLETTER LISTSERV PRESENTATIONS TRANSPARENCY FRAMEWORK FEATURED WEBSITES ACCREDITATION RESOURCES ASSESSMENT EVENT CALENDAR ASSESSMENT NEWS MEASURING QUALITY INVENTORY POLICY ANALYSIS ENVIRONMENTAL SCAN DEGREE QUALIFICATIONS PROFILE

24

25 Why a DQP or Why Emphasize Degree-Level Expectations/Outcomes Shift the focus from what is taught to what is learned by providing institutions with a template of widely agreed-upon competencies required for the award of degrees.

26 Degree Qualifications Profile A learning outcomes profile Competency demonstrated through performance At degree levels aligned with expectations for the respective degree For five key consensus learning areas

27 Degree Qualifications Profile Applied Learning Civic Learning Intellectual Skills Broad, Integrative Knowledge Associate Bachelor Master s Specialized Knowledge

28 Degree Qualifications Profile DQP builds on research about how people learn and is designed to ensure both knowledge and ability to integrate and apply DQP builds on LEAP and other efforts that have credibility among faculty, and includes outcomes already endorsed by accreditors and employers

29 LEAP Essential Learning Outcomes Knowledge of Human Cultures and the Physical and Natural World Focused on engagement with big questions, enduring and contemporary Intellectual and Practical Skills Practiced extensively across the curriculum, in the context of progressively more challenging problems, projects, and standards for performance Personal and Social Responsibility Anchored through active involvement with diverse communities and real-world challenges Deep, Integrative Learning Demonstrated through the application of knowledge, skills, and responsibilities to new settings and complex problems

30 Degree Qualifications Profile Knowledge both integrated and applied using Broad/Integrative knowledge demonstrating Specialized knowledge Intellectual Skills honed and integrated across the curriculum, applied to challenging problems Civic Learning demonstrated ability to integrate the work of responsible citizenship Applied Learning demonstrated through assignments involving research, projects, creativity

31 Degree Qualifications Profile A learning outcomes profile Competency demonstrated through performance At degree levels aligned with expectations for the degree For five key consensus learning areas For benchmarking and experimentation: a beta version for testing and further development

32

33 NILOA s role with the DQP NILOA is harvesting (collecting, analyzing, summarizing, synthesizing) what can be learned from all of the funded and unfunded work currently going on with the DQP

34 Regional Accreditors ACCJC (15) HLC (23) SACS (22) WASC (28) Organizations AASCU (6 in 3 state systems) AAC&U (21) CIC (25) States Oregon (24) Lumina-Funded Projects

35 NILOA and the DQP Document what is being done, by whom, and distill lessons learned in current DQP projects at the campus/system level tracking current and relevant future work; Share your DQP story? What are you doing? Who is involved? What are you learning? What assistance do you need? Tell us via the DQP Institutional Activity Report:

36 NILOA and the DQP Document what is being done, by whom, and any lessons learned in current DQP projects at the campus/system level--mapping current work; Identify synergies and cross-cutting issues that will enable Lumina to provide timely assistance, and help project leaders and campuses find allies dealing with similar issues; Provide support to campuses promoting promising practices, addressing challenges, sharing lessons and models (with special attention to student assessment); Gather information to help guide the upcoming revision of the DQP.

37 What We re Learning Across Projects DQP is a conversation starter: offers a common vocabulary for talking about outcomes Curricular mapping: Where are students mastering these competencies? Where are the gaps? Certify transfers, align and streamline systems Faculty engagement and ownership are essential, which take time Doing assessment right is a continuing, perennial challenge.

38 The DQP Vision of Assessment Integral to teaching and learning, not an add-on exo-skeleton Focuses on what students can do action verbs

39 Occasional Paper #16 The Degree Qualifications Profile: Implications for Assessment Peter T. Ewell & Carol Geary Schneider This paper offers guidance for how to gather evidence about the extent to which the competencies described in the DQP are mastered at the levels claimed. The challenges associated with assessing DQP proficiencies are outlined.

40 Faculty must drive assessment Identify which competencies described in the DQP they address in their courses/ labs/studios Identify which competencies are major objectives (probably only a few) Map those competencies to the existing relevant assignments Tweak existing or create new assignments/ student work to elicit the appropriate student behavior

41 Faculty must drive assessment Validate competence through assignments: lab specifications test questions performance protocols exhibit instructions field work questions paper topics with rubrics

42 Valid Assessment of Learning in Undergraduate Education (VALUE) Rubrics Inquiry and analysis Critical thinking Creative thinking Written communication Oral communication Reading Quantitative literacy Information literacy Teamwork Problem solving Civic knowledge and engagement Intercultural knowledge and competence Ethical reasoning and action Foundations and skills for lifelong learning Integrative learning

43 The DQP Vision of Assessment Integral to teaching and learning, not an add-on exo-skeleton Focuses on what students can do action verbs For all students, not just a sample Cumulative and connected

44 What enriching educational opportunities are available and how many students take advantage of them? What distinctive opportunities are provided to students?

45

46 High-Impact Activities First-Year Seminars and Experiences Common Intellectual Experiences Learning Communities Writing-Intensive Courses Collaborative Assignments and Projects Science as Science Is Done ; Undergraduate Research Diversity/Global Learning Service Learning, Community-Based Learning Internships Capstone Courses and Projects

47

48 Employers assess the potential value of high-impact educational practices 84% 81% 81% 73% % saying each would help a lot/fair amount to prepare college students for success Students complete a significant project before graduation that demonstrates their depth of knowledge in their major AND their acquisition of analytical, problem-solving, and communication skills (62% help a lot) Students complete an internship or community-based field project to connect classroom learning with realworld experiences (66%) Students develop research skills appropriate to their field and develop evidence-based analyses (57%) Students work through ethical issues and debates to form their own judgments (48%) 48 Raising The Bar October/November 2009 Hart Research for

49 Common Intellectual Experiences When students read and write or conduct inquiries about the same material, they are more likely to: talk about substantive matters outside of class study together see connections between different courses integrate and synthesize material

50 Learning Communities Formal program where groups of students (FY) take two or more classes together LC programs that integrate material across courses are associated with greater student engagement and learning

51 % 30 First -Year Student Number of Hours Studying by Learning Community Status LC 10 Non-LC More than 30 Hours

52 Service Learning Community-based project as part of a regular course Positively associated with deep learning and personal development

53 Research with a Staff Member Majority utilized existing info (libraries, WWW), and almost half worked in laboratory and fieldwork settings Reviewing literature and interpreting findings most closely related to deep learning Data collection had the weakest relationship

54 Study Abroad Positively related to engagement and learning outcomes upon return to home campus Living with host nationals related to more integrative and reflective learning, and personal and social gains Length of time spent abroad did not make a difference

55 Culminating Activities Capstone course, senior project or thesis, comprehensive exam, field placement A third (32%) of seniors reported having completed such an experience Another 29% said they planned to do so before graduating

56 Effects of Participating in High-Impact Activities on Deep/Integrative Learning and Gains Deep Learning Gains General First-Year Gains Personal Gains Practical Learning Communities Service Learning Senior Study Abroad Student-Faculty Research Internship Service Learning Culminating Experience p <.001, ++ p <.001 & Unstd B >.10, +++ p <.001 & Unstd B >.30

57 Effects of Participating in High-Impact Activities on Student Engagement Level of Academic Challenge Active and Collab. Learning First-Year Student- Faculty Interaction Supportive Campus Env. Learning Communities Service Learning Senior Study Abroad Student-Faculty Research Internship Service Learning Culminating Experience p <.001, ++ p <.001 & Unstd B >.10, +++ p <.001 & Unstd B >.30

58 HIP Benefits High-impact practices have been shown to be positively related to Persistence GPA Deep approaches to learning More frequent student-faculty interaction Enhanced critical thinking Improved writing skills Greater appreciation for diversity Higher overall student engagement Impact is often larger for underserved students Bronwell, J. & Swaner, L. (2010); NSSE (2007); Kuh (2008)

59 Impact of High-Impact Activities Enhanced When Students: Strive to reach expectations set at appropriately high levels Invest time and effort Interact with faculty and peers about substantive matters Experience diversity Get more frequent feedback Reflect & integrate learning Discover relevance of learning through real-world applications Demonstrate competence publicly

60

61 Keep in Mind More college experiences can be or are high impact in addition to those on the AAC&U list Implementation quality matters! Implications for advising

62 High-Impact Practices and the Disparities Within Frosh: Service Learning and LCs Parity among racial/ethnic groups Fewer 1 st gen students Fewer part-time students Fewer transfer students Fewer older students

63 High-Impact Practices and the Disparities Within Seniors in All HIPs Fewer 1 st gen students Fewer students of color Fewer transfer students Fewer part-time students Fewer older students

64 Faculty Perceptions How important is it to faculty that undergraduates do HIPs? ( very important or important ) Culminating experience (85%) Internship/Practicum (86%) Learning community (53%) Research with faculty (56%) Study abroad (44%)

65 Keep in Mind More college experiences can be or are high impact in addition to those on the AAC&U list Implementation quality matters! Implications for advising The characteristics of high-impact activities can be infused into any classroom, lab, studio or other learning setting

66 What If Every student had at least one high-impact experience in the first year and another later linked to the major

67 Activities How can you determine which HIPs are available at your school and which students do them? What are some other potential HIPs at your school? What procedures are need to insure that HIPs are of high quality? What assessment approaches will work best with which HIPs? How can you generate interest in HIPs at your school? What other issues do you want to discuss related to HIPs?

68 Assessing Student Engagement in High-Impact Practices To what extent does your institution provide these experiences? [ = have on campus; = required; estimate the % of various student populations in these activities] On Our Campus Required for all % Students involved % First Generation % Transfer Students % African American % Latino Students % Asian American % other % Adult Students Learning Community First Year Seminars Research w/ Faculty

69 What if We made student employment a high-impact activity.

70 U of Iowa Student Employment Project Guided Reflection on Work (GROW) Supervisors had two structured conversations with every student employee during spring semester and coded conversations into the following categories: How the job and academics complement each other ( How is your job fitting in with your academics? ) Transfer between work and academics ( What are you learning here at work that is helping you in school? ) Transfer between academics and work ( Are you learning anything in class that you can apply here at work? ) Transfer between work and future career ( Give me a couple of examples of things that you are learning here at work that you will be using in your future profession? )

71 U of Iowa Student Employment Project Guided Reflection on Work (GROW) Student Employment Survey used to examine differences between pilot and non-pilot participants.

72 Student Employment Outcomes: Guided Reflection on Work (GROW) Outcome % agree/strongly agree Mean My supervisor helps me make connections between my work and my life as a student. My job has helped prepare me for the world of full-time work. My job has helped me improve my written communications. Pilot Participants Non-Pilot Pilot Non- Pilot 60% 51% % 51% % 21%

73 Student Employment Outcomes: Guided Reflection on Work (GROW) Outcome % agree/strongly agree Mean Pilot Participants Non- Pilot Pilot Non- Pilot I can see connections between my job and my major/coursework. 56% 36% My job has helped me learn about career options. 30% 39%

74 Student Employment Outcomes: Guided Reflection on Work (GROW) Outcome % agree/strongly agree Mean Because of my job, I am able to work effectively with individuals with a variety of backgrounds, experiences, and cultures. My job has helped me use critical thinking skills Pilot Participants Non- Pilot Pilot Non- Pilot 82% 77% % 57%

75 Student Employment Outcomes: Guided Reflection on Work (GROW) Outcome % agree/strongly agree Mean My job helped me develop more effective time management skills. My job helped me improve my oral communication skills. Pilot Participants Non-Pilot Pilot Non- Pilot 76% 76% % 72% My job helped me develop conflict resolution skills. 74% 61%

76 Activities How can you determine which HIPs are available at your school and which students do them? What are some other potential HIPs at your school? What procedures are need to insure that HIPs are of high quality? What assessment approaches will work best with which HIPs? How can you generate interest in HIPs at your school? What other issues do you want to discuss related to HIPs?

77 Final Thoughts Make sure HIPs are done well Seven features that make HIPs effective The state of HIP participation on campus Equal opportunities for all Connections to positive outcomes Generating interest Faculty are key!

78 Questions & Discussion

79