General Education and Core Competencies. Paul D. Camp Community College Dir. Assessment & IR General Education Committee 2/11/11

Transcription

1 General Education and Core Competencies Paul D. Camp Community College Dir. Assessment & IR General Education Committee 2/11/11

2 Table of Contents Introduction. 1 VCCS General Education 3 VCCS Core Competencies. 5 Communication: Writing Core Competency 6 Communication: Oral Core Competency. 14 Critical Thinking 19 Cultural and Social Understanding. 25 Information Literacy. 33 Personal Development 40 Quantitative Reasoning. 47 Scientific Reasoning.. 55 ii

3 Introduction SACS requirement for general education is the following [Principles of Accreditation: Foundations for Quality Enhancement, p. 49]: The institution identifies college-level competencies within the general education core and provides evidence that graduates have attained those competencies. (College-level Competencies) Virginia Community College System (VCCS) General Education The Virginia Community College System (VCCS) setup a VCCS Task Force on General Education in 2004 with examining the current status of general education in relationship to SCHEV core competencies requirements and the new SACS guidelines for general education. For the VCCS, General Education pertains to all two-year graduates. The Taskforce s discussions centered on the skills, knowledge, and value systems students need to be generally educated citizens. The concept of a generally educated citizen is similar to Newton s Effective Citizen Model of general education and reinforces the VCCS definition of general education which states that general education is that portion of the collegiate experience which addresses the knowledge, skills, attitudes and values characteristic of educated persons. While the VCCS General Education requirements list eight areas of general education, SCHEV outlines only six core competency areas. In addition, while the VCCS general education areas use key descriptive terms to inform and guide faculty and academic administrators, the VCCS response to the SCHEV core competencies outline student learning outcome objectives. Virginia s core competencies have also been held up as a model for other states by AAC&U s Principles of Good Practices. In order to identify areas of overlap, the Taskforce members mapped back the objectives from all six core competency areas fit within the existing general education requirements. The core competency objectives also had the advantage of being faculty driven. Each set of objectives was produced or revised by area advisory groups consisting of faculty members from across the VCCS. Because no advisory group was in place to review draft Cultural and Social Understanding Objectives and the Personal Development Objectives, these draft objectives were sent out to faculty members at various colleges for input. The result was the development of objectives in seven areas: Communication, Critical Thinking, Cultural and Social Understanding, Information Literacy, Personal Development, Quantitative Reasoning and Scientific Reasoning. The General Education Taskforce examined several general education delivery models including core programs, distribution programs, and competency-based models. The Taskforce proposes the use of a competency-based model for general education instruction delivery. This model provides the individual VCCS colleges with the maximum amount of flexibility in delivering instruction while providing a mechanism for ensuring that students across the VCS are meeting the objectives set forth in the General Education requirements. The model also allows the VCCS colleges to meet their core competency reporting requirements without duplicating assessment efforts. Finally, the model will provide individual colleges with student learning outcomes data in the area of general education for the purpose of meeting the new general education guidelines set forth by SACS. The competency-based model allows individual colleges the freedom to align VCCS general education requirements, SCHEV core competencies and SACS general education requirements under one outcome-based initiative (Approved by the Advisory Council of Presidents, April 2006). 1

4 The new guidelines issued by SCHEV s 2007 Task Force on Assessment and adopted by Council continue to identify that assessment should: Be congruent with the institution s mission and goals; Provide the kind of data needed for informed decision-making about curricula and offers both policymakers and the general public useful information on student learning; Continue to fit, rather than drive, the institution; Be reasonable in its requirements for time, resources, and personnel; Be integrated with the institution s larger framework for continuous improvement and public accountability; Employ both valid and reliable measurements of educational experiences and student learning; Focus on the improvement of learning while providing meaningful demonstration of accountability; and Employ the six core areas and explore options to address the Council s preferred value-added approach that speaks to demonstrable changes as a result of a student s collegiate experience. Based on the requirements from SCHEV s adopted Guidelines for Assessment of Student Learning, the VCCS in 2007 established a VCCS Core Competency Assessment Planning Task Group to develop a plan for assessing core competencies using a value-added analytical approach. The SCHEV s Guidelines identified a menu of approved value-added analytical approaches (i.e. longitudinal, cross-sectional, and residual analysis value-added approaches); data collection procedures (i.e., course embedded, single setting, and portfolio); and measurement strategies (i.e., selected response and constructed response). The VCCS adopted in 2008 to use the cross-sectional analytical approach, course embedded and single setting data collection procedures, and selected response measurement strategies in an effort to conform to SCHEV guidelines and implementation documents. The VCCS will assess annually new firsttime degree students and upcoming degree graduates on two of the six core competencies. All core competencies will be assessed every three years. Each college will, however, need to assess annually graduates in all of the core competencies using other measures (direct and indirect). 2

5 Virginia Community College System (VCCS) General Education The Virginia Community College System (VCCS) defines its general education program as " that portion of the collegiate experience that addresses the knowledge, skills, attitudes, and values characteristic of educated persons. It is unbounded by disciplines and honors the connections among bodies of knowledge. VCCS degree graduates will demonstrate competency in the following general education areas: communication, critical thinking, cultural and social understanding, information literacy, personal development, quantitative reasoning, and scientific reasoning." [VCCS Policy Manual Section 5.0.2]. The specific general education goals and student learning outcomes that all VCCS degree graduates will be able to demonstrate competency and that each community college needs to assess are the following [VCCS Policy Manual Section ]: Communication: A competent communicator can interact with others using all forms of communication, resulting in understanding and being understood. Degree graduates will demonstrate the ability to: (1) understand and interpret complex materials; (2) assimilate, organize, develop, and present an idea formally and informally; (3) use standard English; (4) use appropriate verbal and non-verbal responses in interpersonal relations and group discussions; (5) use listening skills; and (6) recognize the role of culture in communication. Critical Thinking: A competent critical thinker evaluates evidence carefully and applies reasoning to decide what to believe and how to act. Degree graduates will demonstrate the ability to: (1) discriminate among `degrees of credibility, accuracy, and reliability of inferences drawn from given data; (2) recognize parallels, assumptions, or presuppositions in any given source of information; (3) evaluate the strengths and relevance of arguments on a particular question or issue; (4) weight evidence and decide if generalizations or conclusions based on the given data are warranted; (5) determine whether certain conclusions or consequences are supported by the information provided, and (6) use problem solving skills. Cultural and Social Understanding: A culturally and socially competent person possesses an awareness, understanding, and appreciation of the interconnectedness of the social and cultural dimensions within and across local, regional, state, national, and global communities. Degree graduates will demonstrate the ability to: (1) assess the impact that social institutions have on individuals and culture-past, present, and future; (2) describe their own as well as others personal ethical systems and values within social institutions; (3) recognize the impact that arts and humanities have upon individuals and cultures; (4) recognize the role of language in social and cultural contexts; and (5) recognize the interdependence of distinctive world-wide social, economic, geo-political, and cultural systems. Information Literacy: A person who is competent in information literacy recognizes when information is needed and has the ability to locate, evaluate, and use it effectively (adapted from the American Library Association definition). Degree graduates will demonstrate the ability to: (1) determine the nature and extent of the information needed; (2) assess needed information effectively and efficiently; (3) evaluate information and its sources critically and incorporate selected information into his or her knowledge base; (4) use 3

6 information effectively, individually or as a member of a group, to accomplish a specific purpose; and (5) understand many of the economic, legal, and social issues surrounding the use of information and access and use information ethically and legally. Personal Development: An individual engaged in personal development strives for physical well-being and emotional maturity. Degree graduates will demonstrate the ability to: (1) develop and/or refine personal wellness goals; and (2) develop and/or enhance the knowledge, skills, and understanding to make informed academic, social, personal, career, and interpersonal decisions. Quantitative Reasoning: A person who is competent in quantitative reasoning possesses the skills and knowledge necessary to apply the use of logic, numbers, and mathematics to deal effectively with common problems and issues. A person who is quantitatively literate can use numerical, geometric, and measurement data and concepts, mathematical skills, and principles of mathematical reasoning to draw logical conclusions and to make well-reasoned decisions. Degree graduates will demonstrate the ability to: (1) use logical and mathematical reasoning within the context of various disciplines; (2) interpret and use mathematical formulas; (3) interpret mathematical models such as graphs, tables and schematics and draw inferences form them; (4) use graphical, symbolic, and numerical methods to analyze, organize, and interpret data; (5) estimate and consider answers to mathematical problems in order to determine reasonableness; and (6) represent mathematical information numerically, symbolically, and visually, using graphs and charts. Scientific Reasoning: A person who is competent in scientific reasoning adheres to a self-correcting system of inquire (the scientific method) and relies on empirical evidence to describe, understand, predict, and control natural phenomena. Degree graduates will demonstrate the ability to: (1) generate an empirically evidenced and logical argument; (2) distinguish a scientific argument from a non-scientific argument; (3) reason by deduction, induction and analogy; (4) distinguish between causal and correlational relationships; and (5) recognize methods of inquiry that lead to scientific knowledge. 4

7 VCCS Core Competencies The Virginia Community College System s Report of the VCCS Task Force on Assessing Core Competencies, clarified and enhanced general education outcomes for students. Writing: In a written discourse the student will demonstrate the ability to state the purpose that addresses the writing task in a thoughtful way; organize content with effective transitions and effective beginning and ending paragraphs; develop logical and concrete ideas with effective use of paragraph structure; use appropriate and precise word choice; demonstrate few mechanical and usage errors with evidence of control of diction. Information Literacy: The information literate student will demonstrate the ability to determine the nature and extent of the information needed; access needed information effectively and efficiently; evaluate information and its sources critically and incorporate selected information into his/her knowledge base and value system; use information effectively to accomplish a specific purpose; and, understand many of the economic, legal, and social issues surrounding the use of information and access and use information ethically and legally. Quantitative Reasoning : The student will demonstrate the ability to use logical and mathematical reasoning within the context of various disciplines; interpret and use mathematical formulas; interpret mathematical models; use arithmetic, algebraic, geometric, and statistical models to solve problems; estimate and consider answers to mathematical problems in order to determine reasonableness; recognize and communicate the appropriate applications of mathematical and statistical models; and, represent mathematical information numerically, symbolically, and visually, using graphs and charts. Scientific Reasoning: The student will be able to generate an empirically evidenced and logical argument; distinguish a scientific argument from a non-scientific argument; reason by deduction, induction and analogy; and, distinguish between causal and correlational relationships. Critical Thinking: The student will demonstrate the ability to discriminate among degrees of truth or falsity of inferences drawn from given data; recognize unstated assumptions or presuppositions in given statements or assertions; determine whether certain conclusions necessarily follow from information; weigh evidence and decide if generalizations or conclusions based on given data are warranted; and, distinguish between arguments that are strong and relevant and those that are weak and irrelevant to a particular question at issue. Oral Communication: The student will demonstrate skill in idea development and verbal effectiveness by the use of language and the organization of ideas for a specific audience, setting and occasion and to achieve a purpose; nonverbal effectiveness, assuring that the nonverbal message supports and is consistent with the verbal message and responsiveness, communication skills modified based on verbal and nonverbal feedback. 5

8 The following is a multi-year analysis of each general education and core competencies component: Communication: Analysis Writing Core Competency Writing Core Competency Rubric Comparison of Graduates Graduates Group Total Papers Mean Score Median Score 2003 PDCCC VCCS PDCCC PHI 115 NUR Students Source: VCCS Writing Rubric using a scale of 1-6 with 1 being a low score. The scoring grid is based the following areas: focus, organization, content, style, conventions, and documentation. Note: 2003 results were based on a sample of PDCCC s best papers vs results based on all degree graduates Note: PHI 115 is the new capstone course for all degree students. NUR Students are the Nursing students. Writing Rubric Proficiency 2008 Graduates Characteristics of Mean Score Effective Writing Organization 4.55 Content 4.74 Focus 4.86 Style 4.88 Conventions 5.35 Documentation NA Note: Score is based on a range of 1-6 with 1 being low. Characteristics of Effective Writing Focus Writer demonstrates an awareness of audience and task. Writer establishes and maintains a clear purpose. Writer sustains a single point of view. Writer exhibits clarity of ideas. Content Information and details are specific to topic. Information and details are relevant to focus. Ideas are fully developed. Content reflects critical thinking, depth, and complexity. Organization 6

9 Logical order or sequence is maintained. Paragraphs deal with one subject. Logical transitions are made within sentences and between paragraphs. Style Writer uses precise language. Writer uses effective word choice representing a range of vocabulary. Writer demonstrates voice, tone, originality of language. Writer uses variety of sentence structures, types, and lengths. Conventions Irregular mechanics (spelling, capitalization, punctuation) are minimum and do not detract from meaning. Usage (e.g., pronoun references, subject-verb agreement) errors are minimal and do not detract from meaning. Sentence structure is standard, for the most part. Irregularities (such as fragments) may be intentional, as a matter of style. Documentation Material from outside sources is well-integrated and documented consistently and correctly Consistent usage of citation style Developmental Completer n=28 No Developmental Writing Core Competency 2010 Graduates N=50 Rhetorical Critical Organization Content & Style Conventions Knowledge Thinking Clarity n=22 Total Note: Based on 6 point rubric developed by VCCS English Faculty and Assessment Coordinators. Paul D. Camp Community College STAGE Testing General Education Communication Note: STAGE test was developed by IR staff members from the Virginia Community College System (VCCS) in 1999 and revised in The score ranges from 1-10 with 1 being low. Benchmark is set at 5. 7

10 Community College Survey of Student Engagement (CCSSE) Frequency Occurrence in Classroom (12c) Core Competency Communication: Writing PDCCC N=274 VCCS CCSSE Cohort PDCCC N=232 VCCS CCSSE Cohort Very little 8.3% 14.9% 11.6% 6.1% 13.0% 11.6% Some 23.0% 29.7% 29.2% 25.8% 28.9% 29.2% Quite a bit 39.0% 36.0% 37.6% 39.2% 36.6% 37.6% Very much 29.8% 19.4% 21.6% 28.9% 21.5% 21.6% Quite a bit or very much 68.8% 55.4% 59.2% 68.1% 58.1% 59.2% Writing Competency Rating of PDCCC Graduates by Faculty & Staff Writing Note: Survey Rating is based on a scale from 1-5 with 1 being low and 5 being high Value-Added in Writing Core Competency Before Entering and After Graduation Before After Diff Before After Diff Before After Diff Writing Source: Value-Added Graduate Survey based on a 5 point scale with 5 being high and 1 being low. Value-Added in Writing Core Competency Before Entering and After Graduation Before After Diff Before After Diff Before After Diff Writing Capstone Course (PHI 115: Practical Reasoning) Written Communication PDCCC Graduates Term Written Communication Fall % Spring % Source: Blackboard: Core Competencies percent proficient 8

11 Capstone Course (PHI 115) Writing Communication Core Competencies Assessments % Proficiency and Mean Score (0-3) Using Rubric Core Competency & Objectives Action Plan Writing Communication 74.2% 84.6% Focus: State purpose that addresses the writing task in a thoughtful way. Graduates will achieve at least 80% proficiency. Organization: Organize content with effective transitions and effective beginning and ending paragraphs with 80% proficiency. Graduates will achieve at least 80% proficiency. Content: Develop logical and concrete ideas with effective use of paragraph structure. Graduates will achieve at least 80% proficiency. Style: Use appropriate and precise word choice where language and sentence structure are alive, mature, and varied. Graduates will achieve at least 80% proficiency. Conventions: Demonstrate few mechanical and usage errors, with evidence of control of diction. Graduates will achieve at least 80% proficiency. Note: ratings are based on a 0-3 point scale with 3 being the highest % % % % % % % % % % 3.00 For 2010, research, analyze, and modify material in PHI 115 to convey how to establish a purpose that addresses the task. Reevaluate material that focuses on developing complex writings. For 2010, redesign various assignments in PHI 115 to clearly demonstrate the correlation of organization and transitioning of concepts. For 2010, research, analyze and modify material in PHI 115 to convey how to establish a purpose that addresses developing valid ideas and applying them effectively. For 2010, develop a list of vocabulary words and identify the common usage of the words and research strategies related to thesaurus application in writing for the PHI 115 class. For 2010, design lessons that promote more interaction between the instructor and students in the PHI 115 course and utilize the discussion board to design lessons that promote sampling. 9

12 Results : According to the VCCS writing rubric, PDCCC writing skills were equal to or higher than the VCCS benchmarked in 2003 (4.63 vs. 3.94), however, the college s mean score in 2003 was based on a sample of the best papers. In 2008, the writing scores consisted of all degree graduates. In 2008, its mean score was The 2008 graduates who also took the new capstone course (PHI 115) scored even better with The weakest area in writing competency appears to be in the area of organization. According to the 2008 Community College Survey of Student Engagement (CCSSE), the frequency of occurrence in the classroom of writing clearly and effectively has decreased slightly from 2005 (68.1% quite a bit or very much in 2008 vs. 68.8% for 2005). However, the level of occurrence is higher than the VCCS averages (55.4% in 2005 and 58.1% in 2008) and the CCSSE averages (56.9% in 2005 and 59.2% in 2008). The perception by faculty and staff (using survey results), show that the writing skills of its graduates have improved in 2008 vs (3.89 vs. 3.64). This is reinforced by the graduate value-added (Pre-Post) survey which show the perception of skill level when students first entering the college and at graduation from There was an improvement in all three years. The writing skill level at graduation had increased from 4.42 in 2007 to 4.49 in The college had recognized for the past few years that PDCCC competencies in all of the core competency areas needed improvement, especially when compared to the VCCS as a benchmark. As a result, the college established a capstone course (PHI 115) in Changes to improve the course in included, handouts and booklets developed in each core competency module to assist students and web-sites were added to each core competency module so that students could visit and acclimate themselves to terms, principles, and formulas associated with specific subjects. Each student is also invited to sit in on courses related to any module/subject that he or she may have experienced difficulty in satisfactorily complying with the requirements (The courses must be instructed by one of the team teachers). The faculty teaching the course also advocates the use of PDCCC Library, PDCCC Learning Assistance Center (LATC), and tutorial modules developed by the VCCS and PDCCC. As a result, the teaching pedagogy and the proficiency level in the capstone have improved from 2006 (data was lost for comparison due to blackboard update). The proficiency level for has remained about the same with 80.7% in fall 2007 and 78.3% in spring st Cycle Action Plan (Closing the Loop): The college needs to continue to improve on its capstone course. The capstone team of instructors needs to use rubrics for each core competency in order to identify weaknesses in any of the skill levels under each core competency. The capstone instructors can use the three rubrics developed by the VCCS (written communication, oral communication, and information literacy), but the team needs to create new ones for quantitative reasoning, scientific reasoning, and critical thinking. 10

13 Each fall term, all of the English 111 classes need to be assessing written communication using rubrics to identify specific weaknesses and to develop action plans for improvement. These course assessments will be completed by the end of term and sent to their dean and to the Director of Assessment & IR. A major point that the deans need to emphasize with their faculty is that assessment of writing competency is not just an evaluation of ENG 111 but of all disciplines. As a college, scholarly writing across the curriculum has to be integrated in other courses. Each discipline must work collaboratively to ensure our students will be successful and encouraged to use the writing rubric in evaluating writing assignments. The General Education Committee will identify other courses which have primary objectives under written communication and decide which ones should have the instructor assess written communication skill levels. The college will also develop written communication modules to be placed on the college s web site to assist faculty teaching the capstone course and any other faculty requiring written assignments for their course. All lead faculty should survey employers of their graduates on skill levels related to core competencies. The Director of Assessment & IR has an employer survey online available for lead faculty use. Results : In the graduate value-added (Pre-Post) survey which show the perception of skill level when students first entering the college and at graduation from There had been an improvement each year from In 2009, there was a slight decrease from 4.49 in 2008 to 4.38 in However the value added in 2009 increase 0.66 points vs points in In the capstone course, PHI 115, graduates writing proficiency was 74.2%. This was below the goal of all graduating achieving 80% proficiency. The weakest sub-category was in focus with 67.8% proficiency. The strongest sub-category was in conventions with 82.8% proficiency. Other sub-categories included organization (70.2%), content (77.8%), and style (72.6%). 2 st Cycle Action Plan (Closing the Loop): Over the past few years, English faculty at PDCCC have taken numerous steps to improve written communication skills. English faculty have employed use of rubrics to evaluate students' writing, integrated writing assignments with an interdisciplinary approach, increased the cut-score for student enrollment in English 111, adopted Patterns for College Writing - A Rhetorical Reader and Guide by Kirszner and Mandell, which has exemplars of various genres of writing, participated and taught writing workshops, and incorporated more technological resources to improve writing. 11

14 Based on the findings of students' proficiency levels in written communication, English faculty will align all English courses with Student Learning Outcomes, research and analyze best practices to improve writing, develop specific strategies geared toward PDCCC's student population that will enhance writing, and research the development and implementation of English 111 variations which may enhance levels of proficiency in written communication. The faculty should continue to tweak the writing rubric. Changes in the capstone course, PHI 115, as noted in the table above should be implemented. Results : From the writing rubric developed by the Virginia Community College System (VCCS), 2010 graduates who did not need any developmental courses performed better than graduates who had taken a developmental course. Overall, graduates were weakest in writing conventions (3.88). This was also a decrease from 2008 graduates with a mean score of 5.35 on a six-point scale. The strongest scores were in rhetorical knowledge (4.28). On the STAGE test which was developed by VCCS Assessment Coordinators, 2010 graduates scored a 6.89 which was above the benchmark score set at On the Faculty and Staff Survey, the 2010 graduates were rated 3.94 on a 5-point scale. This was an increase from 2008 graduates with 3.89 and the 2007 graduates with The Graduate Survey based on a 5-point scale with 1 being low showed value-added for the 2010 graduates. Their score increased from 3.99 upon entering the college to 4.45 at graduation. The 2010 graduates also showed an increase from the 2009 graduates score of The Capstone Course based on a 80% proficiency rubric showed that the 2010 graduates were proficient in writing communication skills 84.6% of the time vs. 74.2% in The rubric also showed the 2010 graduates had a rating of 2.55 vs (based on a 0-3 scale with 3 being the highest). The weakest area for 2010 graduates was in the area of organizing content with effective transitions and effective beginning and ending paragraphs. 3 rd Cycle Action Plan (Closing the Loop): To improve student success in organization for written communication, use of effective transitions, and development of beginning and ending paragraphs, English faculty used They Say, I Say by Gerald Graff and Cathy Birkenstein, which contains sentence templates and transitional words that show students how to juxtapose them within their writing to help with coherence and organization. In addition, faculty provided numerous exemplars of effective and ineffective paragraphs/essays that provided students concrete examples of weak and strong written expressions. These analytical skills were employed as students wrote their own papers. 12

15 Finally, faculty reviewed the writing rubric with students and demonstrated how organization is evaluated in their writing; moreover, faculty required outlines for student papers, which also aided students in developing their writing. In the science courses, four current event writings were required in Biology 101. In Biology 142, instructors enhanced the use of the Learning Resource Center for research and writing. This was done by increasing the previous requirement of two writing units to four. BIO 270, 150, 205, NAS 125 and GOL 111 required mandatory project writings. Although the previously assigned presentations sufficed, the new requirements were aimed at improving students organization of content and cohesion of ideas. 13

16 Communication: Analysis Oral Communication Core Competency Oral Communication Test Developed by James Madison University PDCCC Graduates vs. VCCS Mean Scores PDCCC VCCS (n=105) (n=2075) (n=94) (n=2447) Note: Benchmark = PDCCC mean/vccs mean Community College Survey of Student Engagement (CCSSE) Frequency Occurrence in Classroom (12d) Core Competency Communication: Oral PDCCC N=274 VCCS CCSSE Cohort PDCCC N=232 VCCS CCSSE Cohort Very little 8.1% 19.0% 15.6% 9.3% 16.4% 14.5% Some 27.6% 32.8% 32.2% 28.9% 30.3% 30.7% Quite a bit 42.2% 32.2% 34.3% 33.8% 33.5% 35.0% Very much 23.1% 16.9% 17.9% 29.0% 19.7% 19.8% Quite a bit or very much 65.3% 49.1% 52.2% 62.8% 53.2% 54.8% Oral Communication Competency Rating of PDCCC Graduates by Faculty & Staff Oral Communication Note: Rating is based on a scale from 1-5 with 1 being low and 5 being high Oral Communication Value-Added in Oral Communication Core Competency Before Entering and After Graduation Before After Diff Before After Diff Before After Diff Source: Value-Added Graduate Survey based on a 5 point scale with 5 being high and 1 being low. 14

17 Oral Communication Value-Added in Oral Communication Core Competency Before Entering and After Graduation Before After Diff Before After Diff Before After Diff Capstone Course (PHI 115: Practical Reasoning) Oral Communication Comparisons PDCCC Graduates Term Oral Communication Fall % Spring % Source: Blackboard: Core Competencies percent proficient Capstone Course (PHI 115) Oral Communication Core Competencies Assessments % Proficiency and Mean Score (0-3) Using Rubric Core Competency & Objectives Action Plan Oral Communication 92.6% 75.6% Appropriateness: Idea development, use of language and the organization of ideas for a specific audience, setting, and occasion are appropriate. Graduates will achieve at least 80% proficiency. Verbal Effectiveness: Idea development, use of language and the organization of ideas are effectively used to achieve a purpose. Graduates will achieve at least 80% proficiency. Nonverbal Effectiveness: The nonverbal message supports and is consistent with the verbal message. Graduates will achieve at least 80% proficiency. Responsiveness: Communication may be modified based on verbal and nonverbal feedback. Speakers/listeners demonstrate active listening behavior. Graduates will achieve at least 80% proficiency. Note: ratings are based on a scale of 0-3 with 3 being the highest % % % % % % % % 1.83 For 2010, no modifications at this time. For 2010, plan at least one face-to-face meeting in PHI 115 for all students to provide live examples of verbal effectiveness. For 2010, show videos in PHI 115 that provide examples of effective and ineffective nonverbal communication. For 2010, no modifications at this time. 15

18 Results : The results from the oral communication test shows that PDCCC has improved its mean scores in 2007 from 2006 (56.10 vs ). These mean scores are, however, below the VCCS mean score of 61 for 2006 and The college made 90.7% of benchmark in 2006 and 92.1% in According to the 2008 Community College Survey of Student Engagement (CCSSE), the frequency of occurrence in the classroom of speaking clearly and effectively has decreased slightly from 2005 (62.8% quite a bit or very much in 2008 vs. 65.3% for 2005). However, the level of occurrence is higher than the VCCS benchmarks (49.1% in 2005 and 53.2% in 2008) and the CCSSE averages (52.2% in 2005 and 54.8% in 2008). The oral communication perception by faculty and staff (based on survey data) also shows an increase in oral communication rating skill levels (based on a scale from 1-5 with 1 being low) from 3.73 in 2007 to 4.02 in This is reinforced by the graduate value-added (Pre-Post) survey which show the perception of skill level when students first entering the college and at graduation from There was an improvement in all three years. The oral communication skill level at graduation had increased from 4.31 in 2007 to 4.46 in In examining the oral communication component in the capstone course (PHI 115), the oral communication proficiency level (determined by the instructor) for had increased from 75.8% in fall 2007 to 86.6% for spring st Cycle Action Plan (Closing the Loop): The college needs to continue to improve on its capstone course. The capstone team of instructors needs to use the oral communication rubric in order to better identify weaknesses in any of the skill levels under the oral communication core competency. Each fall and spring term, all speech classes need to be assessing oral communication using the VCCS rubric and to develop action plans for improvement. Speech (SPD 100) course assessments will be completed by the end of term and sent to their dean and to the Director of Assessment & IR. The General Education Committee will identify other courses that have primary objectives under oral communication and decide which ones should have the instructor assess written communication skill levels. An analysis of the college s curriculum map will help in this process. All lead faculty should survey employers of their graduates on skill levels related to core competencies. The Director of Assessment & IR has an employer survey online available for lead faculty use. 16

19 Results : The graduate value-added (Pre-Post) survey shows the perception of skill level when students first entering the college and at graduation from There was an improvement each year from For 2009, the oral communication skill level at graduation had decreased slightly to 4.37 vs in In the capstone course, PHI 115, the overall proficiency level was 92.6%. This exceeded the goal of 90% proficiency. In the sub-categories, all areas met the standard. The categories included appropriateness (100%), verbal effectiveness (90%), non-verbal effectiveness (90%), and responsiveness (90.4%). 2 st Cycle Action Plan (Closing the Loop): Oral Communications, taught within the Principles of Public Speaking Class (CST100) and the Practical Reasoning class (PHI115), has underwent some major changes within the past three years. Teaching in CST100 has changed from theory-based learning to skill-based learning, with students going through authentic learning exercises in order to apply knowledge gleaned from class materials. Students are required to show a higher level of comprehension by taking information they have learned and applying the techniques in real-life situations. Also, three years ago, a standardized rubric for grading oral presentations was not in place. Now the Communications Department at PDCCC uses a rubric for grading all oral assessments (see attached), which has brought about more consistency within classes. Oral Communications will not be implementing any changes for the next academic year; rather, it is necessary to obtain more data based on the current rubric and curriculum. Students are thriving and learning based on the current course setup. Faculty should continue to tweak the oral communication rubric. Make it clearer as to what meets proficiency and what doesn t. Make the changes recommended in the capstone course, PHI 115, (see table above). Results : On the Faculty and Staff Survey, the 2010 graduates were rated 3.97 on a 5-point scale with 1 being low. This is within the benchmark set at It is also above the 2007 rating of 3.73, but below 2008 with The Graduate Survey based on a 5-point scale with 1 being low showed value-added for the 2010 graduates. Their score increased from 3.76 upon entering the college to 4.40 at graduation. The 2010 graduates also showed an increase from the 2009 graduates score of

20 The Capstone Course based on an 80% proficiency rubric showed that the 2010 graduates were proficient in oral communication skills 75.6% of the time vs. 92.6% in The rubric also showed the 2010 graduates had a rating of 2.17 vs (based on a 0-3 scale with 3 being the highest. The weakest area for 2010 graduates was in the area of responsiveness based on verbal and nonverbal feedback (67% proficiency). 3 rd Cycle Action Plan (Closing the Loop): No new changes were anticipated in Oral Communications for ; however, additional instruction on the Speech Rubric was given. Instead of giving students a brief introduction to the four categories to be assessed (verbal effectiveness, nonverbal effectiveness, appropriateness, and responsiveness), the rubric was broken down into four different lectures and activities. The students learned more about the Speech Rubric categories, applied that information to their graded speeches, and achieved higher overall scores. In several other courses, faculty have included group oral communications activities. For example, in selected English classes students were required to deliver group oral presentations. In the science courses, Biology 101 and 141 required students to do presentations in pairs on concept overview. In Biology 150 and 205 oral presentations were optional but encouraged. NAS 125 and BIO 270 were offered online. Those students had an optional requirement to present video overviews of their projects. Proficiency levels were measured at 82% amongst those who participated. 18

21 Analysis Critical Thinking Critical Thinking Test Developed by James Madison University PDCCC vs. VCCS Test Standards PDCCC VCCS PDCCC VCCS N=115 N=79 Induction Deduction Analysis Inference Evaluation Total Note: Test Developed by JMU and modified from Watson-Glaser Critical Thinking Appraisal Note: PDCCC Benchmark = PDCCC sub-category mean/vccs sub-category mean Paul D. Camp Community College STAGE Testing General Education Critical Thinking Note: STAGE test was developed by IR staff members from the Virginia Community College System (VCCS) in 1999 and revised in The score ranges from 1-10 with 1 being low. Benchmark is set at 5. Community College Survey of Student Engagement (CCSSE) Frequency Occurrence in Classroom (12e) Core Competency Critical Thinking PDCCC N=274 VCCS CCSSE Cohort PDCCC N=232 VCCS CCSSE Cohort Very little 7.9% 8.8% 7.4% 8.4% 7.7% 6.6% Some 21.3% 28.9% 27.3% 21.5% 25.6% 25.8% Quite a bit 46.1% 39.9% 41.8% 38.7% 41.0% 41.5% Very much 24.8% 33.5% 23.5% 31.4% 25.7% 26.1% Quite a bit or very much 70.9% 73.4% 65.3% 70.1% 66.7% 67.6% 19

22 Critical Thinking Competency Rating of Graduates by Faculty & Staff Critical Thinking Note: Rating is based on a scale from 1-5 with 1 being low and 5 being high Critical Thinking Value-Added in Critical Thinking Before Entering and After Graduation Before After Diff Before After Diff Before After Diff Source: Value-Added Graduate Survey based on a 5 point scale with 5 being high and 1 being low. Critical Thinking Value-Added in Critical Thinking Before Entering and After Graduation Before After Diff Before After Diff Before After Diff Capstone Course (PHI 115: Practical Reasoning) Critical Thinking PDCCC Graduates Term Critical Thinking Fall % Spring % Source: Blackboard: Core Competencies percent proficient 20

23 Capstone Course (PHI 115) Critical Thinking Core Competencies Assessments % Proficiency and Mean Score (0-3) Using Rubric Core Competency & Objectives Action Plan Critical Thinking 90.2% % 1.84 Evaluation: Present sources that discriminate among degrees of truth or falsity of inferences drawn from given data with 80% proficiency 89.7% % 2.54 Research, analyze and modify 20% of existing assessment materials in PHI 115. Analysis: Recognize unstated assumptions or presupposition in given statements or assertions with 80% proficiency Determine whether certain conclusions follow from data in given statements. Present a fundamental paradigm for various methods developed to interpret information with 80% proficiency Inference: Survey evidence based on the given data. Generalize if the conclusion is valid or invalid based on supporting premises with 80% proficiency Inductive & Deductive Reasoning: Distinguish between arguments that are strong and relevant and those that are weak and irrelevant with 80% proficiency 90.0% % % % % % % % 0.85 Redesign online materials in PHI 115 since face-to-face class performance was 40% greater than online students. Modify lessons to survey and convey foundational principles utilized in critical thinking evaluation and processing in PHI 115. Redesign identified material to better present methodologies used in the simplification and expression of complex critical thinking concepts. Redevelop identified material to better present methodologies used in the identification of major and subordinate premise in PHI 115. Design more lessons that assist in designating the validity of conclusions. Redesign identified material to better assist students in identifying valid and invalid data. Enhance or modify existing lessons to clearly describe and define inductive and deductive reasoning so students can designate valid arguments of critical thinking. A separate assessment will be developed to better isolate this objective in PHI 115. Results : The results from the critical thinking test show that PDCCC has improved its overall mean score in 2007 from 2006 (13.86 vs ). This represents 90% of benchmark in 2007 vs. 87% of benchmark in 2006 (PDCCC s benchmark score will be equal to or higher than the VCCS average). It has also shown improvement in all sub-scores (induction, deduction, analysis, inference, evaluation) in 2007 compared to All of PDCCC s scores, however, are below the 21

24 VCCS mean scores in all categories and below the test standards in all areas. PDCCC s benchmark comparison with the VCCS improved in all sub-categories from 2006 to 2007 (induction was 89.3% in 2006 vs. 91.0% in 2007; deduction was 83.3% in 2006 vs. 88.6% in 2007; analysis was 88.5% in 2006 vs. 90.2% in 2007; inference was 88.4% in 2006 vs. 89.3% in 2007; evaluation was 82.3% in 2006 vs. 91.0% in 2007; and the total was 86.8% in 2006 vs. 90.0% in 2007). According to the 2008 Community College Survey of Student Engagement (CCSSE), the frequency of occurrence in the classroom of thinking critically and analytically has decreased slightly from 2005 (70.1% quite a bit or very much in 2008 vs. 70.9% for 2005). The level of occurrence was below the VCCS benchmarks in 2005 (70.9% vs. VCCS 73.4%), but higher in 2008 (70.1% vs. VCCS 61.7%). The frequency of occurrence in the classroom was higher in 2005 and in 2008 compared to the CCSSE cohorts (70.9% vs. CCSSE 65.3% in 2005 and 70.1% vs. CCSSE 67.6% in 2008). The critical thinking perception by faculty and staff (based on survey data) show an increase in critical thinking rating skill level (based on a scale from 1-5 with 1 being low) from 3.51 in 2007 to 3.85 in This may be due to the introduction of a capstone course which reinforces all areas of gen ed. This is reinforced by the graduate value-added (Pre-Post) survey which show the perception of skill level when students first entering the college and at graduation from 2006 to In all three years, graduates felt that there was significant improvement in skill level between when they first entered the college and at graduation. In evaluating the graduation data, there was also a slight improvement from 2007 to The critical thinking skill level at graduation had increased from 4.33 in 2007 to 4.36 in In examining the critical thinking component in the capstone course (PHI 115), the critical thinking proficiency level (determined by the instructor) from had increased from 75.8% in fall 2007 to 86.6% for spring st Cycle Action Plan (Closing the Loop): The college needs to continue to improve on its capstone course. The capstone team of instructors needs to develop a critical thinking rubric in order to better identify weaknesses in sub-categories of critical thinking (induction, deduction, analysis, inference, and evaluation). Each fall and spring term, the critical thinking rubric should be used in the PHI 115 class assessment and in the development of action plans for course improvement. When the class action plan is completed at the end of term, it will be sent to their dean and to the Director of Assessment & IR. The General Education Committee will identify other courses that have primary objectives under critical thinking and decide which ones should have the instructor assess critical thinking skill levels. An analysis of the college s curriculum map will help in this process. 22

25 The college should develop a critical thinking module and post it on the college s web site so that faculty will be able to use it in their classes when they have critical thinking assignments. All lead faculty should survey employers of their graduates on skill levels related to core competencies. The Director of Assessment & IR has an employer survey online available for lead faculty use. Results The graduate value-added (Pre-Post) survey shows the perception of critical thinking skill level when students first entering the college and at graduation from 2006 to In all four years, graduates felt that there was significant improvement in skill level between when they first entered the college and at graduation. In evaluating the graduation data, there was, however, a slight decrease from 2008 to 2009 (4.36 vs. 4.29). In the capstone course, PHI 115, graduates met the goal of 80% proficiency in all categories. The overall proficiency level was 90.2%. The sub-categories included evaluation (89.7%), analysis (90%), inference (92.4%), and inductive and deductive reasoning (92.4%). For fall 2009, however, analysis and inductive and deductive reasoning fell below 80%. 2 st Cycle Action Plan (Closing the Loop): Within the previous three years, the Practical Reasoning Committee and coordinator have instituted a variety of methods to improve the efficiency and effectiveness of critical thinking skills through the capstone course, (PHI-115, Practical Reasoning). Initially, the course was structured in a community learning/team teaching format. The methodology transitioned from six instructors to three instructors and then to one instructor. The CLEP manual was used as the course book and 85% of the selective resources were used to enhance its limitation. As a result of fluctuating performance scores and to promote cohesiveness and consistency of the course, Think has been chosen to serve as the course manual. Within the school term of , rubrics have been added in the course grading and assessment methods to better identify strengths and weaknesses in student learning outcomes. The capstone course has expanded from being offered only during the fall semester to being offered online. PHI-115 has also been added to the summer semester academic calendar. Data collected from each revision encourages the following actions: (1) PHI-115 has been added to all academic degree curriculum of PDCCC; (2) coordinator of the course will communicate directly with each department chair instead of a committee to make improvements on student learning outcomes success by graduates; (3) more instructive presentations will be added to blackboard exercises, quizzes, and test to measure student learning outcomes; and (4) changes will be implemented as identified by the data within the capstone course. The critical thinking rubric developed last year will continue to be tweaked. The PHI 115 capstone course will be modified to address weaknesses in students proficiencies (see action plan in table above). Additional samples and tutorial information, related to the analysis component and determining conclusions and proficiently interpreting data component will be incorporated into the PHI-115 critical thinking module to enhance student proficiency. A request will also be made to faculty requesting that a concerted effort be made to emphasize strengthening the two components across the curriculum. 23