Outcome: Compare and contrast different research methods used by psychologists including their respective advantages and disadvantages.

Transcription

1 BS Psychology FY14 Closing the Loop: Previous FY Assessment Summary Annual Assessment Summary Ongoing Providing Department: BS Psychology Responsible Roles: In the text box below, please complete your annual assessment summary in which you close the loop on each outcome from the previous years' SMART. Please see the information tab ("i") for additional details. Overall, departmental assessment strategies and results were evaluated to be effective and positive during FY2014. However, departmental strategies can continue to be streamlined and improved, such as the possible revision of current assessments. A brief summary of assessment areas and results follow. Outcome: Identify key characteristics of major content domains in psychology (e.g., cognition and learning, developmental, biological, and sociocultural) At the conclusion of their PSYC 1101 Introduction to General Psychology course, pre-post testing indicated students demonstrated increased familiarity with major concepts in psychology as reflected by higher post-test scores. However, the goal of a 10% increase was not met (7.50% increase). Pretests were administered via an online survey for all sections. Posttests were completed in class as stand-alone assessments or embedded in final exams, or online via D2L as part of a final exam. Variables such as method and timing of the post-test are being further evaluated for FY For example, we have standardized pre-test and post-test administration procedures across all course sections. We will also be evaluating a new set of items from a revised test bank that is consistent with the updated 2013 recommended learning outcomes for undergraduates in psychology. Outcome: Compare and contrast different research methods used by psychologists including their respective advantages and disadvantages. At the conclusion of PSYC 2700 Statistics Applied to the Behavioral Sciences, pre-post testing indicated students demonstrated increased facility at identifying research methods used by psychologists including their respective advantages and disadvantages as reflected by higher post-test scores. At the conclusion of PSYC 2800 Research Methods, pre-post testing indicated students demonstrated increased facility at identifying common fallacies and poorly supported assertions regarding behavior as reflected by higher post-test scores. At the beginning of PSYC 3000-level courses the overall average score across questions for testing retention of material from PSYC 2700 and PSYC 2800 is at an acceptable rate. Students demonstrated retention of statistical concepts and knowledge related to research methods in psychology from PSYC 2700 Statistics Applied to the Behavioral Sciences, as evidenced by an average score of 78.83% on applicable questions. Students also demonstrated retention of statistical concepts and knowledge related to research methods in psychology from PSYC 2800 Research Methods, as evidenced by an average score of 84.33% on applicable questions. We will be evaluating a new set of items for the PSYC 2700 and PSYC 2800 assessments from a revised test bank that is consistent with the updated 2013 recommended learning outcomes for undergraduates in psychology. Page 1 of 14

2 Outcome: Identify aspects of individual and cultural diversity and the interpersonal challenges that often result from diversity and context. This is our first assessment of these aspects and we are pleased by the responses from our graduating seniors endorsing personal awareness of individual and cultural diversity and the interpersonal challenges that can result from diversity and context. Across the 12 relevant items from the survey, and across all respondents, results of the inventory revealed surveyed students had endorsed experiences with all of them. The mean percentage of experiences across students was 85.50%. This is our first use of the item subset from the Psychology Skills Inventory. Over the next year we will be evaluating a revised instrument that appears to be better suited for addressing this goal. Outcome: Write using APA style. With regard to writing, we have evidence to continue our emphasis on APA-style writing assignments across our curriculum. In our PSYC 2800 and 3000-level courses, students ability to communicate psychological concepts and methods in APA style were evaluated to be good. We will continue to use this outcome as currently formulated so as to build a sufficient baseline data pool (e.g., 3-5 years) before possible modifications to the strategy or desired outcomes are considered. Outcome: Develop evidence of attaining skill sets desired by psychology-related employers. We are pleased by the response from our graduating seniors for the most common experiences while in Psychology at GC. The mean percentage of individual experiences endorsed has increased from 64 to 73.50% since FY The lowest ranked skills are NOT part of our required curriculum (e.g., providing counseling). While we have met our goal of 70%, we will continue to discuss as a department ways we can raise the acknowledgement of the middle ranked items and explore ways of increasing students range of experiences within the required curriculum. Overview and Future Directions Aggregate and course-level assessment results for FY2014 were communicated to the department at the beginning of the Fall 2014 semester during both department-wide and small group meetings. Assessment methods and results were discussed in context of the courses, and the overall departmental assessment plan. The departmental consensus was to continue current strategies and refine processes where possible. Our departmental strategy has also been shared with psychology faculty from other departments at a national conference. Anecdotal feedback has been very positive from other faculty, individuals involved with undergraduate psychology efforts, and members of the APA task force that developed the recommended learning outcomes for psychology undergraduates. Overall, we have evidence to support our current efforts. In light of the current data, established strategies and methods are being conveyed to new departmental faculty as part of their orientation to the department. Beginning with AY14 we have cohorted Freshman Psychology Majors in ENGL 1101 where they focus on APA writing style. In addition, these same Freshman are cohorted in MATH 2600 (Statistics) in order to allow the Math professors to focus on content specifically related to psychology, and to build a peer community among our declared majors in their first year here. Page 2 of 14

3 A goal from FY2012 and 2013 was to evaluate the timing and modality of departmental posttests. In turn, we have standardized the administration of both pretests and posttests through the use Qualtrics. We have also identified a revised edition of the psychology major experiences survey used with graduating seniors and it is planned for initial use during FY2015. Lastly, previous program goals and assessments were based on the American Psychological Association s (APA) Recommended Learning Outcomes for Undergraduate Psychology Majors (2007). In 2013, the APA updated recommended learning outcomes for undergraduates in psychology. Both departmental goals and assessments were reviewed in FY2014 and our goals were updated. Also as result of the review, we will be evaluating a new set of items for the PSYC 1101, PSYC 2700, and PSYC 2800 assessments from a revised test bank that is consistent with the updated 2013 APA recommended learning outcomes for undergraduates in psychology. Sources: American Psychological Association. (2007). APA guidelines for the undergraduate psychology major. Washington, DC: Author. American Psychological Association. (2013). APA guidelines for the undergraduate psychology major: Version 2.0. Washington, DC: Author. Retrieved from Perry, J. L., Foust, M. S., & Elicker, J. D. (2013). Measuring the varied skills of psychology majors: A revision and update of the Academic Skills Inventory. Office of Teaching Resources in Psychology: Society for the Teaching of Psychology. Retrieved from White, J. N. (2014, January). Assessment strategies for course-level and departmental-level outcomes. Invited presentation at the National Institute on the Teaching of Psychology, St. Pete Beach, FL. New Sources Field No items to display. Page 3 of 14

4 Program Missions Psychological Science Program Mission The Department of Psychological Science is committed to supporting faculty who deliver high quality teaching, create research opportunities, and foster close student-faculty mentorship. We offer enthusiastic and innovative instruction by professors who teach courses in their specialty areas. Because this passion for psychology is sustained through active research and professional growth, the department is dedicated to supporting outstanding research facilities and continued faculty development. Our curriculum encourages undergraduate students to fully explore the multi-disciplinary field of psychological science. Through our teaching, research, and mentorship we instill respect for multiple viewpoints and approaches, while inspiring intellectual curiosity and personal growth. Responsible Roles: Page 4 of 14

5 Program Goals 1: Knowledge Base in Psychology: Students should demonstrate fundamental knowledge and comprehension of the major concepts, theoretical perspectives, historical trends, and empirical findings to discuss how psychological principles apply to beh. problems. Responsible Roles: Course(s) or Other Activities Addressing the Goal PSYC 1101 Introduction to General Psychology PSYC 2700 Statistics Applied to the Behavioral Sciences PSYC 2800 Research Methods PSYC 3XXX Selected 3000-level courses 2: Scientific Inquiry and Critical Thinking: The skills in this domain involve the development of scientific reasoning and problem solving, including effective research methods. Responsible Roles: Course(s) or Other Activities Addressing the Goal PSYC 2700 Statistics Applied to the Behavioral Sciences PSYC 2800 Research Methods PSYC 3XXX Selected 3000-level courses 3: Ethical and Social Responsibility: The skills in this domain involve the development of ethically and socially responsible behaviors for professional and personal settings. Responsible Roles: Course(s) or Other Activities Addressing the Goal Survey of a randomly selected group of 25% of graduating seniors 4: Communication: Students should demonstrate competence in written, oral, and interpersonal communication skills. Page 5 of 14

6 Responsible Roles: Course(s) or Other Activities Addressing the Goal PSYC 2800 Research Methods PSYC 3XXX Selected 3000-level courses PSYC 4XXX Selected 4000-level courses 5: Professional Development: The skills in this domain refer to abilities that sharpen student readiness for post-baccalaureate employment, graduate school, or professional school. Responsible Roles: Course(s) or Other Activities Addressing the Goal Survey of a randomly selected group of 25% of graduating seniors Page 6 of 14

7 Student Learning Outcomes 1.1.1: Identify key characteristics of major content domains in psychology (e.g., cognition and learning, developmental, biological, and sociocultural) APA Goal 1: Knowledge Base of Psychology. Develop a working knowledge of psychology s content domains (APA 1.2) Pre-Post testing of applicable learning goals from test bank of American Psychological Association (APA) Recommended Learning Outcomes for Undergraduate Psychology Majors. Tested at Basic Level 10% increase (post-pre)/pre per question on pre-post test Overall, students scored approximately 7.5% higher on their Post Test scores (M = 70.84%) over their Pre Test scores (M = 63.34%). Do these results indicate that this outcome was met?: Overall, the average score across questions for testing retention of this material is at an acceptable rate. At the conclusion of their PSYC 1101 Introduction to General Psychology course, pre-post testing indicated students demonstrated increased familiarity with major concepts in psychology as reflected by higher post-test scores. However, the goal of a 10% increase from pre- to post-test was not met. Variables such as mode and timing of the post-test are being further evaluated for FY For example, we have standardized pre-test and post-test administration procedures across all course sections. No changes will be made to the outcome at this time. The composition of questions for the current assessment will be evaluated and compared against a new set of items from a new and revised test bank of items that is consistent with the 2013 updated APA recommended learning outcomes for undergraduates in psychology : Interpret simple graphs and statistical findings APA Goal 2: Scientific Inquiry and Critical Thinking. Demonstrate psychology information literacy (APA 2.2) Page 7 of 14

8 Written reports of simple graphs and statistical findings provide an ideal context for assessing effective interpretation and correct implementation of APA style Short exam questions from a randomly selected group of 25% of students in PSYC 2700 will be evaluated to be Good or better from a rubric designed to evaluate the interpretation of statistical data output and use of APA style. A rubric was developed to evaluate data interpretation and use of APA style. The evaluation rubric had anchor points of 0 = Poor, 1 = Adequate, 3 = Good, 5 = Excellent, for each of four primary paper components. Items were evaluated from 10 students, across both sections of PSYC 2700 during Spring Overall the submissions were evaluated to be Good (M = 3.70, SD = 0.88). Students ability to interpret basic statistical output in APA style was evaluated to be good. We will continue to use this outcome as currently formulated so as to build a sufficient baseline data pool (e.g., 3-5 years) before possible modifications to the strategy or desired outcomes are considered 2.2.1: Compare and contrast different research methods used by psychologists including their respective advantages and disadvantages APA Goal 2: Scientific Inquiry and Critical Thinking. Interpret, design, and conduct basic psychological research (APA 2.4) Course: PSYC Statistics Applied to the Behavioral Sciences. Pre-Post testing of applicable learning goals from test bank of American Psychological Association (APA) Recommended Learning Outcomes for Undergraduate Psychology Majors. Tested at Basic Level Page 8 of 14

9 10% increase (post-pre)/pre per question on pre-post test Overall, students in PSYC 2700 during Fall 2013 and Spring 2014 scored approximately 21.07% higher on their Post Test scores (M = 80.47%) over their Pre Test scores (M = 59.40%). At the conclusion of PSYC 2700 Statistics Applied to the Behavioral Sciences, pre-post testing indicated students demonstrated increased facility at identifying research methods used by psychologists including their respective advantages and disadvantages as reflected by higher post-test scores. No changes will be made to the outcome at this time. The composition of questions for the current assessment will be evaluated and compared against a new set of items from a new and revised test bank of items that is consistent with the 2013 updated APA recommended learning outcomes for undergraduates in psychology : Compare and contrast different research methods used by psychologists including their respective advantages and disadvantages APA Goal 2: Scientific Inquiry and Critical Thinking. Interpret, design, and conduct basic psychological research (APA 2.4) Course: PSYC Research Methods. Pre-Post testing of applicable learning goals from test bank of American Psychological Association (APA) Recommended Learning Outcomes for Undergraduate Psychology Majors. Tested at Basic Level 10% increase (post-pre)/pre per question on pre-post test Overall, students in PSYC 2800 during Fall 2013 and Spring 2014 scored approximately 11.23% higher on their Post Test scores (M = 85.27%) over their Pre Test scores (M = 74.04%). Page 9 of 14

10 At the conclusion of PSYC 2800 Research Methods, pre-post testing indicated students demonstrated increased facility at identifying research methods used by psychologists including their respective advantages and disadvantages as reflected by higher post-test scores. No changes will be made to the outcome at this time. The composition of questions for the current assessment will be evaluated and compared against a new set of items from a new and revised test bank of items that is consistent with the 2013 updated APA recommended learning outcomes for undergraduates in psychology : Compare and contrast different research methods used by psychologists including their respective advantages and disadvantages APA Goal 2: Scientific Inquiry and Critical Thinking. Interpret, design, and conduct basic psychological research (APA 2.4) "Background knowledge probe" in 3000-level classes at beginning of semester to assess understanding of statistics and research methods covered in PSYC 2700 & PSYC 2800 Students in level courses will score at least 70% on Background Knowledge probe Overall, students in 3000-level courses during Fall 2013 and Spring 2014 scored 82.50% on Background Knowledge Probe questions. At the beginning of PSYC 3000-level courses the overall average score across questions for testing retention of material from PSYC 2700 and PSYC 2800 is at an acceptable rate. Furthermore, students demonstrated retention of statistical concepts and knowledge related to research methods in psychology from PSYC 2700 Statistics Applied to the Behavioral Sciences, as evidenced by an average score of 78.83% on applicable questions. No changes will be made to the outcome at this time. The composition of questions for the current assessment will be evaluated and compared against a new set of items from a new and revised test bank of items that is consistent with the 2013 updated APA recommended learning outcomes for undergraduates in psychology. Page 10 of 14

11 3.1.1: Identify aspects of individual and cultural diversity and the interpersonal challenges that often result from diversity and context APA Goal 3. Ethical and Social Responsibility in a Diverse World. Adopt values that build community at local, national, and global levels (APA 3.3) Psychology Skills Inventory as a self-assessment of skills and knowledge obtained from the degree. Interviews with a randomly selected group of 25% of graduating seniors should demonstrate that students have had experience with 70% of relevant items from a Psychology Skills Inventory that cover skills taught within the Georgia College Psychology Program. Across the 12 relevant items from the survey, and across all respondents, results of the inventory revealed surveyed students had endorsed experiences with all of them. The mean percentage of experiences across students was 85.50%. Nine of the items were reported by 70% or more of the graduating seniors: I have accepted that many questions in psychology have no clear-cut answer, 100%; I have accepted that one's answers to a psychology question might depend on their theoretical position, 100%; I have discussed the rights and welfare of people or animals in a scientific experiment, 100%; I have discussed whether a research project was ethical or unethical, 100%; I can participate in a discussion about how my culture or background influences my values, 100%; I have accepted that beliefs about human behavior should be based on empirical evidence rather than on personal belief, 97.4%; I am able to use what I ve learned in my psychology classes to cooperate with individuals from cultures other than my own, 97.4%; I can participate in a discussion about how a particular psychological phenomenon differs depending on racial or cultural influences, 97.4%; I have discussed with others the impact of specific research projects on our society, 87.2%. We are pleased by the responses from our graduating seniors endorsing personal awareness of individual and cultural diversity and the interpersonal challenges that can result from diversity and context. This is our first use of the item subset from the Psychology Skills Inventory. Over the next year we will be evaluating a revised instrument that appears to be better suited for addressing this goal. Perry, J. L., Foust, M. S., & Elicker, J. D. (2013). Measuring the varied skills of psychology majors: A revision and update of the Academic Skills Inventory. Office of Teaching Resources in Psychology: Society for the Teaching of Psychology. Retrieved from Page 11 of 14

12 4.1.1: Write using APA style APA Goal 4. Communication. Demonstrate effective writing for different purposes (APA 4.1) Course: PSYC Research Methods. Written reports of research provide an ideal context for assessing effective writing and correct implementation of APA style. Students in PSYC 2800 will have papers evaluated to be Good or better from a rubric designed to evaluate the use of APA style. A rubric was developed to evaluate the use of APA style, including the ability to communicate clearly and the ability to critically evaluate a research problem. The evaluation rubric had anchor points of 0 = Poor, 1 = Adequate, 3 = Good, 5 = Excellent, for each of four primary paper components. Papers were evaluated from 11 students, across 3 sections of PSYC 2800 during Spring Overall the papers were evaluated to be Good (M = 3.78, SD = 1.05). Students ability to communicate psychological concepts and methods in APA style were evaluated to be good. We will continue to use this outcome as currently formulated so as to build a sufficient baseline data pool (e.g., 3-5 years) before possible modifications to the strategy or desired outcomes are considered : Write using APA style APA Goal 4. Communication. Demonstrate effective writing for different purposes (APA 4.1) Page 12 of 14

13 Written reports of research provide an ideal context for assessing effective writing and correct implementation of APA style Students in 3000-level courses will have papers evaluated to be Good or better from a rubric designed to evaluate the use of APA style. A rubric was developed to evaluate the use of APA style, including the ability to communicate clearly and the ability to critically evaluate a research problem. The evaluation rubric had anchor points of 0 = Poor, 1 = Adequate, 3 = Good, 5 = Excellent, for each of four primary paper components. Papers were evaluated from 38 students, across 7 sections of 3000-level courses during Spring Overall the papers were evaluated to be Good (M = 3.52, SD = 0.81). Students ability to communicate psychological concepts and methods in APA style were evaluated to be good. We will continue to use this outcome as currently formulated so as to build a sufficient baseline data pool (e.g., 3-5 years) before possible modifications to the strategy or desired outcomes are considered : Develop evidence of attaining skill sets desired by psychology-related employers APA Goal 5. Professional Development. Develop meaningful professional direction for life after graduation (APA 5.5) Psychology Skills Inventory as a self-assessment of skills and knowledge obtained from the degree. Interviews with a randomly selected group of 25% of graduating seniors should demonstrate that 70% of students have had experience with items from a Psychology Skills Inventory that cover skills taught within the Georgia Page 13 of 14

14 College Psychology Program. Across all of the possible 68 experiences, and across all respondents, results of the inventory revealed surveyed students had endorsed experiences with all of them. The mean percentage of experiences across students was 73.50%. The five most common experiences by graduating seniors (all reported at 100%) were as follows: I have taken a psychology exam where a large portion of the grade was based on essays; used psychology-related databases (e.g., PsychInfo) to search for information; have used the Internet to obtain information about a psychology topic; have used statistical software programs such as SPSS to analyze data; have discussed the rights and welfare of people or animals in a scientific experiment; have discussed whether a research project was ethical or unethical. The five lowest ranked experiences from the inventory were: I have independently designed and administered a psychology-related survey to a large (25+) group of individuals, 21.2%; have participated in a research project in the private sector (e.g., business, industry), 20%; have participated in a research project in the public sector (e.g., government agency), 20%; have gathered information by interviewing or ing experts in a psychology-related field, 17.9%; I have written research grants (or mini grants), 12.8%. We are pleased by the response from our graduating seniors for the most common experiences while in Psychology at GC. The mean percentage of individual experiences endorsed has increased from 64 to 73.50%. The lowest ranked skills are NOT part of our required curriculum. We have met our goal of 70% and current results suggest progress. We will continue to discuss as a department ways in which we can raise the acknowledgement of the middle ranked items and explore ways of increasing students range of experiences within the required curriculum. We will continue to use this outcome as currently formulated so as to build a sufficient baseline data pool (e.g., 3 years) before possible modifications to the strategy or desired outcomes are considered. Page 14 of 14