IPP Learning Outcomes Report. Faculty member completing template: Greg Kim Ju, Marya Endriga (Date: 1/17/12)

IPP Learning Outcomes Report Program: Department: Psychology MA (General) Psychology Number of students enrolled in the program in Fall, 2011: 48 (Appendix A) Faculty member completing template: Greg Kim Ju, Marya Endriga (Date: 1/17/12) Period of reference in the template: 2006 07 to present 1. Please describe your program s learning outcomes trajectory since 2006 07: Has there been a transformation of organizational culture regarding the establishment of learning outcomes and the capacity to assess progress toward their achievement? If so, during which academic year would you say the transformation became noticeable? What lies ahead; what is the next likely step in developing a learning outcomes organizational culture within the program? [Please limit your response to 200 words or less] Since 2006 07, the trajectory of learning outcomes has been shaped by 1) The Psychology MA Student Learning Objectives (2002; Appendix B); the department s self study (2006), 2) the most recent program review (2007), and 3) the department s annual assessment initiatives (2007 present). These factors have shaped a strong organizational culture of program faculty who have adopted a more systematic approach to the use and assessment of program learning outcomes. In particular, since 2006 07, the Psychology MA program has undergone a transformation to refine its learning outcomes, utilize specific measurement tools to assess these learning outcomes, and use these data to better inform the program. What lies ahead for the program is: 1) the prioritization of program learning outcomes for annual assessment planning, 2) continued development of methodologies and instrumentation to assist in assessment of learning outcomes, and (3) formalization of strategies that close the loop between assessment and instruction/curriculum development. 2. Please list in prioritized order (or indicate no prioritization regarding) up to four desired learning outcomes ( takeaways concerning such elements of curriculum as perspectives, specific content knowledge, skill sets, confidence levels) for students completing the program. For each stated outcome, please provide the reason that it was designated as desired by the faculty associated with the program. [Please limit your response per outcome to 300 words or less] 1) Competence in Acquiring, Reviewing, and Evaluating Information from the Psychological Literature. Competence in acquiring, reviewing, and evaluating information from the psychological literature is one of the areas identified in the Psychology MA program student learning objectives. Competence in the acquisition, review, and evaluation of psychological

literature was designated by program faculty as a primary learning outcome targeted for ongoing assessment. It has always been a primary value for the program because it is the common element shared by all of the program subdisciplines (e.g., biological, experimental, cognitive, social, clinical) and is a critical value for theory and research in doctoral programs. 2) Competence in Generating and Articulating Research Problems and Designing Sound Research Studies. Competence in generating and articulating research problems and designing sound research studies is one of the areas identified in the Psychology MA program student learning objectives. Competence in generating and articulating research problems and designing sound research studies was designated by program faculty as a primary learning outcome targeted for ongoing assessment. It has always been a primary value for the program because students need this understanding to effectively use assessment and measurement and to formulate and implement research strategies. Further, this is an important competency for students to independently conduct their own thesis/project. 3) Competence in Analyzing and Interpreting the Results of Data and Drawing Inferences and Conclusions from Empirical Results. Competence in analyzing and interpreting the results of data and drawing inferences and conclusions from empirical results is one of the areas identified in the Psychology MA program student learning objectives. Competence in analyzing and interpreting the results of data and drawing inferences and conclusions from empirical results was designated by program faculty as a primary learning outcome targeted for ongoing assessment. It has always been a primary value for the program because students need an understanding of psychological measurement and techniques of data analysis to conduct their own thesis/project. Further, students need this competency to advance to doctoral programs and beyond. 4) Competence in Writing Psychological Reports and Giving Professional level Oral Presentations. Competence in writing psychological reports and giving professional level oral presentations is one of the areas identified in the Psychology MA program student learning objectives. Competence in writing psychological reports and giving professionallevel oral presentations was designated by program faculty as a primary learning outcome targeted for ongoing assessment. It has always been a primary value for the program because students need this competency to complete and defend their own thesis/project. Further, students need this competency to succeed in doctoral programs and to function in a wide range of professional settings. 3. For undergraduate programs only, in what ways are the set of desired learning outcomes described above aligned with the University s Baccalaureate Learning Goals? Please be as specific as possible. [Please limit your response to 400 words or less] N/A

4. For each desired outcome indicated in item 2 above, please: a) Describe the method(s) by which its ongoing pursuit is monitored and measured. b) Include a description of the sample of students (e.g., random sample of transfer students declaring the major; graduating seniors) from whom data were/will be collected and the frequency and schedule with which the data in question were/will be collected. c) Describe and append a sample (or samples) of the instrument (e.g., survey or test), artifact (e.g., writing sample and evaluative protocol, performance review sheet), or other device used to assess the status of the learning outcomes desired by the program. d) Explain how the program faculty analyzed and evaluated (will analyze and evaluate) the data to reach conclusions about each desired student learning outcome. [Please limit your response to 200 words or less per learning outcome] 1) Competence in Acquiring, Reviewing, and Evaluating Information from the Psychological Literature (AYs: 2007 Present, e.g., http://webapps2.csus.edu/assessment/reports/07 08/07 08%20Psychology.pdf). This is an ongoing annual assessment project. a. Methods. Two forms: 1) Graduate student thesis/projects reviewed by committee members on two dimensions (Literature Review, Context for Thesis/Project) and; 2) Graduate student self assessment on three dimensions (Archival Research, Acquiring Information, Evaluating Research) before starting the program and after advancing to thesis candidacy (M=2.12 years). b. Sample. 1) N=22 master s students with ratings from 1 3 thesis committee members (N=51 ratings). 2) N=28 master s students with pre and post responses. c. Instruments. 1) Thesis/Project Competencies Assessment Form (TCAF): assesses two dimensions rated on a 4 point Likert scale from Below Minimal Competence (1) to High Level of Competence (4) (Appendix C) and; 2) Graduate Student Skills Assessment Form (GSSAF): assesses three dimensions rated on a 4 point Likert scale from Virtually None (1) to Proficient (4) (Appendix D). d. Analysis and Conclusions. TCAF means showed 3.59 and 3.63 for Literature Review and for Context, respectively. GSSAF means showed significant improvements (p =.001) in Archival Research, Acquiring Information, and Evaluating Research. Results indicate that the program has been successful with students achieving competence in knowledge, explanatory skills and research skills. 2) Competence in Generating and Articulating Research Problems and Designing Sound Research Studies (AYs: 2007 Present, e.g., http://webapps2.csus.edu/assessment/reports/07 08/07 08%20Psychology.pdf). This is an ongoing annual assessment project. a. Methods. Two forms: 1) Graduate student thesis/projects reviewed by committee members on two dimensions (Description of Research Problem, Methodology Addressing Problem) and; 2) Graduate student self assessment on two dimensions

(Generating Research Problems, Designing Research Studies) before starting the program and after advancing to thesis candidacy (M=2.12 years). b. Sample. 1) N=22 master s students with ratings from 1 3 thesis committee members (N=51 ratings). 2) N=28 master s students with pre and post responses. c. Instruments. 1) TCAF: assesses two dimensions rated on a 4 point Likert scale from Below Minimal Competence (1) to High Level of Competence (4) (Appendix C) and; 2) GSSAF: assesses two dimensions rated on a 4 point Likert scale from Virtually None (1) to Proficient (4) (Appendix D). d. Analysis and Conclusions. TCAF means were 3.62 for Description of Research Problem and 3.55 for Methodology Addressing Problem. GSSAF means showed significant improvements (p =.001) in Generating Research Problems and Designing Research Studies. Results show that the program has been successful with students achieving competence in areas of knowledge, explanatory skills and research skills. 3) Competence in Analyzing and Interpreting the Results of Data and Drawing Inferences and Conclusions from Empirical Results. (AYs: 2007 Present, e.g., http://webapps2.csus.edu/assessment/reports/07 08/07 08%20Psychology.pdf). This is an ongoing annual assessment project. a. Methods. Two forms: 1) Graduate student thesis/projects reviewed by committee members on two dimensions (Data Analysis, Conclusions from Analysis) and; 2) Graduate student self assessment on two dimensions (Performing Data Analysis, Drawing Conclusions) before starting the program and after advancing to thesis candidacy (M=2.12 years). b. Sample. 1) N=22 master s students with ratings from 1 3 thesis committee members (N=51 ratings). 2) N=28 master s students with pre and post responses. c. Instruments. 1) TCAF: assesses two dimensions rated on a 4 point Likert scale from Below Minimal Competence (1) to High Level of Competence (4) (Appendix C) and; 2) GSSAF: assesses two dimensions rated on a 4 point Likert scale from Virtually None (1) to Proficient (4) (Appendix D). d. Analysis and Conclusions. TCAF means ranged from 3.54 for Data Analysis to 3.45 for Conclusions from Analysis. GSSAF means showed significant improvements (p =.001) in Performing Data Analysis and Drawing Conclusions. Results show that the program has been successful with students achieving competence in areas of data analysis and drawing interpretation. 4) Competence in Writing Psychological Reports and Giving Professional level Oral Presentations (AYs: 2007 Present, e.g., http://webapps2.csus.edu/assessment/reports/07 08/07 08%20Psychology.pdf). This is an ongoing annual assessment project.

a. Methods. Two forms: 1) Graduate student thesis/projects reviewed by committee members on three dimensions (Writing of Thesis/Project, Oral Presentation, Responses to Questions during Oral Defense) and; 2) Graduate student selfassessment on one dimension (Professional Writing) before starting program and after advancing to thesis candidacy (M=2.12 years). b. Sample. 1) N=22 master s students with ratings from 1 3 thesis committee members (N=51 ratings). 2) N=28 master s students with pre and post responses. c. Instruments. 1) TCAF: assesses three dimensions rated on a 4 point Likert scale from Below Minimal Competence (1) to High Level of Competence (4) (Appendix C) and; 2) GSSAF: assesses one dimension rated on a 4 point Likert scale from Virtually None (1) to Proficient (4) (Appendix D). d. Analysis and Conclusions. TCAF means ranged from 3.56, Writing of Thesis/Project, to 3.67, Oral Presentation, to 3.36, Responses to Questions. GSSAF means showed significant improvement in Professional Writing (clear and grammatically correct writing, proper use of APA style) (p =.001). Results show that the program has been successful with students achieving competence in areas of professional writing and presentations. 5. Regarding each outcome and method discussed in items 2 and 4 above, please provide examples of how findings from the learning outcomes process have been utilized to address decisions to revise or maintain elements of the curriculum (including decisions to alter the program s desired outcomes). If such decision making has not yet occurred, please describe the plan by which it will occur. [Please limit your response to 200 words or less per item] 1) Competence in Acquiring, Reviewing, and Evaluating Information from the Psychological Literature. This set of learning outcomes was found to be a strength of the MA program. We have decided to maintain these aspects of our graduate curriculum structure, including continuing to assign term projects that require literature reviews in nearly all graduate courses and budgeting FTES to allow students to enroll in PSYC 299 (independent study) to complete literature reviews and thesis proposals under close faculty supervision. 2) Competence in Generating and Articulating Research Problems and Designing Sound Research Studies. These learning outcomes were assessed to be strengths of the MA program. As a result of assessment findings and graduate course impaction, we have increased the frequency with which we offer two core methods and statistics courses (PSYC 200 and PSYC 203) from once every two years to three times every two years (2011 Graduate Handbook p. 9 http://www.csus.edu/psyc/graduate/graduate%20handbook Fall2011.pdf) in order for graduate students to complete these courses in a timely manner and prior to engaging in major independent research projects. Continued data collection and analysis will allow this learning

outcome to be monitored on an ongoing basis. Updated results will be incorporated into our AY 2012 13 self study to determine whether curricular or instructional changes are necessary. 3) Competence in Analyzing and Interpreting the Results of Data and Drawing Inferences and Conclusions from Empirical Results. This moderately strong area, nevertheless, leaves room from improvement in the MA program. The AY 2007 80 Annual Assessment report noted that discussion among faculty members indicated future plans to emphasize to their students the need to carefully consider what conclusions may be drawn from the empirical results of their research (p. 11). In addition, we have increased the frequency with which we offer two core methods and statistics courses (PSYC 200 and PSYC 203) from once every two years to three times every two years (2011 Graduate Handbook p. 9 http://www.csus.edu/psyc/graduate/graduate%20handbook Fall2011.pdf) in order for graduate students to complete these courses in a timely manner and prior to engaging in major independent research projects. Continued data collection and analysis will allow this learning outcome to be monitored on an ongoing basis. Updated results will be incorporated into our AY 2012 13 self study to determine whether curricular or instructional changes are necessary. 4) Competence in Writing Psychological Reports and Giving Professional level Oral Presentations. Writing of psychological reports is a strength of the Psychology MA program. These results are consistent with graduate course norms that require substantial APA style writing in all courses. In addition, writing supervision and feedback from thesis advisors is extensive. Oral presentation skills are also a strength of the Psychology MA program. The 2007 08 Annual Assessment report notes that this data refers to the oral defense of the thesis, specifically, in which students do not know in advance what these questions will be, and faculty members may intentionally pose more challenging questions at this point in the final attempt to act as instructors to the students (p. 10). Given this nature of the thesis/project defense, data demonstrate that students are performing admirably and with high competence, and should translate well to less pressured situations (which would likely be rated even higher). No curricular changes are recommended at this time. Continued data collection and analysis will allow this learning outcome to be monitored on an ongoing basis. Updated results will be incorporated into our AY 2012 13 self study to determine whether curricular or instructional changes are necessary. 6. Has the program systematically sought data from alumni to measure the longer term effects of accomplishment of the program s learning outcomes? If so, please describe the approach to this information gathering and the ways in which the information will be applied to the program s curriculum. If such activity has not yet occurred, please describe the plan by which it will occur. [Please limit your response to 300 words or less] Our most recent alumni survey was completed in AY 2006 07 and was keyed to a subset of missions and goals set forth in the department s 2006 self study. As shown in Appendix E, the

23 item survey contained four quantitative parts, one qualitative part and demographic questions. A 5 point rating scale was used with higher ratings indicating higher educational quality. Of the 238 surveys sent out, 48 were returned, resulting in a 24% return rate which is generally consistent with the mailed survey methodology. Of those 48 surveys, 15 represented students who had received both a B.A. and a M.A. (n=8) or only a M.A. from our department. Survey results pertaining to the Psychology MA program showed that most ratings averaged a scale value of 4: Training, M=3.75 4.75; Activities/Experiences, M=3.0 4.86; Perceived Faculty Competence/Activity, M=3.3 4.75; Guidance/Preparation, M=2.67 4.86. Substantial changes have been made since this survey to help maintain areas of strength and address areas of improvement. For example, in the Training area, our methods courses give more exposure to different phases of the research process, e.g., conducting literature searches, analyzing data. In the area of Activities/Experience we have begun budgeting our FTES allotment to protect supervisory course offerings, such as PSYC 295F (fieldwork), PSYC 294 (Cooperative Research) and PSYC 295A (Teaching Assistantships). To support Perceived Faculty Competence/Activity, we began alternating semesters of 3 class and 4 class teaching loads, thereby increasing faculty assigned time for scholarly activity. To improve Guidance/Preparation, we shifted to a mentoring model with more one on one mentoring, guidance and supervision from faculty to students, which encourages timely completion of thesis/project. We recently adopted a system where student advisors each semester complete and sign off on a graduate student progress report to provide timely, written notification of any problems and the opportunity to discuss them and to provide any guidance to remediate problems. 7. Does the program pursue learning outcomes identified by an accrediting or other professional discipline related organization as important? Does the set of outcomes pursued by your program exceed those identified as important by your accrediting or other professional discipline related organization? [Please limit your response to 300 words or less] The program has utilized guidelines by the American Psychological Association (APA), recognized by the U.S. Department of Education and the Council of Higher Education Accreditation as the national accrediting authority for professional education and training in psychology, to theoretically identify and anchor its learning outcomes. The report entitled Scholarship in Doctoral Education (APA, 2006) identifies several critical learning outcomes for graduate students in or pursuing a PhD degree. We consulted this report to shape learning outcomes and then anchored these outcomes using APA s Guidelines and Principles for Accreditation of Programs in Professional Psychology (Appendix F). This guideline is intended to facilitate consistent quality and excellence in education and training in professional psychology. Germane to our program learning objectives is Domain B Program Philosophy, Objectives and Curriculum Plan. This domain emphasizes the importance of implementing a coherent curriculum that facilitates understanding and competence in a) the breadth of scientific psychology, its history, its research methods, and its applications (biological, cognitive, history

and systems, measurement, methodology, data analysis); b) the scientific, methodological, and theoretical foundations of practice in substantive areas of a program (individual differences, human development, abnormal/dysfunctional behavior, professional standards and ethics; c) diagnosing or defining problems through assessment/measurement and formulating and implementing intervention strategies (assessment/diagnosis, effective intervention, supervision); d) issues of cultural and individual diversity, and; e) attitudes important for lifelong learning, scholarly inquiry, and problem solving in relation to scientific and professional knowledge. It is important to note that the set of outcomes pursued by our program is consistent with the APA guidelines for doctoral programs. Since our MA program focuses on doctoral preparation, we believe it is important to educate and train our students with learning outcomes consistent with accredited doctoral programs to develop highly competent students who can advance to and succeed in PhD programs. 8. Finally, what additional information would you like to share with the Senate Committee on Instructional Program Priorities regarding the program s desired learning outcomes and assessment of their accomplishment? [Please limit your response to 200 words or less] The Psychology MA program has traditionally maintained a strong organizational culture with program faculty regarding assessment of student learning outcomes. As indicated above, current program learning outcomes have been shaped by self study efforts in 2006, feedback from a program review in 2007, and findings from annual assessment reports balanced against the educational and training needs of students preparing for doctoral programs. We will continue to use the current learning outcomes and refine our assessment tools. For example, we recently implemented a review of graduate student APA posters presented at professional conferences as another method to assess Competence in Analyzing and Interpreting the Results of Data and Drawing Inferences and Conclusions from Empirical Results. Preliminary results of 24 sets of research poster ratings by eight full time faculty show that ratings reflect outstanding quality (M=3.3 M= 3.4, with 1=Benchmark to 4=Capstone). We also will be completing a self study in the upcoming academic year (2012 13) in which we plan to conduct a focused inquiry on our graduate curriculum. The outcome of this process as well as the subsequent Program Review will inform further modifications to our student learning outcomes and their assessment.

APPENDIX A Psychology MA and Conditionally Classified* Students Enrolled Fall 2011 Current Enrolled Majors By Department/College - Primary Academic Plans Term: 2118 - Fall 2011 College: SSIS - College of Soc Sci & Inter Std Career: PBAC - Postbaccalaureate Department(s): PSYC - Psychology Selected Plan(s): PSYCNONECC - Psychology Cond. Classified, PSYCNONEMA - Psychology MA Run Date: Feb 2, 2012 Department Desc Plan Type Academic Plan Academic Plan Desc Total Students Psychology MAJ PSYCNONEMA Psychology MA 39 MAJ - Total 39 PRP PSYCNONECC Psychology Cond. Classified* 9 PRP - Total 9 Psychology - Total 48 Summary - Total for Selected Career(s), College/Department(s), Academic Plan(s) 48 *Conditionally classified students are admitted to the program but have course remediation that is required.

APPENDIX B Psychology MA Program Student Learning Objectives 1. Competence in Acquiring, Reviewing, and Evaluating Information from the Psychological Literature. 2. Competence in Generating and Articulating Research Problems and Designing Sound Research Studies. 3. Competence in Analyzing and Interpreting the Results of Data and Drawing Inferences and Conclusions from Empirical Results. 4. Competence in Writing Psychological Reports and Giving Professional level Oral Presentations.

APPENDIX C Thesis/Project Competencies Assessment Form

APPENDIX D Graduate Student Skills Assessment Form

APPENDIX E Student Outcome Assessment Alumni Survey Spring 2007

APPENDIX F APA Guidelines and Principles for Accreditation of Programs in Professional Psychology APA Guidelines and Principles for Accreditation of Programs in Professional Psychology Domain B: Program Philosophy, Objectives, and Curriculum Plan The program has a clearly specified philosophy of education and training, compatible with the mission of its sponsor institution and appropriate to the science and practice of psychology. The program s education and training model and its curriculum plan are consistent with this philosophy. 1. The program publicly states an explicit philosophy of training by which it intends to prepare students for the practice of psychology. The program s philosophy, educational model, and curriculum plan should be substantially consistent with the mission, goals, and culture of the program s sponsor institution. They must also be consistent with the following principles of the discipline: a) Psychological practice is based on the science of psychology, which, in turn, is influenced by the professional practice of psychology; and b) Training for practice is sequential, cumulative, graded in complexity, and designed to prepare students for further organized training. 2. The program specifies education and training objectives in terms of the competencies expected of its graduates. Those competencies must be consistent with: a) The program s philosophy and training model; b) The substantive area(s) of professional psychology for which the program prepares students at the entry level of practice; c) An understanding of professional issues, including ethical, legal, and quality assurance principles. 3. In achieving its objectives, the program has and implements a clear and coherent curriculum plan that provides the means whereby all students can acquire and demonstrate substantial understanding of and competence in the following areas: a) The breadth of scientific psychology, its history of thought and development, its research methods, and its applications. To achieve this end, the students shall be exposed to the current body of knowledge in at least the following areas: biological aspects of behavior; cognitive and affective aspects of behavior; social aspects of behavior; history and systems of psychology; psychological measurement; research methodology; and techniques of data analysis; b) The scientific, methodological, and theoretical foundations of practice in the substantive area(s) of professional psychology in which the program has its training emphasis. To achieve this end, the students shall be exposed to the current body of knowledge in at least the following areas: individual differences in behavior; human development; dysfunctional behavior or psychopathology; and professional standards and ethics; c) Diagnosing or defining problems through psychological assessment and measurement and formulating and implementing intervention strategies (including training in empirically supported procedures). To achieve this end, the students shall be exposed to the current body of knowledge in at least the following areas: theories and methods of assessment and diagnosis; effective intervention; consultation and supervision; and evaluating the efficacy of interventions; d) Issues of cultural and individual diversity that are relevant to all of the above; and e) Attitudes essential for lifelong learning, scholarly inquiry, and professional problem solving as psychologists in the context of an evolving body of scientific and professional knowledge.