Background. 1 During the fall semester of 2008, graduate faculty members reviewed all syllabi for graduate

Transcription

1 Background Manship School of Mass Communication Annual Report on Mass Communication Graduate Student Learning Assessment May 30, 2011 Amy Reynolds, Associate Dean, Graduate Studies & Research The Manship School graduate program follows an assessment matrix that covers discipline specific accreditation (ACEJMC) and SACS accreditation. Student learning is assessed in several ways in the graduate program, including the following: (1) review of syllabi by the graduate committee (indirect measure) (2) external professional review of designated courses (direct measure) (3) review of all theses, professional projects and dissertations by the graduate committee (direct measure) (4) completion of the graduate student assessment instrument by each graduate student s advisory committee (direct measure) (5) exit survey administered to all graduate students who complete their program (indirect measure) (6) alumni survey (indirect measure) These varying measures are used in combination to provide yearly assessments. Not every measure above is used every year. This year ( academic year), our assessment did not include the review of syllabi or the alumni survey. 1 Beyond what is noted above, our assessment this academic year did include responding to the results of our ACEJMC accreditation site team visit. Every six years, accredited journalism and mass communication programs are evaluated by an ACEJMC site team and given recommendations for improvement. The specific outcomes from that evaluation are noted in the next section. During academic year , the following assessments occurred: (1) The graduate committee reviewed all theses, professional projects and dissertations completed during the academic year. (2) The school administered the exit survey to all master s and doctoral students who attended the graduation ceremony during the spring 2011 semester. (3) The advisory committee members for master s and doctoral students who defended (thesis, professional project, or dissertation) in each semester ( academic year) completed the graduate student assessment instrument. ACEJMC Site Team Evaluation The Accrediting Council on Education in Journalism and Mass Communications (ACEJMC) is the agency responsible for the evaluation of professional journalism and mass communication programs in colleges and universities. Every six years, programs seeking accreditation write an extensive self-study, and provide program data related to nine standards mission, governance and administration; curriculum and instruction; diversity and inclusiveness; full-time and part-time faculty; scholarship: research, creative 1 During the fall semester of 2008, graduate faculty members reviewed all syllabi for graduate courses and made several recommendations. As a result, the school implemented changes to courses and program requirements. In fall 2009, four graduate courses were selected for external professional review. Course work from a sample of students in each class was collected during the semester. A committee of professional reviewers assessed the course work during the spring semester The next review of syllabi is scheduled for fall The next alumni survey is scheduled for 2012.

2 and professional activity; student services; resources, facilities and equipment; professional and public service; and, assessment of learning outcomes. The program submits the self-study to ACEJMC and it is part of the evaluation process. Also during the accreditation year, a site team of 4-7 trained administrators, faculty, and professionals from our field visit the program and evaluate it on those same nine standards. The self-study and the report written by the site team are the primary documents used by the council to determine re-accreditation. This is done at the undergraduate and master s degree levels. In , the Manship School received full re-accreditation. On the graduate assessment, the site team noted that the program had only two weaknesses: A limited number of graduate courses outside the core curriculum and an overly-broad definition of professional skills courses. As a result of the site team s assessment, the full faculty acted on several recommendations that a faculty committee put forth during the spring (2010) semester. That committee recommended splitting the master s degree into two tracks, one professional and one scholarly. During the summer of 2010, discussions about reshaping the master s degree continued, and the two-track idea was refined and specifically defined. In the fall semester (2010) the full faculty voted to make the following changes to the master s degree: 1. Reduce the core by one course (drops from 16 to 13 hours) 2. Create a new professional, skills-based track to better serve the needs of professional students, and to address the site team s concern about more clearly defining the professional skills courses 3. Create five new graduate courses to anchor the professional track. Three of these new courses Crisis Communication, Political Communication Writing, and Public Affairs Reporting received campus approval during the academic year. Two additional courses, Strategic Communication I and Strategic Communication II, are in development and we will seek campus approval and permanent course numbers for them during the academic year. Four of these five courses are scheduled for the school year. We are offering the two strategic communication courses as special topics courses as we finalize course proposals to send forward to the campus for approval next year. This addresses the site team s concern that we have a limited number of graduate courses outside the core curriculum. The School received campus approval of the new degree during the spring 2011 semester, and the new curriculum is available to students beginning this fall (2011). Review of Theses, Professional Projects and Dissertations The review of theses, dissertations and professional projects for the academic year included 14 theses, three professional projects and four dissertations. We also had one master s student complete the degree through the comprehensive exam process. Graduate School and Manship School process require each dissertation committee have five faculty members (one as the graduate dean s representative from outside the School, then at least three of the other four coming from the Manship School) all with standing as graduate faculty members on campus. At least half the committee must be full members of the graduate faculty. For theses, projects and exams at the master s level, committees have three faculty members, all within the Manship School, and at least one a full/senior member of the graduate faculty. All committee members are graduate faculty. The diligence with which we constitute committees is the best safeguard for quality. All of the dissertations, theses, projects and the one exam from this past academic year highlighted the rigor of the process. In all cases, students passed their defenses, and spent a minimum of one week making final adjustments/changes to their work to complete the final project and graduate. On average, students spent about 3-4 weeks making final adjustments beyond the defense.

3 Projects All of the projects evaluated were strong. Each showcased the students strength in writing and communication skills; the ability to incorporate a research element into a professional work; clear and concrete knowledge of a student s professional area of training; and, the ability to produce high-quality professional work that would serve as a component in a student s portfolio. Based on the high quality of the projects and the feedback from the student exit survey (see exit survey section later in the report), the faculty should work to facilitate more professional student projects, particularly if a student chooses the newly created professional track of the degree. Theses Nearly all of the theses were above average, and each was well written and showed the students ability to conceptualize a major research project and bring it to fruition. The biggest weakness in the theses was on the methodological side. About a fifth of the theses were weak in executing quantitative methodologies, mostly on the design side. Many did not have enough data or significant findings to support their hypotheses, and in a couple of cases an early re-design of the method might have fixed the student s problem. Encouraging committees to ask students to pre-test quantitatively designed projects is recommended. It s worth noting that this weakness only showed in a few theses and was not a sufficient weakness to fail any of the students. It s possible that some of the students who struggled methodologically with a thesis would have been better served on a professional track with a professional project. In most of the cases where students struggled methodologically, these students were professional students. They didn t take a heavy load of scholarly courses within the degree program, so the choice of a thesis at the end of their program may not have been the most appropriate. As scholarly students pursue the new scholarly track, the preparation for a thesis is much more rigorous and should eliminate the minor methodological issues observed. Exam The comprehensive exam option is new for our program. This year, we had our first student complete the master s degree by taking the comprehensive exams. This student struggled with the process, and barely passed. Her exam committee believes that her struggles were related to a lack of understanding of what is involved in a comprehensive exam at the master s level. The recommendation is to edit the current guide to the comprehensive exams to better explain the process to future students. The student s committee believes that had the student been more aware of what the comprehensive exam process involved she would have fared better. This was particularly true on the written side. The student effectively saved herself in the oral defense, where she showcased the depth and breadth of her knowledge. The disconnect between her performance in the written and oral components, and subsequent discussions with the student, leads us to believe that she was not as clear of the written exam expectations as she should have been. Part of that responsibility rests with the student, but part also rests with the program. Dissertation The dissertations reviewed were disappointing. In no case did a student fail, nor should he/she have failed. But, the level of overall success was average (see assessment instrument findings in the next section) for the four students included in the academic year reviews. The biggest weaknesses came in the area of methodology. Like the few master s students who struggled methodologically, these students also had difficulty in methodological design, interpreting data and results and applying statistical procedures. In some cases two involving students in which English was a second language the writing was weaker than we would normally expect to see. In all other areas, the dissertations were above average. Students showed the ability to apply relevant theories; to demonstrate depth of knowledge; to make a contribution to the field; and, to think critically, creatively and independently. It s difficult to make sweeping generalizations based on so few students, but based on the review of these dissertations, the recommendation is to ask the Graduate Committee to review the doctoral curriculum and explore ways to enhance the methodological training at the doctoral level.

4 Graduate Student Assessment Instrument, During the fall semester of 2008, the associate dean for graduate studies and research introduced a new instrument for the measurement of graduate learning outcomes. The graduate student assessment instrument consists of ten statements that are rated on a 5-pt scale (1=unsatisfactory, 2=below average, 3=average, 4=above average, 5=superior or N/A if unable to judge). The statements assess the student s knowledge of mass communication theories, the understanding and application of research methods and statistical procedures, core values and competencies 2, written and oral communication skills, and mastery of course content. It also allows for comments by committee members for program improvement. Each graduate student s advisory committee completes the instrument after the student s Final Examination (defense) every semester. The results of the student assessment for , compared to the previous year s results, are as follows: Average Score for Each Criterion/Item (5-pt. scale) MMC Students: All Knowledge and application of mass communication theories Ability to conceptualize and conduct research Understanding and application of research methods Understanding and application of statistical/analytical procedures Understanding of course content of degree program Understanding of area of specialization Understanding of (AEJMC) core values and competencies Overall quality of thesis or professional project or dissertation Oral communication skills Written communication skills MMC Students ( ): Project, Thesis, Exam Proj Thes Exam Knowledge and application of mass communication theories Ability to conceptualize and conduct research Understanding and application of research methods Understanding and application of statistical/analytical procedures Understanding of course content of degree program Understanding of area of specialization Understanding of (AEJMC) core values and competencies Overall quality of thesis or professional project or dissertation Oral communication skills Written communication skills Overall Average PhD Students Knowledge and application of mass communication theories The core values and competencies are the ACEJMC professional values and competencies, which are listed at the end of this report.

5 Ability to conceptualize and conduct research Understanding and application of research methods Understanding and application of statistical/analytical procedures Understanding of course content of degree program Understanding of area of specialization Understanding of (AEJMC) core values and competencies Overall quality of thesis or professional project or dissertation Oral communication skills Written communication skills Overall Average (MMC) Overall Average (PhD) Overall Average (both programs) In , master s students scored highest in their understanding of their area of specialization, but lowest in their understanding of statistical and analytical procedures and written communication skills. In , doctoral students ranked highest in their understanding of their area of specialization and the course content of their degree program, and lowest in their understanding of statistical and analytical research procedures. This year s results are similar. Overall, master s students scored highest in understanding their area of specialization and lowest in understanding and application of research methods. Doctoral students scored highest in understanding ACEJMC core values and competencies, and lowest in understanding and application of research methods. Previous efforts to address the weaknesses on the methodological side included the addition of new courses: MC7095 Media History: Research and Writing, MC7014 Qualitative Research Methods, and MC7202 Experimental Applications in Mass Communication Research. To enhance the theoretical side, two new theory courses were added as well. But, these courses were only offered in two-year rotations, so the number of students (particularly master s students) who could take these courses wasn t maximized. These courses need to be offered more often, and the doctoral curriculum needs reexamination to ensure that doctoral students are receiving the kind of rigorous methodological training they need. The observed weakness in methodology in this assessment instrument matches the weakness observed in the dissertation review. In terms of the master s degree, the faculty thinks the new tracks will largely address this issue because a student s coursework will better prepare and match his/her objectives and outcomes (professional or scholarly). As noted in the previous section, meaningful differences existed among projects, theses and the exam outcomes. When the assessment instrument data is examined in this more precise way, the students who completed the projects demonstrated the strongest abilities, while the lone student who took the exams fared most poorly. The assessment instrument data supports the recommendation that the two tracks should independently prepare students for all final projects, whether professional or thesis. That is, the final project should match the student s track (professional = project; scholarly = thesis); and, the coursework on each track should prepare a student for the appropriate final project. The recommendation to better explain the exam process to students is also supported. Exit Survey The exit survey is administered every May. Seven graduate students received their degrees in the spring of 2011, although one did not complete the exit survey. No doctoral students graduated in May The average GPA of these students is The average time it took all of the students to graduate

6 is a little over three years. Four master s students finished the program in two years, and one of the master s students was a part-time student. Of the graduates, one specialized in advertising; four specialized in public relations, and one in political communication. Of the six graduate students, two are male and four are female. Assistance and resources Only two master s students completed an internship while in the program, and both had paid positions. One of the graduate students consulted the internship coordinator. All but one, however, did consult the counselors, and all five found the counselors to be helpful (the associate dean for graduate studies is the primary adviser/counselor for graduate students). Work and funding All six students said they worked while in school. Four said they worked on campus for 20 hours per week or less. One said they worked off campus for more than 30 hours per week, and another said they worked both on and off campus for more than 40 hours per week. Only five students answered the question about funding sources. Of those five, four listed graduate assistantship or fellowship as their primary source of funding. One said he or she funded his education through the TOPS program. Careers Only four of the six students started their job search prior to graduation. Of these four, half have found jobs. They will be working as a program coordinator at the Reilly Center at LSU and as a communications specialist at BASF. Of the three students who state that they will not be entering the work force full-time, one will be continuing on in a doctoral program, one will continue to law school, and one will work part-time until full-time work comes along. Faculty Overall, the Manship faculty received good reviews from the graduate students. All agreed the faculty had high expectations for their work and that those expectations were realistic. Five of the six agreed that the faculty had relevant professional experience, and four of the six agreed that faculty made them aware of practical applications of the course content, with one student disagreeing. All respondents said faculty provided adequate feedback. Four students said faculty provided students with opportunities or involvement in course-related projects that extend beyond the classroom, with one student strongly disagreeing with that statement. All agreed the faculty treated students with respect. Career skills The Manship School stresses teaching skills students can use in their professional careers. The results of this survey indicate the school does a good job of providing students with these skills. All agreed faculty expected them to write well, and all said the school enabled them to write well. All six of the students said the Manship School trained them to speak well. All six agreed that faculty expected them to think critically, and all said the school enabled them to think critically. All said the school enabled them to problem solve and to perceive the similarities and differences in ideas. The six students also said the school enabled them to generate original thought, and all said the school enabled them to critically evaluate their own work and the work of others. All but one of the students said faculty expected them to be familiar with media history, and all said faculty expected them to have an understanding of mass communication theory. All students said the Manship School enabled them to have an understanding of mass communication theory and an understanding of the freedom of speech and the press. Five of the six students said that their experiences at the Manship School enabled them to deal with media ethics. Five of the six also said faculty expected them to understand statistics and to understand research methods and that the school provided them with an understanding of statistics. All said the school gave them an understanding of mass communication research methods. The six students said it was important to have a diverse faculty and it was important to have a diverse student body. All of the students said they understood issues of diversity and that they work well in groups. Four of the six students said the school provided them with sufficient job skills to enter their chosen field, with one student disagreeing. Technology

7 The Manship School is dedicated to providing the best technology the University can offer to its students. Five of six agreed faculty adequately use technology in the classroom. All agreed the school provides state-of-the-art equipment and facilities. Five of the six said the school enabled them to use computers effectively. Conclusions The graduates gave adequate overall reviews of their Manship education. All of the students said the quality of instruction was good and their courses were appropriately challenging. All six students place a high value on their education. Several of the students left comments at the end of the survey. Many identified the faculty as one of the greatest strengths of the program. However, others also stated that the faculty should be more open to working with students on professional projects. Other areas identified for improvement include the core curriculum and promotion of the program on a national level. Overall, the students voice concerns over the balance of the academic and professional aspects of the master s program. They seem to think the program is oriented more toward academic interests than professional interests. As discussed earlier, they gave relatively low ratings for opportunities to work with faculty on outside projects, indicating the faculty could do a better job of making students aware of practical applications for coursework and of professional opportunities outside of the classroom. It is unknown whether or not these students sought out guidance, and whether or not these students sought out the professionally oriented courses. To a certain extent, it is possible that these students failed to understand how to personalize the master s program to fit their own needs and goals. For example, only two students had an internship. It seems that students interested in professionally oriented education would seek out an internship. Only one student used the school s internship coordinator, and it was to gather general information. It could be that these students were generally unhappy with the program s core curriculum and would have preferred more leeway to take courses in their respective areas of concentration. These results come from a very small sample size (n = 6). There were not enough graduating master s students to get a representative sample on this survey. These results, therefore, cannot be generalized to the rest of the graduate students. Use of Results to Improve Program Several general themes emerge from the data. First, the master s degree program needs to do a better job meeting the needs of its professional students in terms of coursework and in terms of encouraging more professional students to complete a professional project instead of a thesis. The exit survey suggestions that students felt the program was unbalanced that the emphasis was too scholarly, and that they did not have as many professional course options. The two new degree tracks and the creation of new professional courses outside the core should bring the program back in balance. The ACEJMC site team, the exit survey and the assessment data all support the idea that the professional side of the program required attention. The new master s degree curriculum is a big step in the right direction to addressing this problem. For next year, the program will ensure that the newly created professional courses are offered, and that more students will have the option to pursue a professional project as the final step to degree completion. Second, the review of projects, theses, dissertations and exams, and the graduate student assessment instrument both show that our scholarly track for the master s students and the doctoral curriculum need to focus on enhancing methodological training. The program needs to make sure it s offering methods courses regularly, and it needs to review the doctoral curriculum with an eye toward improving methodological training within the program. Finally, the program needs to change the administration of the exit survey. Because graduate students finish their degrees year-round, we re missing the feedback from a large number of students by only administering the exit survey in the spring. We had 6 students out of 21 complete the survey for this

8 academic year. That number is too small to really be helpful to our assessment. Beginning this summer (2011), we will administer the exit survey at each graduation summer, fall and spring.

9 ACEJMC Professional values and competencies Individual professions in journalism and mass communication may require certain specialized values and competencies. Irrespective of their particular specialization, all graduates should be aware of certain core values and competencies and be able to: understand and apply the principles and laws of freedom of speech and press, for the country in which the institution that invites ACEJMC is located, as well as receive instruction in and understand the range of systems of freedom of expression around the world, including the right to dissent, to monitor and criticize power, and to assemble and petition for redress of grievances; demonstrate an understanding of the history and role of professionals and institutions in shaping communications; demonstrate an understanding of gender, race, ethnicity, sexual orientation and, as appropriate, other forms of diversity in domestic society in relation to mass communications. demonstrate an understanding of the diversity of peoples and cultures and of the significance and impact of mass communications in a global society. understand concepts and apply theories in the use and presentation of images and information; demonstrate an understanding of professional ethical principles and work ethically in pursuit of truth, accuracy, fairness and diversity; think critically, creatively and independently; conduct research and evaluate information by methods appropriate to the communications professions in which they work; write correctly and clearly in forms and styles appropriate for the communications professions, audiences and purposes they serve; critically evaluate their own work and that of others for accuracy and fairness, clarity, appropriate style and grammatical correctness; apply basic numerical and statistical concepts; apply tools and technologies appropriate for the communications professions in which they work.