Exit Interview with Dean, Department Chair and Leadership Team: March 22, 2012

Transcription

1 University of La Verne College of Education and Organizational Leadership Doctoral Program in Organizational Leadership External Review of Self Report External Examiner Report Prepared by: Jim Cox, Ph.D. Campus Visit: Saturday, March 10, 2012 Exit Interview with Dean, Department Chair and Leadership Team: March 22, 2012 Purpose of the External Review Process The purpose of this report is to validate, to the extent possible, the content of the Self Report, submitted by the faculty of the doctoral program in Organizational Leadership at the University of La Verne. Additionally, calling upon my 40 years of experience as an educational evaluation specialist, I will offer comments and suggestions relating to the overall quality of the doctoral program. Readers are cautioned that any comments made regarding strengths and weaknesses are made in the context of continuous improvement, not judgment. All educational programs have strengths and relative shortcomings. Any weaknesses cited in this review are not to be judged as finite weaknesses, but rather areas which may be less strong than others and deserving of attention. External Review Process The following steps were taken to complete this validation report: Reviewed the self report and accompanying documents submitted by the department Designed a questionnaire to be submitted to all Year 3 students who, at the time, were in session on campus. Visited the campus and conducted interviews with faculty and students Cross referenced information from the department s self study with that from the interviews and Year 3 questionnaire data. Included here were several telephone calls to faculty members requesting additional information or asking for clarification of issues. Prepared the external reviewer report This process occurred over a three week period, from March 5 through March 26, Overview of this Report A university prepared document, External Reviewer Guide for the Ed.D. Program and Department Reviews of Academic Programs, provided the structure for this report. The document cited five elements for the external reviewer to address: (1) learning outcomes, (2) methods and procedures to 1

2 assess learning outcomes, (3) program capacity, (4) overall strength of the program, and (5) action recommendations. This report follows that outline. I. Learning Outcomes For each year of the program, learning outcomes are the basis of the content and learning activities. Year 1, labeled Arena 1, is the personal/interpersonal arena, in which students develop knowledge and skills related to personal awareness and interpersonal effectiveness. The continuing message is that One must know oneself before effectively leading others. Arena 2, the second year, is the team arena, dedicated to developing effective groups and teams. The premise is that contemporary organizations depend upon the effective work of teams, and that the quality of any organization will be determined in large part by the quality of its working units. Knowing about and developing effective working groups requires extensive knowledge and skill. Arena 3, the organizational arena, requires an in-depth examination of change processes, theory, models, and leadership leading to organizational transformation. Interviews with faculty and students continuously sent this message: theory to practice theory to practice theory to practice that s where our focus is. Year1. The EDD program s first year cites 19 clearly stated outcomes relating to preparing students for authentic leadership in the personal and interpersonal areas. The emphasis is on self reflection, recognizing one s strengths as well as shortcomings (the shadow self) and how they deter effective leadership. Eleven self reflective instruments are used to give students a multi dimensional view themselves. Also included are 37 separate readings and 25 field work activities. They collectively provide a very large umbrella under which students are able to view themselves and their interaction with others. Understanding themselves allows them to assess their potential for becoming effective leaders, and to establish a road map for pursuing that to which they aspire. The faculty members most responsible for the Year 1 program provided me with the documents necessary to confirm that what I stated above is reality. Year 2. Year 2 uses eight outcomes to build knowledge and skills relating to group and team effectiveness. A quote placed on the cover of the ULV Pre-Facilitation Fieldbook emphasizes the value that the program places on Year 2 content. It states, Every time you hold a meeting, hold a conference or create a team, you have the opportunity to enhance relationships and further the mission of the organization. The most powerful daily tool of leadership is to effectively bring people together. Students engage in 19 activities during the year, once again emphasizing a dominant program intent: theory to practice. Each of the 19 identifies a combination of outcomes upon which to focus. As with Year 1, the faculty members most responsible for the Year 2 program provided me with the documents necessary for confirmation. 2

3 Year 3. A statement on page 26 of the self study establishes beautifully, the context for Year 3 of the program. It states, Contemporary organizations exist is social, political, and economic environments that change rapidly and unpredictably. Transferring learning into the work place in which students are personally engaged and challenged is the setting for 20 activities, each focusing on various combinations of 10 outcomes. The activities have been strategically designed so as to allow single outcomes to be a focus in many activities. That is to say, of the 20 activities in which students are engaged, 14 focus on at least seven of the ten. The opportunities for continuous reinforcement of skill development are great. Research. Ten outcomes, across the three-year research sequence move the students from establishing a theoretical base for conducting research through the inquiry process, and subsequently to the more specific areas of research design and statistics. The finite parts of the dissertation, from problem definition to data analysis and reporting are elements of the second year. An important aspect of the research portion of the EDD program is a second year group activity asking the students to conduct a piece of action research paralleling the steps involved in dissertation research. This progression prepares the students for the third year, conducting research for the dissertation. Students are closely monitored during the progression of their building research capabilities. Several activities with accompanying guides for assessing progress are elements of the sequence. 2. Methods and Procedures to Assess Learning Outcomes Continuous, ongoing monitoring and assessment of student progress is a cornerstone of the doctoral program. Data driven decision making is truly actualized as substantiated by the number of assessment tools that I reviewed. In summary, there are formal assessment tools for each of activities and assignments in both the Arenas and Research, all three years. In total, it breaks out like this: Arena 1 19 Outcomes 1. Leadership Autobiography 2. Type and Leadership Paper Rubric 3. Leadership and Balance 4. Your Shadow Self 5. Persuasive Speaking Feedback 6. Nonviolent Communication Fieldwork assessment 7. Coaching Fieldwork 8. Introduction to Leadership Theory Arena 2 8 Outcomes 1. Pre-Facilitation Practice in Nine Action Research Cycles 2. Virtual Teamwork and Presentation 3. Summative Paper on the LDP re. Key Learnings 4. Three-Column Journal of Observations Rubric 5. Three Cycles of Process Consultation 6. Paradoxes of Group Life Paper 7. Outcome Meeting Design Rubric 8. Multiple Problem Solving Sessions Rubric Arena 3 10 Outcomes 1. Case Study Initiative Guide 2. Reflection Assignment 3. Organization Theory Areas Literature Overview 4. Virtual Team Evaluation 5. For A Systems Overview of My Organization 6. MOU Agreement 7. Change Webliography 8. /Data Plan and Feedback Research 10 Outcomes 1. Grading Criteria for Annotated Bibliography 2. Grading Criteria for Personal Theory of Inquiry Assignment 3. Grading Criteria for Setting the Stage Assignment 4. Grading Criteria for PAR Literature Review 5. Grading Criteria for Action Research Report 6. Grading Criteria for CP Early Development Assignment 7. Statistical Modules Criteria 3

4 9. Leadership Development Plan 10. DISC Written Report 11. Transformational Leader-ship Case Study 12. Five Best Practices Model Goal Statement and Work book 13. Third Party Intervention Case Study 14. Diversity and Identity paper 15. Observing Active Listening Skills 9. Monograph Assignment Rubric 10. Paradoxes of Group Life Rubric 11. World Café Assignment Rubric 12. Final s of the Team Arena First Semester 13. Facilitation Coaching and Development Rubric 14. Utilizing Creative Problem Solving in Group Facilitation 15. Appreciative Inquiry Rubric 9. Social Change Initiative Capstone Assignment 10. Change Agent Code of Ethics 11. Reflection on Capstone 12. Futures Story 8. Statistics Project Fall Year 2 Criteria 9. Concept Paper Fall Year 2 Criteria 10. Spring Research Year 2 Project Criteria 11. Concept Paper Spring Year 2 Criteria Not only can I confirm that 53 assessments are made during the three years (exclusive of the dissertation), but I also would commend the faculty for attending to the need to provide reliable assessments for student progress. The presence of good rubrics and rating checklists contribute greatly to assessment reliability creating a set of reliable assessments through the eyes of the EDD faculty is essential. 3. Program Capacity This element was not reviewed via interviews or observation, other than to visit the facility in which the March seminar was being held for the combined Years 1, 2, and 3 groups. Though not a university owned building, it was a great set up for the work at hand. Three adjacent rooms housing the Year groups were used, with ease of movement among the three. No assessment was made of equipment, resources, staff, or support services. Nor were they discussed in the department s self study. One part of a survey administered to Year 3 students in March, 2012 (see next section for a description), addressed three elements intended to contribute to the overall quality of the program. These were Session facilities, Library services, and Learning groups. It can be concluded from the figures reported below that the Learning groups contributed greatly, and that the local session facilities and the library services were certainly not deterrents, contributing their fair share to the overall program s success. Contributed greatly to the quality of the EDD program Moderately contributed to the quality of the EDD program Had little effect on the quality of the EDD program N % N % N % Session facilities Library services Learning group

5 4. Overall Strength of the Program This aspect of the external review received the greatest amount of attention among the five cited for the review in the ULV External Reviewer Guide. Thirty-nine Year 3 students completed a form in March, 2012, designed to assess their perceptions of selected elements of the EDD program (see Appendix A). In one section of the survey, a four point scale was used to rate the quality of the leadership components, assignments, and projects in Years 1, 2, and 3. The figure below reports that of these 39 students, for all three years, 95 percent indicated that the leadership components were strong or exceptionally strong, and close to that same percent indicated that the assignments and projects shared the same strength and contribution. There can be no question regarding the quality that the students attributed to these elements of the program. Number/percent responding to the top two of four choices, from Exceptionally Strong to Minimal Contribution Year 1 Year 2 Year 3 N % N % N % Leadership component Leadership assignments/projects In addition, data were collected from these students regarding the extent to which the EDD program contributed to their personal development in 12 leadership skill areas. On a six point, Major Contribution to Little Contribution, the distribution of how students responded is shown below. In 9 of the 12 skills, at least 29 of the 39 students (over 70 percent) indicated that the program had made a major contribution (ratings of 6 or 5) to their repertoire of leadership skills. And while not all of the 12 received such high marks, observe that in the far right column, there were almost none that noted little contribution. Major contribution (6-5) Contribution (4-3) Little contribution (2-1) N % N % N % Communicating well with others Dealing with conflict skillfully Coaching others effectively Promoting teamwork Facilitating teams effectively Using appropriate information in decision making Communicating the vision of the organization Building a healthy organizational environment Establishing effective organizational systems Promoting continuous improvement in the organization Building positive relationships Using technology effectively

6 Three additional items in the survey cited some additional very important strengths of the program. Note the figures below. Opportunities to demonstrate and opportunities to integrate knowledge and learning were exceptionally strong for 80 percent of the respondents. The third, making a major contribution to the field, related to the dissertation research, which most of the students are still in process. Perhaps that accounts for fewer exceptionally strong ratings. Opportunity to demonstrate knowledge of leadership skills as they relate to personal/ interpersonal, team, and organization Opportunity to integrate learnings and to apply theory to practice Opportunity to make a major culminating contribution to the field Element is exceptionally strong Element mostly contributed to the overall quality Mixed or minimal contribution N % N % N % The final item on the survey asked for a summative overall rating of the EDD program s quality. A 4 point scale from exceptionally strong to minimally so was offered. Twenty-five of 38 students (one did not respond) or about two-thirds, marked a 4, Exceptionally Strong; the remaining 13 marked Strong. None of the 38 marked a 2 or 1. Having used scales like this for similar purposes throughout my lengthy career, I can state unequivocally, These results are remarkable! These data can be triangulated with similar kinds of statements from staff and the two students whom I interviewed to strengthen this conclusion. Additional evidence regarding the overall strength of the program, while perhaps not needed, is, nevertheless, impressive. Communitas is a Latin word that means people coming together for the good of the community. Communitas is a non-profit organization inspired by the vision of a better world and is an attempt to do something about it through volunteerism, investment, and ethical, sustainable, business practices. The University of La Verne s doctoral program in Organizational Leadership was recently given the Communitas award for Community Service and Social Change, one of three in the nation. And to close this section, Dr. Lou Obermeyer, an alumni of the ULV doctoral program, was awarded the Association of California School Administrators Superintendent of the Year Upon receiving this prestigious award, she stated: I owe so much of my success to the University of La Verne Doctoral Program in Organizational Leadership how lucky I ve been to be connected with such a forwardthinking program. 6

7 5. Action Recommendations Any organization that operates under the tenet of continuous improvement, rather than judgment, when presented with data and recommendations immediately understands two things: 1. Data and recommendations do not imply weaknesses in a program. They are considerations for improvement. Often recommendations represent relative weaknesses in a strong program. An example of what I mean would be rating five things on a poor to good scale from 1 to 10. We have four 8s, 9s, or 10s and one 6. While 6 does not represent a weakness, it is a relative weakness. 2. Recommendations can be made without full knowledge of a situation. Thus, if a recommendation is made, and it can be countered with additional information, no harm, no foul. Considering these, the following recommendations are made to the Department of Organizational Leadership s EDD program: Theory to Practice is a continuous message of the EDD program. However, the research sequence does not appear to connect the two as clearly as it might. Considering that all the doctoral students are working adults, most in very responsible positions, efficiency toward program completion should be an important criterion for quality, as well as effectiveness. Data from the Year 3 student surveys, as well as with the interviews with two high achieving students suggest that this sequence be studied. As one student stated, There is a disconnect between our research courses and the dissertation. Wouldn t it be possible to bring part of the dissertation process into Year 1 Research, and relate it to theory? Currently the theory part seems unrelated. It would sure save a lot of time and reduce our anxious moments. Number 4 in the report above, Overall Strengths of the Program, cited significant strengths in Years 1, 2, and 3 Leadership components and activities with percentages in the 80s and 90s. However, Research content for Year 1 received percentages indicating strength in the 50s, moving to the 60s for Year 2. Year 1 uses several self assessment and self reflective tools. When speaking with the students, most were given high ratings, but in some cases, statements like, Most are good, but some parts do seem to lack relevancy. A review of those tools would be in order. This program is successful, in large part because of the incredible commitment of its faculty. Considering all that the program is and aspires to become, of necessity, it is very labor intensive. Assistance in various forms is needed to maintain its level of excellence. A recommendation is that creative ways of assisting faculty be considered. Of course, everyone is aware that these days there are few funds anywhere, but if this program is to continue as it is, with a focus on continuous improvement, multiple resources to assist faculty and students 7

8 should be considered (e.g. graduate assistants; research assistants; funding sources for research). The following are recommendations that don t cover as broad a spectrum as those above, but are worth considering: Consider lessening the number of outcomes for Year 1. Currently there are 19. That seems a bit too many to focus on during a year, having given the structure of the program. Might the 19 be condensed into 9 or 10, for example? It may lessen the load a bit. This does not mean eliminating anything. It suggests repackaging. Take special care when developing assessment scales. This is especially significant, having given the importance attached to continuous assessment. If each point on the scale is to be labeled with words, consider semantically, the distance between the various descriptions. Some of the scales I observed lacked equal semantic distance (e.g. The Methods Review Form). Unequal semantic distance makes it difficult to extract meaning from the results. Scaling is not an easy thing to do. There are many issues to consider when selecting an appropriate scale. 8

9 Appendix A The ULV Doctoral Program: These Are My Thoughts and Feelings The purpose of this inquiry is to gain the perceptions of Year 3 ULV EdD students in Organizational Leadership. You will be asked to reflect on your time in the program and to offer your opinions and suggestions regarding selected elements. This activity is being conducted in the context of continuous improvement. Resulting data will be used to identify the relative strengths and areas to improve. Your honest opinions are very much appreciated. A summary of what you say will also provide information for one element of our program s WASC accreditation activity. Your confidentiality is assured. Thanks for helping. Part 1 Directions: Below are some products that you developed and/or used during your doctoral program. Rate the usefulness of the process of working with each of these pieces. Use the following three point scale, and place the number associated with the rating in the blanks provided. 3 = Very Useful 2= Useful to some degree 1 = Not very useful Tools to assess my performance as a leader (e.g. Leadership Practices Inventory by Kouzes and Pozner; Emotional Intelligence) The Leadership Development Plan Comprehensive Review portfolio Part 2 Directions: Below are 12 leadership skills that the doctoral program has focused upon in the program. What contribution has the doctoral program made to enhance your skill level. Using the following 6 point scale, place your rating, on the blanks provided: Major Contribution Contribution Little Contribution * * * * * * 1. Communicating well with others 2. Dealing with conflict skillfully 3. Coaching others effectively 4. Promoting teamwork 9

10 5. Facilitating teams effectively 6. Using appropriate information in decision making 7. Communicating the vision of the organization 8. Building a healthy organizational environment 9. Establishing effective organizational systems 10. Promoting continuous improvements in the organization 11. Building positive relationships 12. Using technology effectively Part 3 Directions: This section deals with research in relation to the dissertation process. Selected elements of the dissertation and writing are listed below. Given the two years of research courses you have taken, indicate your degree of knowledge re: your learning using the following scale: I m very I m somewhat I m less I lack knowledgeable knowledgeable knowledgeable knowledge * * * * 1. Writing a clear purpose statement 2. Writing a main research question 3. Logically connecting the problem statement to the purpose statement to the research questions 4. Articulating the significance of your study 6. Writing and synthesizing a literature review 7. Selecting the desired design (e.g. quantitative or qualitative, or mixed method 8. Understanding how to identify a population and sample 9. Understanding the components of a dissertation Part 4 Directions: This section deals with elements of a program that can support or deter the overall quality of any program. Consider each of the elements listed and indicate, in your judgment, whether that element 3-contributed greatly to the EdD program s overall quality, 2-moderately contributed to the overall quality, or 1-had relatively little effect on the overall quality. Element Session facilities Library services Learning Group 3-Greatly contributed to the overall quality of the program 2-Moderately contributed to the overall quality of the program 1-Had little effect on the overall quality of the program 10

11 Part 5 Directions: This section offers the opportunity to give your overall assessment of selected aspects of the doctoral program. Use the rubric as shown below and place the number associated with your rating in the blanks provided. 4 = This element is exceptionally strong. 3 = This element mostly contributed to the overall quality of the program. 2 = This element has a mixed contribution to the overall quality of the program. (While generally satisfactory, some parts can be strong while other parts are problematic.) 1 = This element only minimally contributes. 1. Leadership content of Year 1 2. Leadership assignments and Projects for Year 1 3. Research content of Year 1 4. Research assignments and projects for Year 1 5. Leadership content of Year 2 6. Leadership assignments and Projects for Year 2 7. Research content of Year 2 8. Research assignments and projects for Year 2 9. Leadership content of Year Leadership assignments and projects for Year The opportunity to demonstrate knowledge of leadership skills as they relate to personal/interpersonal, team, and organization 12. The opportunity to integrate learnings and to apply theory to practice 13. The opportunity to make a major culminating contribution to the field 14. Quality of the overall ULV doctoral program If you marked in the 1 or 2 columns, what suggestions do you have that would have resulted in a more positive response? Briefly respond in the space provided. Item 1 Item 2 Item 3 11

12 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 10 Item 11 Item 12 Item 13 Thanks so much for your time. Return your completed form as directed. Results of this activity will be communicated to you once they are known. 12