Northern Illinois University. Report to the University Assessment Panel Fall 2007

Transcription

1 Northern Illinois University Report to the University Assessment Panel Fall 2007 Program: EdD in Instructional Technology The Doctor of Education program in Instructional Technology prepares students to be leaders in research and the development of new knowledge and applications in the field. Program alumni are employed in positions of leadership in the use of technology in diverse settings including all levels of education, business and industry, the health care professions, government, and not-for-profit organizations in the U.S. and other countries. The doctoral program is not narrowly career-oriented, but rather focuses on the conceptual framework of the field. Building on the competencies of practice acquired at the master's degree level or from remediation of deficiencies, emphasis is placed on developing a unified grasp of the knowledge, skills, and theory inherent in the field, as the base for original research. Goals for students enrolled in the Ed.D. Program in Instructional Technology stress competence and originality. Students completing the program demonstrate a wide range of skills and abilities related to application of technology to learning. Whereas master's graduates can correctly follow and apply methodologies and models, doctoral graduates are able to develop theories and conduct research in appropriate methodologies. 1. Student Learning Objectives The following skills summarize the department s expectations of all graduates of the doctoral program in Instructional Technology: 1. Synthesize theory and research, and past experiences and training in instructional technology applications in the development of theoretical constructs for new or better applications. 2. Produce research in the applications or implications of instructional technology in educational contexts and society in general. 3. Demonstrate mastery of research design and techniques by engaging in research activities within courses and independently. 4. Demonstrate mastery of common educational theory, including human development and learning, human behavior, and the organization or administration of education. 5. Demonstrate the ability to test theory, research and ideas while developing an appreciation for diverse and multiple perspectives, by engaging in spirited dialogue with faculty, other graduate students, and instructional technology professionals. 6. Demonstrate knowledge and understanding of ethical guidelines for research and evaluation professionals. 7. Apply assessment techniques to measure student learning and the effectiveness of instructional technology-based products and programs. 8. Demonstrate effective oral and written communication skills through the creation of written papers and projects, and the presentation of individual and group works, as assessed by faculty and peers. 9. Complete the program within 7 years from matriculation, with due regard for unavoidable interruptions and program leaving.

2 2. Explanation of Methods Method Description Timeline Person Responsible Objectives Addressed Instructor and Course Evaluations Internship/ Practicum Performance Performance on Candidacy Examination Length of Time to Degree Graduation Rates All students in all courses will be surveyed for their opinions about the course and their instructor. Feedback from students, internship/practicum field supervisors, and supervising NIU faculty will be collected and analyzed. Students prepare written responses to a two part candidacy exam that is evaluated by a committee of at least three faculty members. This exam is conducted during the last two s of course work. Number of s from student matriculation to graduation. Ratio of the number of students who graduate over the total number of students, based on the of matriculation. in all courses Dept Chair; DPC Internship/ practicum coordinator(s) Semester ETT 643 and ETT 698 Course Instructors, IT faculty 1,2,3,4,5,6,7 1,5,6,7,8 1,2,3,4,5,6 9 9 Alumni Placement and Employer Questions specific to the doctoral degree in IT program will be asked of past students. Questions specific to job placement and the doctoral degree in IT program will be asked of employers and past students. Fall for one, five, and ten years after completion One year following notification of employment 5, 8 2, 3, 4, 5, 7 Professional Activities Information related to professional participation, such as membership in key organization, participation in professional conferences, writing for professional and scholarly journals. 5, 6, 8

3 Outcomes-by-Methods. Table demonstrating which outcomes are addressed by each method of assessment. Organize theory and research/ develop theoretical constructs Engage in and produce research Mastery of research design and techniques Mastery of common educational theory Engage dialogue with other IT professionals Instructor and Course Evaluations Internship and Practicum Performance Candidacy Exam Length of Time to Degree Graduation Rates Alumni Placement/ Employer Prof.Act. Ethics X Apply assessment techniques Effective oral and written communicatio X

4 n skills Timely completion of program Instructor and Course Evaluations Internship and Practicum Performance Candidacy Exam Length of Time to Degree X Graduation Rates X Alumni Placement/ Employer Prof.Act.

5 3. Evidence Instructor and course evaluations. Prior activities. Each course, regardless of format or delivery is evaluated by students using an instrument that was developed in Each the results are reviewed for each course in the EdD program by the instructors, Department Chair and Department Personnel Committee. Findings. Since implementation the instrument has been well received by students and faculty and no significant course issues were identified. The only exception is with courses that are delivered wholly or partially online. In these instances it is sometimes difficult to reach all students for input on the course evaluations. Also, there are some questions related to the learning environment and communication that simply do not apply. There are also facets of online learning that go unassessed. To address these issues a department committee comprised of faculty and the Department Chair are working to explore the adoption of an online evaluation system that would include evaluation measures related to online course delivery. Internship/Practicum performance. Prior activities. Beginning in Fall 2006 data for Internship/Practicum performance is collected and managed through the College LiveText data system. Since that time 1 in the EdD program in Instructional Technology have completed an internship or practicum with data submitted for aggregation in the LiveText system. This student was evaluated by their university supervisor with regard to meeting their established goals and presentation of a field experience portfolio. The field supervisor also submitted an evaluatio. Finally, the student completed a self evaluation. Findings. The data show that this student met set standards at the target or acceptable levels. Candidacy Exams. Prior Activities. There are two parts to the doctoral candidacy examinations. Part 1. serves as the Generals exam in which all candidates respond to the same six questions. Guided by a rubric, a committee of three faculty members scores the exam. LiveText is used to tabulate data for students performance on the exam and communicate results to the students. Part 2. of the examination is a draft of the dissertation proposal. These proposals are orally defended in front of a dissertation committee comprised of a chair and at least two senior graduate faculty members. Findings. Since Fall 2006, 45 students have completed part 1 of the candidacy exam. All but 8 students received passing scores on their first attempt. All but 1 student passed the exam on the second attempt. The student that did not pass was counseled to apply for the MSEd program. The student followed faculty advice and anticipates graduating from the MSEd program in December Three of the students that passed the exam, have defended and graduated. The remaining 41 students are at various stages of dissertation completion. Note that the high number of candidacy exams assessed are the result of an off-campus doctoral cohort that moved through the program of study at the same pace. Length of time to degree. Prior activities. In 2003 a new integrated data management system, SHARK, was developed and included systems to help manage student progress. Since 2003, all student information has been loaded into the database. The Department Chair, Program, and faculty are able to track individual students, their progress and program needs from the stage of initial inquiry as a prospective student, through all phases of the program, to degree completion. Findings. Based on SHARK data EdD students in Instructional Technology complete their degree program in an average of 6.33 years. Although a little longer than national norms, our graduation rates are good (see below) and we must recognize that the vast majority of our EdD students do not maintain full-time status. Most are working full-time and working toward their degree in a part-time capacity. Furthermore, many of our students enter the EdD program from undergraduate and graduate programs outside of instructional technology and in some cases outside of the field of education. These students are then required to add a series of 3-4 courses to their programs of study adding months to their length of time to degree. However, in the Fall of 2003, the program began its first off-campus doctoral cohort. As previous research on cohort models shows, completion rates are higher and length of time to degree is shorter. Of 19 students admitted, 3 are complete, and 2 more will graduate in the spring of That puts 25 percent of this cohort graduating well ahead of our average time to completion (about 3.5 years). Graduation Rates. Prior activities. In 2003 a new integrated data management system, SHARK, was developed and included systems to help manage student progress. Since 2003, all student information has

6 been loaded into the database. The Department Chair, Program, and faculty are able to track individual students, their progress and program needs from the stage of initial inquiry as a prospective student, through all phases of the program, to degree completion. Findings. Since we do not have easily accessible data for student admitted prior to 2003, it is premature to speculate on our graduation rate. However, as with our measures for length of time to degree in the EdD program, all indicators point to the trend that our graduation rates will follow a similar pattern, surpassing a 50% completion rate for on campus students. Alumni. Prior activities. University alumni survey. Findings. A very low response rate (2003 N=1, 2004 N=3, 2005 N=0) and a lack of systematic means to connect survey analysis with program improvements reduced the usefulness of this effort. A new survey instrument is in development and expected to be piloted by the fall of Placement/Employer. Prior activities. Informal data collection systems (e.g., exit interviews, continued contact with faculty advisors), together with alumni survey data, provided information on job placements for our graduates. These data, while rich in character and description, do not provide a sufficiently complete picture upon which we could determine numbers or rates of placements. In addition to the university alumni survey, employers of prior students in this program will be sent a departmentallygenerated survey asking questions regarding the quality of knowledge and skills exhibited by the employee. Findings. Our students consistently provide anecdotal reports of having a job upon completion of their program, typically because most were employed full time during their doctoral studies and were pursuing the degree for job advancement within their current employer. Very few students reported not being able to secure a position upon completion. A new exit survey is currently in development, to be piloted in the Fall of Together with a more formal alumni follow-up, we hope to be able to generate the kinds of numeric data that would support our descriptive findings. Professional Activities. Prior activities. Faculty have always encouraged membership in professional organizations such as the Association for Educational Communications and Technology (AECT), the Association for the Advancement of Computing in Education (AACE), and the International Society for Performance Improvement (ISPI). Faculty have also encouraged student involvement in professional conferences where they can help with proposals, suggest venues, critique papers or co-present with a faculty member. Findings. Exit/alumni surveys have not typically provided sufficient information concerning the degree to which doctoral students have engaged in these professional activities. New questions on the survey(s), as well as an expanded effort to collect this kind of data from students, and disseminate it (such as through department and college newsletters) needs to be expanded. A new exit survey is currently in development, to be piloted in the Fall of Use of Results Faculty in the ETRA department have been highly engaged in the development and uses of the assessment processes and data. The department personnel committee has been meeting to discuss how to integrate the new information obtained on our instructor and course evaluations, and bringing recommendations back to the faculty as a whole for their review and approval. The department curriculum committee, working with the various program areas, has been refining elements of the curriculum. Department staff have been hard at work implementing the department management information system, SHARK, and LiveText, in an effort to be able to provide more detailed information in a more rapid fashion. Smaller faculty ad hoc committees have been working to develop the surveys still not in use. All of this represents an intense amount of work effort to refine and improve this program. 5. Further Information Needed Using the information from our prior activities, several assessment methods should be revised and improved: Instructor and Course Evaluations. Develop instrument that can be delivered online and include questions relevant to course that include online components.

7 Internship/Practicum Performance. Continue to use LiveText to aggregate data. Candidacy Examinations. Continue to use LiveText to aggregate data. Length of Time to Degree. Continue to examine data from SHARK. Graduation Rates. Continue to examine data from SHARK. Alumni Follow-Up. Develop and implement new alumni exit survey. Placement/Employer Follow-Up. Develop and implement new exit and employer follow-up surveys. Professional Activities. Incorporate in exit survey and alumni follow-up. Improve methods for collecting and disseminating. 6. Timeline for Collecting Additional Information Instructor and Course Evaluations. Currently an ad hoc faculty committee in cooperation with the Department Chair is working on the investigating and developing an online course evaluation system. The aim is to have a new system piloted in spring/summer 2008 with full implementation in fall Internship/Practicum Performance. With the LiveText system in place, the department is currently collecting and analyzing data. Candidacy Examinations. With the LiveText system in place, the department is currently collecting and analyzing data. Length of Time to Degree. With the SHARK system in place, the department is currently collecting and analyzing data. Graduation Rates. With the SHARK system in place, the department is currently collecting and analyzing data. Alumni Follow-Up. The new alumni survey is currently in development. We plan for initial testing during the Summer of 2008, with revisions occurring during the Fall. Full implementation of the new survey is expected by the end of the Fall Placement/Employer Follow-Up. The new exit and employer survey is currently in development. We plan for initial testing during the Summer of 2008, with revisions occurring during the Fall. Full implementation of the new survey is expected by the end of the Fall Professional Activities. New questions are being generated to include on both the exit survey and alumni follow-up survey. 7. Resources Needed The department is currently covering all of the costs associated with the various assessment activities noted above. As the efforts under development are put into practice some additional resources may be needed; if so, those additional resources will be sought at that time.