Keith D. George. The University of Alabama BER 660. November 23, Dr. R. Schumacker

Transcription

1 CIPP Evaluation of Highland Rim School District Mobile Device Management Software Training Program Keith D. George The University of Alabama BER 660 November 23, 2013 Dr. R. Schumacker

2 Program Description The Information Services Department at Highland Rim County Schools implemented a new Mobile Device Management (MDM) System over the past two years. The implemented system is called Casper and is marketed as an all- in- one MDM solution for schools and businesses. Members of the District Information Systems department attended an on- site training providing by the vendor that lasted three days. The training consisted of the staff learning the installation and deployment procedures as well as the various features of the system. The goals of implementing the new system were to ease the process of managing the increasing number of mobile devices that had been placed in classrooms. The Information Systems department sought to streamline the installation and updating of apps, increase device security and tracking, and maintain accurate inventory of app installation. A select number of school- level personnel were subsequently trained on the software by Information Systems staff. This training, however, was conducted by internal staff and was considered insufficient by the school- level personnel because it was much more condensed and no follow- up was provided. In fact as additional issues arose, the Information Systems department actually revoked the administrative privileges of the school- level staff. Eventually the Information Systems department made the decision to discontinue the use of the software.

3 Purpose and Scope of the Evaluation The Casper software was implemented at great cost in both funding and time. Most of the functionality was never fully implemented. Additional around the time that the school- level personnel became comfortable with the software, the decision was made to discontinue its use. A new MDM program was to be purchased and implemented. This change of direction caused significant extra work on the part of the school- level personnel. It also appears to have discouraged these local staff members regarding the implementation of another system. An evaluation was requested to determine if additional training, an extended piloting period, or additional support could increase the probability of a successful implementation. More directly, the delays caused by school- level personnel s waiting for district- level staff to respond to what were conceivably simple fixes increased frustration, but also interfered with instructional time of students. If a device could not have easy and quick troubleshooting completed, the school staff were required to retain the device leaving that student without the device. The evaluation began with a survey to all school- level staff responsible for device management. The survey asked if delays longer than three days had occurred after requests for assistance related to the management of student or teacher devices. Of the twenty- six schools that responded, twenty- four of them, or ninety- two percent, had experienced extended delays. Of these respondents, eighty- six percent felt that after the solution was implemented they would have been able

4 to solve the issue if they had appropriate access and proper training. Several common tasks that should be able to be done at the school site proved to be above the knowledge level of school personnel. For instance, only twenty- seven percent of the school personnel reported that they could remotely distributed paid applications to the devices in their school. As the move was made to the new mobile device management (MDM) tool, which is called Airwatch, the decision was made to implement a training program that would result in more effective use of school- level personnel. The training began with school- level personnel attending a general knowledge training session presented by the District- level staff. The second step, implemented one- month later, consisted of school- level personnel attending an in- depth training conducted by the Airwatch training staff. During the one- month between the first and second steps, personnel were tasked with beginning the implementation of Airwatch at the school level to determine any issues, challenges and questions that they would need resolved. The Airwatch training would include significant time allotted to the solving of real issues. In the final step of the training process, school- level personnel would complete a written test to evaluate their knowledge of the Airwatch system. Evaluation Methodology The evaluator determined that the CIPP Evaluation Model developed by Stufflebeam (Fitzpatrick, Sanders & Worthen, 2012). This framework involves the evaluation of four distinct decisions faced by managers and administrators (Fitzpatrick, Sanders & Worthen, 2012, p. 174). These four decisions include

5 Context Evaluation (which addresses the needs and problems of the clients, in this case are the school- level personnel), Input Evaluation (which helps the administrators choose the best strategy for implementation), Process Evaluation (which examines the process to ensure the plan is be implemented as designed), and Product Evaluation (which determines whether the needs were met and if any changes or revisions are needed.) The survey that had been administered earlier contained many of the same questions that would later be included on a post- test. Although they had been presented in a Are you able to... format in the survey and would be presented in a more directly assessable manner on the post- test, the evaluator felt that this would provide enough of a measure of success that an additional pre- test was not necessary. Context Highland Rim School District is considered a larger district in this Southern region state. It includes some suburbs and the rural areas surrounding one of the five largest cities in the state and includes schools that are considered high achievement as well as several struggling schools. The district information systems staff consists of just nine people serving twenty- six schools and over 12,000 students. The district information systems staff designed a training program to introduce school- level staff to the Airwatch system and each school was required to send one representative. The training program was designed to address the concerns raised in the needs survey that had been administered earlier. The

6 training was presented at the Central Office Training Center. The Information Systems staff that had previously attended Airwatch training presented the session. Since these staff members were already onsite, no additional costs were related to this training, with the exception of substitute pay for six of the school- level personnel. Input Once the initial training had been completed and the school- level personnel had the ability to begin the basic management of devices, they returned to their schools to begin work. In the thirty days that had been allocated between the basic training and the advanced training, these school- level personnel worked to implement the software. They were instructed to work through as many issues as possible and log any problems that they could not work through. They were also encouraged to communicate with one another to solve problems. One week prior to the scheduled advanced training session, all unresolved issues were ed to the Airwatch training team for inclusion in the advanced training session. The advanced training session was once again held at the Central Office Training Center. The Airwatch team of two instructors was brought on site, as were the school- level personnel. The cost of this training was limited as the Airwatch trainers had been included in the software contract and the school- level personnel are already on contract. Again, a small number of substitutes were required for those school- level personnel whose duties required coverage. An ongoing learning community of the school- level personnel was created to give this group an avenue for collaboration and group problem solving. This online

7 community allowed for questions, strategies, suggestions, and tips to be shared easily. It also provided the District information systems staff a single place to offer solutions and information on changes and updates. Process The training program has been very well received thus far by both the school- level and district- level personnel. The district- level staff have reported significantly fewer help tickets related to mobile device management. The school- level staff reports much greater control over the devices, which allows them to more quickly resolve issues and get devices back into the hands of students. As help tickets are received by the district, there is a procedure to document the solution and share it through the professional learning community so that the resolution of each problem results in the entire group learning how to identify, prevent, or resolve the issue without input from the district- level staff. Product The results of the post- test indicated that the training program has been a success, particularly when combined with the reduction of help tickets related to mobile device management. By the end of the training program, over ninety- six percent of the participants correctly identified the procedure to remotely install paid applications onto devices in their school. The original survey reported only twenty- six percent of respondents indicated that they could accomplish this task without input from district staff. This alone resulted in the reduction of help tickets by twenty to thirty percent.

8 The post- test results also provided valuable information on training issues that needed to be addressed in future professional development sessions. These topics were also raised in the professional learning community to help address them quickly following the post- test administration. The post- test is still used to evaluate new school- level personnel even though a combined team consisting of district- and school- level personnel now conducts all of the training. Recommendations Based on the success of the Airwatch training program, it is recommended that the program continue as additional personnel are added to the staff. Additionally, it is recommended that because of the success of the program that the collaborative format that places school- and district- level staff as partners in the success of the program, rather than creating a feeling of Us versus Them, this format should be replicated in other information systems software and application rollouts. The collaborative nature of the Airwatch training programs indicates that additional efforts could greatly reduce help ticket volume, freeing up district staff for more intensive troubleshooting efforts, as well as allowing for greater and faster implementation of new innovations and programs. The program also has increased the abilities of schools to more effectively provide the greatest access to technology for the student, which is the true goal of educational technology efforts.

9 References Fitzpatrick, J., Snaders, J., & Worthen, B. (2012) Program Evaluation: Alternative Approaches and Practical Guidelines (4 th ed.). Boston: Pearson.