Report of External Evaluation and Review WordWave International Limited trading as Merrill Legal Solutions Confident in educational performance Not Yet Confident in capability in self-assessment Date of report: 10 October 2012
Contents Purpose of this Report... 3 Introduction... 3 1. TEO in context...3 2. Scope of external evaluation and review...4 3. Conduct of external evaluation and review...4 Summary of Results... 6 Findings... 8 Recommendations... 14 Appendix... 15 MoE Number: 7900 NZQA Reference: C07814 Date of EER visit: 13 July 2012 2
Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Name of TEO: Type: Location: Delivery sites: WordWave International Limited trading as Merrill Legal Solutions Private training establishment 2/16 Loftus Street, Christchurch Delivery is primarily by distance learning and administered from the Christchurch address First registered: 11 June 2009 Courses currently delivered Code of Practice signatory? Transcriber training for independent contractors of Merrill Corporation No Number of students: Domestic: ten to 20 at any time (approximately 40 transcribers in training during 2011) International: approximately eight of the 20 internationally based transcribers undertook training during 2011 Number of staff: Scope of active accreditation: Distinctive characteristics: One full-time and two part-time staff N/A Merrill Legal Solutions (MLS) is the trading name of the New Zealand-based transcription and training service for Merrill Corporation (an international provider of digital transcription services) and provides transcriber training to its independent contractors at no charge. Trainers, 3
team leaders, and learners are home-based and reside in New Zealand and internationally. The training is primarily delivered by distance learning through web-based modules. Learners progress through three transcriber levels (silver, gold, and platinum). At each level there are compulsory modules and optional client-specific style modules. Two face-to-face modules for trainer skills and quality control are also available. Recent significant changes: Previous quality assurance history: A new training and recruitment coordinator was appointed early in 2011. The NZQA quality assurance audit in 2009 found that MLS did not meet two requirements relating to governance and management. An NZQA validation visit in 2010 identified two issues not met relating to risk management planning and moderation, which have since been addressed. MLS is not required to participate in national external moderation. Other: N/A 2. Scope of external evaluation and review The scope of the external evaluation and review included Governance, management, and strategy Transcriber training The first focus area is mandatory. Transcriber training is the core programme. 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: http://www.nzqa.govt.nz/providers-partners/registration-andaccreditation/external-evaluation-and-review/policy-and-guidelines-eer/introduction. The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report. A team of two evaluators visited MLS on one day. Interviews were held with the general manager, training and quality manager, and training and recruitment 4
coordinator. Interviews were also held with three on-site and three off-site transcribers who have recently been, or are currently, in training. Telephone interviews were also conducted with two trainers. Responses to questions were received by email from two internal stakeholders from the United Kingdom office of Merrill Corporation. Documents, records, and other resources were also sighted and reviewed. 5
Summary of Results Statement of confidence on educational performance NZQA is Confident in the educational performance of WordWave International Limited trading as Merrill Legal Solutions. The quality of the transcriber training provided by MLS to its independent contractors is crucial to its commercial activity as an international transcription service for its parent company, Merrill Corporation. Evidence was available from Merrill Corporation staff and client feedback that the training is effective in ensuring that contractors who have completed the various levels of training (silver, gold, and platinum) acquire the appropriate skills and knowledge to meet the expectations of the commercial operation for very high standards of accuracy and compliance with style guides. MLS management closely monitors the needs of the business to ensure there are enough staff available who are trained in the required clientspecific styles and to the appropriate level. The approach taken to distance learning is effective in ensuring trainee success. MLS management has a good understanding of the attributes and skills of successful transcribers, which informs the stringent recruitment and selection processes. Careful monitoring of learner progress and support for trainees ensures a high level of retention and achievement. All but two of the 40 individuals who commenced training during 2011 and remained on contract successfully completed one or more of the training programmes. MLS trainers are also contracted transcribers and team leaders, who are expert in particular client-specific styles. While the training modules and materials are detailed and prescribed, the trainers regularly communicate with each learner via email, providing timely and useful feedback on their completed exercises. This is effective in fostering ongoing engagement of learners with the training programme. Each training exercise uses client material. In addition, the learning is consolidated by the allocation of work assignments to each contractor, depending on their skills and availability. Successful completion of the training provided by MLS is required of everyone seeking to contract to Merrill Corporation as a transcriber, regardless of previous experience, and the training provides a valued opportunity to work independently with flexible hours. Learners reported that the high standards required during the training prepared them well for their subsequent transcription work, which is exacting and challenging. 6
Statement of confidence on capability in self-assessment NZQA is Not Yet Confident in the capability in self-assessment of WordWave International Limited trading as Merrill Legal Solutions. Since early 2011 MLS has focused on improving the quality of the data available about learner achievement, and a comprehensive approach to self-assessment is being implemented. Indicators of educational and organisational performance have been clearly defined and are measurable. However, at the time of the evaluation visit there was limited evidence available of the impact of the self-assessment process on educational performance. While data is being collected and the progress of individual learners is closely monitored, MLS does not currently consolidate learner achievement information. An annual review is planned which will draw on a variety of sources of data, including learner, trainer, and client feedback and quality control statistics. In regard to learner support, MLS is a well-organised and responsive training provider with a strong commitment to learner achievement and with processes in place to closely monitor learner participation and provide ongoing feedback and support. Learner feedback is reviewed and shared with the trainers. A continuous improvement log has been introduced which tracks changes made in response to feedback. Examples of worthwhile improvements include a restructured training module which has contributed to improved learner retention and motivation. More recently, a major revision of complex training materials for one client-specific style has resulted in a single comprehensive manual, which will shortly be made operational. MLS systems for training and quality control are designed to ensure that client needs are met. This includes clearly defined processes for establishing new clientspecific styles and ensuring training materials remain current. However, the evaluation team found that currently only one client routinely provides written feedback on the quality of the transcription service which is a key outcome of the training delivered. Apart from annual client meetings, MLS generally relies on the absence of negative feedback to infer that their training delivery is reaching required standards. 7
Findings 1 1.1 How well do learners achieve? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Adequate. The MLS training programme was developed in-house and is reportedly unique in the industry; as such, there were no other providers identified for benchmarking of learner achievement. A careful selection process ensures that entrants to the MLS transcriber training programme have the core skills and attributes required for success. Trainees then acquire highly specialised skills and knowledge which are essential for their work as transcribers. Their progress through the training modules (each containing three exercises to be completed within six to eight weeks), and the various client-specific styles and levels of training (silver, gold, and platinum) are closely monitored by the trainers. All but two of the 40 individuals who commenced training over the last 12 months and remained on contract successfully completed one or more of the training programmes and were being allocated work. The body of evidence of learner achievement is currently quite limited. While the progress of individual learners is closely monitored, MLS does not consolidate learner achievement information. MLS staff reported that a new system of gathering data and tracking learners, implemented early in 2011, is proving effective in improving learner retention and success, although no comparative data was available. Individual progress, including completion of modules and the quality control ratings achieved, is reviewed at fortnightly management meetings. In response to evidence that students were stalling during one large training module, MLS restructured it into two smaller parts. Learners reportedly found this more manageable and were motivated by being eligible to undertake assignments after completing the first part. MLS has established standards for assessment and has implemented appropriate processes to ensure these standards are met. Training and assessment materials for each module (which incorporate client material) are reviewed by trainers before use. Each completed training exercise is assessed against a marker s version, according to a clearly defined quality control process which ranks errors. Standards have been established for accuracy (90 per cent) as well as style, speaker identification, and turnaround time (100 per cent in each case). The quarterly post-assessment moderation undertaken by the training and recruitment coordinator is focused on the quality of the feedback given to the students. 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 8
1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. At MLS there is a direct connection between the training programmes and employment for the learners. The learners quickly acquire useful skills, gaining access to work after successful completion of the initial module. Progression through the various client-styles and levels has a measurable value for learners as they qualify for a greater variety and number of work assignments and increased earnings. The training programmes are also essential to sustaining the transcription business, which has precise requirements in terms of accuracy and turnaround. The quality of the training has a direct relationship to the level (and cost) of quality control that is required to produce finished work. The evaluators found that the supply of trained staff generally meets operational demands consistently and well. Internal stakeholders from the United Kingdom office of Merrill Corporation confirmed that the outcome of MLS training was finished work of a very high quality which ensured that contractual requirements were met. The evaluation team noted that MLS generally relies on the absence of negative feedback rather than actively soliciting feedback from Merrill Corporation stakeholders or external clients on the value of the training. Ad hoc client feedback may be recorded in a spreadsheet, but follow-up actions are not currently tracked. However, one client, the Palace of Westminster, which does routinely provide written feedback on transcription assignments, has expressed a high level of satisfaction with the service. Management advised that a new workflow database will shortly be implemented which will request feedback for every piece of finished work, and this will serve to provide immediate and useful information for identifying improvements. 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. The training programme that has been developed at MLS is effective in ensuring that learners develop the specialised skills and knowledge necessary for them to be successful transcribers. In addition, the distance learning mode ensures that trainees develop the capacity to successfully manage the challenges of independent contracting. A rigorous recruitment process (which selects between 5 and 10 per cent of all applicants) ensures that entrants to the MLS transcriber 9
training programme have the core skills and attributes required for success, including high levels of motivation for studying and working independently. Independent individuals who are visual learners and competent with all aspects of written language are likely to be the most successful transcribers. The initial training covers the house style and builds familiarity with the quality control rating system used within the business. Once they have completed the first module on Listening and Reading, learners enter a probationary period and gain immediate access to work assignments. Learners reported that the high standards required during the training prepared them well for their subsequent transcription work, which is exacting and challenging. Some transcribers, depending on their personal circumstances and availability for work, choose to remain at silver or gold level. Motivated learners have clear pathways to upskilling and gaining access to a greater variety of work assignments and more consistent income. Client-specific style requirements are set out in manuals and associated training modules which are initially developed in consultation with the clients. Subsequently, transcription work is closely monitored, and client input is used to refine and improve the materials and to clarify aspects of style. Regular communication with the United Kingdom office of Merrill Corporation ensures that there is a close match between the training plans and methodologies and the needs of the business. 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Adequate. The rating for capability in self-assessment for this key evaluation question is Adequate. The approach to distance learning at MLS provides an appropriate level of information and support to learners and is effective in ensuring that most achieve the required standards. All trainees are provided with comprehensive online materials, including a trainee handbook and detailed training modules and style manuals which include examples of completed work. All communication between MLS and trainees is via email. Trainers are experienced transcribers with specialist expertise and currency in client-specific styles. Each trainee is also assigned to a team leader who provides ongoing support. Trainers are effective in building relationships and maintaining contact with trainees, which has been found to be very important in fostering their success. Learners are provided with timely and useful feedback on completed exercises. Errors are tracked and written feedback is provided, with reference to relevant sections of the manual. MLS monitors the progress of individual trainees by tracking their completion of the three exercises within each training module and the error rating for each piece of work. Learners who are achieving low grades, or not completing each exercise within two weeks, are followed up by the trainers, who offer additional support and encouragement. Client work is used in the training process 10
and the learning is consolidated by the allocation of appropriate work assignments to each trainee, depending on their skills and availability. The transcriber training programmes are tightly prescribed, and the assessment of each completed training exercise is based on the same quality control system that applies to actual client work. As a result, the main requirement of the trainers and the moderation activity, undertaken by the training and recruitment coordinator, is to ensure the consistency and usefulness of the feedback provided to learners. Providing constructive feedback is a focus of the train-the-trainer sessions completed by all trainers, and the central office has recently circulated guidelines to all trainers to reinforce the value of this activity. While learner feedback is collected after each training module, this is generally not shared with trainers until their annual performance appraisal, unless a problem has been identified. Several learners reported that the training materials are very detailed and could benefit from some simplification. Trainers and team leaders have appreciated occasional opportunities to gather together in person and share best practice; however, this has not occurred in recent years. 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. Trainee transcribers are generally self-managing and independent people who are comfortable with the distance learning model and home-based contracting. In addition to testing for core skills and attributes, the recruitment and selection process provides an opportunity to identify trainees objectives and motivations. This information is now being collected, and it is intended that it will be reviewed to identify patterns which can be used to improve any aspect of the recruitment or training process. Learners interviewed by the evaluation team reported that they felt well supported by MLS staff, who were available and responsive. While some learners would have valued the opportunity to meet or talk with their trainers or team leader, they were generally satisfied with the level of email communication and encouragement provided. At the end of each training module, learners are invited to provide feedback, which is reviewed by the training and recruitment coordinator. While a record of feedback is now being maintained, there are only a few examples of follow-up action available at this stage. 11
1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Adequate. Management is shared across the operational and training activities of MLS, which are fully integrated. While several workload and predictive tools are utilised to assess the future demand for trained staff, workload can be uneven and management reported that a tension can arise between training needs and meeting business requirements. However, this is mitigated by clustering the training during off-peak periods as far as possible. The evaluation team found that MLS engages in effective planning and has established clear business and training processes. While most of the communication with staff, trainees, and stakeholders is via email, this seemed to be very regular and was reportedly effective. Minutes of regular cross-functional meetings of Christchurch-based staff provide evidence of the close attention being paid to training standards and learner achievement. During 2011, MLS developed a comprehensive self-assessment process, which is now being implemented. This included the development of measurable indicators of educational and organisational performance and a focus on the collection of quality data. MLS does not currently consolidate and analyse learner achievement information or feedback collected from learners, trainers, or clients. An annual review is planned which will draw on these and other sources of data in order to assess performance against the key performance indicators. The evaluation team noted examples of worthwhile improvements but found that there was limited evidence available of the impact of the self-assessment process on educational performance. 12
Focus Areas This section reports significant findings in each focus area, not already covered in Part 1. 2.1 Focus area: Governance, management, and strategy The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Adequate. 2.2 Focus area: Transcriber training The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Adequate. 13
Recommendations There are no recommendations arising from the external evaluation and review, other than those expressed or implied within the report. 14
Appendix Regulatory basis for external evaluation and review Self-assessment and external evaluation and review are requirements of programme approval and accreditation (under sections 249 and 250 of the Education Act 1989) for all TEOs that are entitled to apply. The requirements are set through the Criteria for Approval and Accreditation of Programmes established by NZQA under section 253(1)(d) and (e) of the Act and published in the Gazette of 28 July 2011 at page 3207. These policies and criteria are deemed, by section 44 of the Education Amendment Act 2011, to be rules made under the new section 253. In addition, for registered private training establishments, the criteria and policies for their registration require self-assessment and external evaluation and review at an organisational level in addition to the individual programmes they own or provide. These criteria and policies are also deemed, by section 44 of the Education Amendment Act 2011, to be rules made under section 253. Section 233B(1) of the Act requires registered PTEs to comply with these rules. NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the EER process approved by the NZQA Board. The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website (www.nzqa.govt.nz). Information relevant to the external evaluation and review process, including the publication Policy and Guidelines for the Conduct of External Evaluation and Review, is available at: http://www.nzqa.govt.nz/providers-partners/registration-and-accreditation/externalevaluation-and-review/policy-and-guidelines-eer/introduction/ 15
NZQA Ph 0800 697 296 E qaadmin@nzqa.govt.nz www.nzqa.govt.nz 16