SINGAPORE WORKFORCE SKILLS QUALIFICATIONS SYSTEM (WSQ) GUIDE TO CONTINUOUS IMPROVEMENT REVIEW (CIR)

Transcription

1 SINGAPORE WORKFORCE SKILLS QUALIFICATIONS SYSTEM (WSQ) GUIDE TO CONTINUOUS IMPROVEMENT REVIEW (CIR) WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 1

2 Contents at a Glance Introduction Content Pg Part I: Overview of Quality Assurance Framework 4 Part II: Continuous Improvement Review (CIR) 8 Essential Elements of Review Part III: Roles and Responsibilities of Approved Training Organisation 17 (ATO) Annex A : Steps-by-Steps Process for ATO 23 Annex B : Self-assessment Checklist 28 Annex C : 2 Year Plan 35 Annex D: CIR Presentation by ATO 37 Annex E : CIR Outcome Measures 49 WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 2

3 Introduction The field of adult training is about providing quality courses to cater to the learning needs of the adult. A quality course would be able to enhance the learning experience of the adult and contribute to the adult in more ways than we can ever imagine. Some examples of the contribution would be 1) increase employability, 2) improve skills set, and 3) innovate work processes. However, as an Approved Training Organisation (ATO), providing quality courses are just one of the many responsibilities you are expected to do. It is really a case of simple logic. To provide quality output i.e. quality courses, you need to have quality input. What is quality input you might ask? To answer the question, we strongly recommend you to explore this guide. We assure you that it will be a meaningful learning journey, and you will be able to take away more than you can imagine. The bottom-line is that this guide serves more than just preparing your organisation for the Continuous Improvement Review (CIR). It aims to guide your organisation in establishing a strong quality Continuous Education and Training (CET) infrastructure system within your organisation. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 3

4 Part I Overview of Quality Assurance Framework In this part Part 1 deals with the WSQ quality assurance framework and the components existing within the framework. On the application side, this part introduces the actions required by any organisation embarking on the journey of becoming a WSQ Approved Training Organisation (ATO). It is a brief introduction of the different processes involved. The objective of Part 1 is to allow you to understand the journey you have taken to becoming an ATO and examine the processes that you would have gone through before the CIR. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 4

5 Part I Overview of Quality Assurance Framework In This Part Introduction to the Quality Assurance Framework What it means to you The Process within the Framework The Concept of WSQ That You Just Have to Know If you are a WSQ ATO, the term WSQ would not be unfamiliar to you. From the moment you have chosen to be a WSQ ATO, you should have a basic understanding of WSQ. However, if it happens to slip your mind at the point you are reading this guide, we are providing you with a quick note to refresh your memory. The Singapore Workforce Skills Qualifications (WSQ) is a robust and integrated continuing education and training system. Designed to build industry competencies, it is founded on international best practices and validated by industries and employers. WSQ caters to adult workers who have widely diverse training needs; it offers a wide range of certification and qualification ranging from Certificate to Graduate Diploma. Quality assurance provides the backbone for the WSQ. A strong quality assurance framework is important to build up public confidence in the WSQ. So as a provider under the WSQ framework, you are recognised by the public to be providing courses that are based on a robust quality CET system. It is thus a joint responsibility between yourself and WDA to continuously contribute and improve on the robustness and quality of the CET system. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 5

6 Mechanism to contribute and improve WSQ quality WDA has developed the quality assurance framework as a mechanism to uphold the quality of WSQ. In addition, this framework serves to continuously improve the quality of WSQ. The quality assurance framework is made up of two processes: i) Pre-delivery Approval There are 2 stages of accreditation under the pre-delivery approval, 1) the organisation approval accreditation, and 2) the course accreditation. The objectives of both accreditations are similar; that is to ensure that organisation has the capability to delivery courses that satisfy the competency requirements of the WSQ and is accessible to workers through appropriate delivery modes and resources. The process also ensures that the course will be delivered by a quality training organisation using qualified adult educators. Upon approval, the following status will be conferred:- (a) An organisation approved to deliver stipulated training and/or assessment services under the WSQ is known as an Approved Training Organisation (ATO); Henceforth the term Approved Training Organisations or ATOs will be used to refer to WSQ accredited organisations. At this stage, we would assume that you have gone through the above mentioned process and the 2 stages within the process. If you have not, we recommend you to visit this link ForTrainingProviders/L325A-001TP-ATO.html, so as to better understand the details behind the accreditation. However, do not stop your reading of this guide. It is still relevant for you to understand the types of mechanism WDA is adopting to ensure the quality of the WSQ frameworks. Moving on, the second process under the quality framework is none other than the theme of this guide Continuous Improvement Review (CIR). ii) Continuous Improvement Review (CIR) Upon approval (organisation and course), every ATO is to maintain and upkeep the standard of delivery in training and assessment. In addition, every ATO is to pursue continuous improvement to uplift their quality of design and delivery. WDA will periodically validate the internal quality assurance system and capability of each ATO under CIR. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 6

7 It is important to note you are not taking this journey alone. Throughout the journey, WDA will assist ATO to develop their capabilities in designing and delivering of quality training and assessment under WSQ. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 7

8 Part II Continuous Improvement Review (CIR) Essential Elements of Review In this part Here is where you learn about the essential criteria of review under the Continuous Improvement Review (CIR) framework. We will begin by explain the purpose of the CIR. From there, we will look in the areas which we will review during the CIR. Along the way, you will find out about the specific components that we are evaluating under each of the essential criteria. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 8

9 Part II Continuous Improvement Review (CIR) Essential Elements of Review In This Part The Purpose of CIR The Essential Elements of Review Continuous Improvement Review Continuous Improvement Review is one of the extremely important processes in the quality assurance framework that WDA puts in place to maintain and improve the quality of WSQ. CIR enables WDA to discern the gaps in quality pertaining to WSQ. As an ATO, you could use CIR as a quality check tool to improve your internal CET structure. It is thus valuable to understand the criteria in the CIR system so as to maximise the usage of it to improve your organisation s capability and quality as an ATO. Why CIR? The main objectives of CIR are: i) Inculcate the practice of seeking continuous improvement in the delivery of training and assessment; ii) Identify best practices and role models amongst the ATOs; and iii) Nurture ATOs to be WDA s partners in providing quality continuing education and training. Therefore, CIR is really a process for both WDA and the ATO to critically reflect on the practices adopted by the ATO. Such reflections provide opportunities for the ATO to refine its CET system, and thus contributing to the overall improvement of WSQ. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 9

10 Essential Criteria of Review All ATOs are subjected to audit on the following essential criteria for CIR: 1. Course Quality Assurance System i) System to ensure courseware and assessment are aligned to WSQ, pedagogically sound and contextualised ii) System to manage profiling of learners to ensure that the course(s) is fit for purpose iii) System to manage training activities iv) System to manage assessment activities 2. Plan-Do-Check-Act (PDCA) System i) System to ensure PDCA is implemented regularly for the comprehensiveness of training, assessment and management system 3. Outcome Evaluation System i) System to monitor and analyse L1 outcomes ii) System to monitor and analyse L2 outcomes iii) System to monitor and analyse L3 outcomes iv) System to monitor and analyse L4 outcomes 4. Adult Educator Management System i) HR system to ensure most qualified adult educators are recruited and inducted ii) HR system to monitor, appraise and manage performance of adult educators iii) HR system to train and develop adult educators 5. WSQ Related Administrative System i) Learner support system ii) System to ensure accurate and timely submission of assessment results WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 10

11 iii) Prior to 3 November 2014, WSQ certificate printing and distribution system. Post 3 November 2014, to ensure availability of WSQ e-cert and its distribution (if applicable) iv) Maintain up to date profile in Skillsconnect 6. Viability of Organisation i) Governance and management of organisation ii) Financial health of organisation 7. Free from Breach of Terms and Conditions i) Records of organisation free from breach of WDA terms and conditions. System-Evidence-Outcome Approach WDA is adopting the system-evidence-outcome (SEO) approach in the evaluation of the CIR criteria. The focus on the SEO approach is because there is strong correlation between sound systems and the quality of outcomes. It has to be reiterated here that the continuation of your approval status/es is/are subjected to the outcome of WDA s scheduled CIR audit which would be graded, based on the scoring matrices for S-E-O. Definition of S-E-O 1. Systems: a set of interacting or interdependent entities forming an integrated whole, giving a macro perspective. Components within the system generally have interconnectivity i.e. functional and structural relationships. 2. Evidences: a thing or things helpful in forming a conclusion or judgment:, for instance, the broken window was evidence that a burglary had taken place. 3. Outcomes: refers to the way something turns out; result; consequence. To ensure that you are on the same page as us, we have provided an example of S- E-O institute for courseware development. (Please see pg. 13 for the given example) You have to note that in the given example, the system is represented by a process flow chart. You can represent your system in many forms e.g. a standard operating manual, a policy guide etc. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 11

12 The key in developing a comprehensive system is how concise and systematic the information is presented. To achieve this, you first have to set your organisation SMART (Specific, Measurable, Agreed upon, Realistic and Time-based) targets when drafting any system at the outset. Succinctly, each system devised by you should address the following 5 questions: i) When would the system be devised and implemented; i.e., effective start date of system, frequency of system implementation. ii) Who would be involved in the system; i.e., the personnel who is directly involved in either the overseeing or implementing of the system iii) What approach/tools would the system used; i.e., the specific approach/tools that the system would be using in its course of implementation. iv) Why did ATO have this system; i.e., the rationale/objective of the system. v) How would each process within the system be evaluated and follow up on; i.e., any necessary follow up actions/evaluation are to be addressed accordingly in the system. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 12

13 Example of S-E-O System: Process flow for Courseware Development Courseware Development Team * Conduct research and survey the market and customers needs Industry Information, Market Research, Course Evaluation Forms, Companies Feedback Perform thorough study of the course objectives and assess capabilities such as trainer s skills, facilities, equipment (Refer Note 1) Perform simple costing to evaluate if the course is profitable Costing Profitable No Case Closed Yes Courseware Developer (Refer Note 2) Develop the Course Programme Design and Overview, Trainers Guide, Lesson Plan and Course Materials as per WSQ standards (1 st draft to be submitted within 1 month to the Courseware Development Team) WSQ Programme Design and Overview, Trainers Guide, Lesson Plan, Course Materials Courseware Development Team Submit to Courseware Development Team for approval * Team consists of Principal, Training Manager and Trainers Approved by Team (See Note 3)? No Course Developer to rework based on Team s directions Yes A Continue on pg 2 of 2 WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 13

14 A Training Manager Brief the appropriate trainers on the training concept before trainers proceed to conduct the new course. Trainers Proceed to conduct the course Perform design validation by analysing course evaluation forms / interviewing trainees and conducting post-course profiling for 3 sessions to gather participants profiles/feedback on the new course Course Evaluation Form (Trainee) Form 1 Course Evaluation Form (Trainer) Form 2 Post-Course Profiling Form 3 Changes in course design? Yes Follow up by Courseware Developer to amend and update the relevant documents and affected staff accordingly No Proceed with next course Note 1 For any deficiencies of trainer s skills, suitable trainers shall be sent for training or training could be outsourced to qualified parttime trainers. Training facilities shall be properly planned. Any new training equipment/products required can be bought, pending the costing calculation. Note 2 New course developers will have to attend pre-development briefing conducted by the Course Development Team. The minutes of the briefing and the attendance must be recorded and compiled for future reference. Note 3 For WSQ programmes, any new courses or revisions done to training materials must be submitted to WDA for approval WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 14

15 It is always useful to enhance each flowchart and all the supporting form/s with document names, document numbers, effective dates and version control tracking mechanisms. As the scoring for the CIR criteria is based on 3 components system, evidence and outcome, having a comprehensive flowchart means that the ATO would score for system. The Evidence in this specific example would be the followings: - Industry Information, Market Research, Course Evaluation Forms, Companies Feedback, etc. - Course Evaluation Form (Trainee) Form 1, Course Evaluation Form (Trainer) Form 2, Post-Course Profiling Form 3, etc. The Outcomes in this specific example would be the followings: - Good L1 scores as trainees find the course relevant; - Repeat corporate clients being served (as the organizations know what they are getting if they are requesting for the same training the second time). Please note that the under this specific example, the ATO has to devise Forms 1, 2 and 3 as well as document the Industry Information, Market Research, Companies Feedback, etc as evidence of deployment of the system so as to score points under the evidence matrix. Likewise, outcome/s would have to be presented as well-collated L1 scores by course or trainer on a periodic (e.g., quarterly, half-yearly, etc) basis. While the list of clients served, especially established corporate clients or walk-ins through words of mouth, etc if available would be useful in securing points in the outcome matrix. In addition, it is important to note that there should be clear and concise documentation for all the criteria under the S-E-O. Any S-E-O without documentation will affect the CIR evaluation outcome. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 15

16 You have come to the end of Part II of this guide. At this point, you should have a general idea of the seven essential criteria of review in the CIR: i) Course Quality Assurance System ii) Plan-Do-Check-Act (PDCA) System iii) Outcome Evaluation System iv) Adult Educator Management System v) WSQ Related Administrative System vi) Viability of Organisation vii) Free from Breach of Terms and Conditions You should also have a general understanding of the S-E-0 approach adopted by WDA in the evaluation of the essential criteria. However, if you still have doubts in your mind, you can always or drop a call to the officer-in-charge of your organisation to seek their advice. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 16

17 Part III Roles and Responsibilities of ATO In this part In the third part of the guide, we will be talking about the preparation requires by an ATO/ATC/AAC when being notified of a CIR by WDA. We will talk about the things that you are required to prepare after you have received the CIR notification from WDA, and the things that you are required to do consistently as an ATO. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 17

18 Part III Roles and Responsibilities of an ATO In This Part The Roles and Responsibilities of an ATO Scope of CIR ATO Continuous Improvement Efforts Outcomes of CIR Fees and Feedback The Roles and Responsibilities Together We Play a Part It is important to note that you do not journey alone when you are an ATO. WDA will provide the necessary assistance to aid you along the way. Thus, before the CIR, during the CIR, and even after the CIR, WDA seeks to work closely with you to improve your overall quality as an ATO. Some of the aids that WDA would provide to you in building your quality are: i) Validate the internal quality assurance of ATOs ii) Build up the competencies of the ATOs through feedback iii) Issue CIR audit report for CIR audit conducted iv) Highlight ATO s strengths and areas for improvement v) Facilitate ATO s participation in capability development initiatives and workshops However, as an ATO, you need to have the correct attitude in building up your own quality. As the saying goes, Your Attitude Determines Your Altitude, you would have to put in effort to improve your own ability and quality as an ATO. Some of the things you are recommended to do constantly are: i) Practice internal quality assurance WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 18

19 ii) Ensure Plan-Do-Check-Action (PDCA) system is in place and is diligently making continuous improvements to meet the required quality or for better quality, especially in areas where required by WDA from previous CIR audits conducted iii) Ensure quality of accredited courseware and adult educators are consistent with actual training and/or assessment iv) Ensure evidences of all systems implementation, and dates of training and assessment are available for audit when requested You have learned about the criteria of review and the S-E-O that WDA used in CIR evaluation. It is time to look into the activities that you are required to do on the day of the CIR. Scope of CIR In part II of the guide, we have gone through the essential criteria of review, and the expected responsibilities of an ATO. We will now move on to look into the specific things you will be required to do when you received the CIR notification from WDA and on the actual day of the CIR. When you received the CIR notification from WDA (one month before the CIR date), you are required to response to the notification within 5 workings days providing the following: i) A series of available dates and time for the CIR ii) A training and/or assessment schedules iii) A self assessment checklist (see Annex B)* iv) A 2 year plan (see Annex C) (Applicable to only ATO offering public courses)* *This is applicable only for public ATOs and is to be provided within 2 weeks from the date of the notification Upon receiving your response with the attached documents, the officer-in-charge of your CIR will then get in touch with you to finalise on the CIR date and time based on the series of dates and time provided by you. The officer may also request more information based on the 2 attached documents that you are required to submit. Once the date and time of the CIR is finalised, you can look forward to the actual CIR date and prepare the necessary presentation and materials for the CIR. According to the stated essential criteria in part II, you will be required to make a presentation based on each criteria. The presentation is recommended to be kept under 15mins (see Annex D for the breakdown of the presentation process). You need to have the supporting evidences ready for verification during the CIR audit. The review covers the following: WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 19

20 i) Showcase of all the implemented systems based on the essential criteria of review e.g. relevant SOP manual ii) Specific evidence of each implemented system, e.g. training observation checklist with requirements clearly communicated to all adult educators, or learners profiling checklists in determining contextualisation of courseware, etc. iii) Outcome of each implemented system, e.g. systematic documentation of all CVs and qualifications of qualified adult educators (against declaration submitted at accreditation), or assessment results with analysis to reflect L1 and L2 outcomes, etc. It is important that you prepare the relevant evidences and supporting documents in advance before your scheduled CIR audit by WDA. The reason of the preparation is really quite simple - you do not want to be running around getting the documents and indirectly reflecting the possibility of an inadequacy in your administrative system. Your Continuous Improvement Efforts Specifically, ATO is required to practice continuous improvement in its ongoing delivery of training and assessment under WSQ. Evidence of these efforts should be available or demonstrated in the form of PDCA through any of the following: i) Quality of design and delivery of courseware, e.g. regular reviews to ensure currency, for continuous improvement, contextualisation to be relevant to a particular industry or target audience etc; ii) Quality of trainers, assessors and curriculum developers, e.g. training and development plan and implementation, performance management system etc.; iii) Assessment Management System, e.g. regular reviews and improvements made to internal quality assurance system particularly for the management and conduct of assessments, ranging from how assessments are designed and conducted to ensure the four principles of assessments reliability, fairness, validity and flexibility as well as positive and negative features experienced in conducting the assessments, and refinements made, assessment validation etc.; iv) Administrative System (supporting training and assessments for accurate and timely records submission to WDA, certificate issuance WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 20

21 (prior to 3 November 2014) and ensuring availability of WSQ e-cert (post 3 November 2014) to trainees); v) Tracking of outcomes, e.g. end-of-course evaluation, attrition rate, passing rate, surveys conducted etc. You can refer to Annex A for the step-by-step of CIR for ATO to have an overview of the CIR processes. Outcomes of CIR Possible Capability Development Initiatives From the CIR pilots conducted, WDA has collated some common areas of improvement arising from findings of the ATOs piloted. These are taken into consideration for capability development initiatives that are being planned. Some of the initiatives in the pipeline include the conduct of regular seminars to equip ATOs with the knowledge to professionalise internal practices, which can be in the form of monthly briefings, one-to-one consultations, or sharing of best practices and outcomes of CIR in groups. ATOs can also look forward to the development of a guidebook to assist them to set up a training system and implement continuous improvement practices. The CIR will further provide information and evidence that will enable ATOs and WDA to identify areas of strength as well as areas in need of focused attention for planned improvement in the short, medium and long term. With these needs identified, WDA can then design suitable interventions to assist the ATOs. Continuation of Approval Statuses In addition to capability development, the CIR replaces the need for application by ATO for renewal or extension of WSQ statuses, i.e. the CIR determines if the approval status of the ATO should be continued. The continuation of approval statuses underpinned by the results of CIR is guided by the CIR Outcome Measures. Other considerations determining the continuation of approval statuses besides the periodic CIR are also guided by the CIR Outcome Measures. In the event that the ATO's WSQ status is not renewed arising from the results of CIR, the ATO can appeal to WDA on the outcome of the CIR. The ATO need to WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 21

22 submit an appeal letter to WDA, Deputy Director (Training Partners Development Division) stating the background and nature of the appeal together with the reasons and supporting evidence as to why the original outcome decision should be reversed or with a non-refundable fee of $ (inclusive of GST) within 2 weeks from the revoke/termination notification letter issued by WDA. The complete CIR Outcome Measures is represented in Annex E. Fees The CIR fees payable are as follows. S/No. Type of Application Period Charges (Inclusive of GST) 1. Continuous Improvement Review (CIR)* Effective from 1 Oct 2014 $1, * Note: With reference from CIR first activity date Feedback WDA welcomes any feedback to seek continuous improvement and fine tune the CIR system. Please send your comments to Ms Alice Chua at [email protected]. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 22

23 Annex A: Step-by-Step Process of CIR for ATO Step Action Needed Remarks Documentary Evidence Timeline Step 1: Receipt CIR Notice of ATOs are scheduled for CIR and notified by WDA through 1 month before the CIR According to CIR schedule Upon receipt of WDA s CIR notification, ATO is to respond to WDA with the completed self-assessment checklist (Annex B) and a 2 year plan (Annex C) (Applicable to only ATO offering public courses) ATO is also to indicate a series of available dates and time in the response to the CIR notification. WDA Officer will evaluate the ATO s readiness for the CIR WDA Officer advises on allocated documentation submission schedule and/or range of duration for possible site audit and possible training/assessment observation sessions. Training and assessment schedules ATO is to respond to WDA within 5 working days upon receipt of CIR notification For areas which ATO does not have a system or not implemented yet, ATO is to submit an action plan providing details on: WDA Officer will finalise the CIR date and time with the ATO based on the provided series of dates and time from the ATO WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 23

24 Step Action Needed Remarks Documentary Evidence Timeline a) Gaps b) Follow-up actions to address gaps c) Time needed to implement changes. ATO is to provide training/assessment schedules and anticipate a period (and not a stipulated date and time) when the site visit would possibly be conducted Step 2: Preparing for CIR Audit ATO to prepare a 15-min presentation outline and coverage advised in Annex D ATO to get ready required evidences and supporting documents and when informed about the site audit, ATO to avail facilities for site audit. WDA-TPDD will go down to ATO s training site to conduct CIR site audit. Standard CIR checks to include : 1. Course Quality Assurance System 2. Plan-Do-Check-Act (PDCA) System 3. Outcome Evaluation System 4. Adult Educator Management System WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page SOP manual/flow chart etc, of the various systems 2. CVs and evidence showing qualifications of trainers, assessors and curriculum developers declared 3. Process of review by ATO to ensure currency of courseware, relevance to industry, target audience and contextualisation 4. Administrative System

25 Step Action Needed Remarks Documentary Evidence Timeline 5. WSQ Administrative Related System 6. Viability of Organisation 7. Free from Breach of Terms and Conditions 8. Evidence-of- Implementation of the Various System 9. Outcome(s) of the Various System WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 25 Show evidences of accurate and timely assessment submission,certification issuance processes (prior to 3 November 2014) and ensuring the availability of WSQ e-cert to the learners (post 3 November 2014). 5. Outcomes tracking, e.g. attrition rate, passing rate, surveys conducted etc 6. All evidences of implementation of the various systems 7. Documentation of outcomes of all the various systems Supplementary evidence sources or supporting documents which may be requested for the CIR audit are: 1. Course files (course materials submitted at point of course application,

26 Step 3: Step Action Needed Remarks Documentary Evidence Timeline During site audit 15 minutes presentation by ATO Site audit and MR interview Conduct audit of training premises (if applicable) The management representative should be present during the site audit includes Assessment Plan, Slides, Participants Handouts and Trainer Guide) 2. Course attendance records 3. Training and assessment records ATO s slides and handouts for presentation. All SOPs related to the implemented systems The site audit would normally not take more than half a day Step 4 : Training/ assessment Observation Step 5 : After the audit ATO to avail training/assessment sessions for WDA Officer s observation Go through areas (strengths and weaknesses) highlighted by WDA Officer WDA Officer will brief provider and discuss follow-up actions if necessary WDA Officer to WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 26

27 Step Action Needed Remarks Documentary Evidence Timeline prepare CIR report Step 6 : Receive CIR audit report and outcome letter Study findings in the report. Take note of rectifications and timeline in CIR outcome letter and take corrective actions CIR outcome measures (shown in Annex E) will apply where ATO has cleared, taken action or failed to take corrective actions within stipulated time/does not respond * Timeline for CIR cases is 50 working days from the date of the last CIR activity. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 27

28 Annex B: Self-assessment Checklist Name of ATO: Name of Management Representative (MR): Contact Details of MR: ( ) Proposed CIR Date: (Tel No) (O) (Hp) Section A Are you able to show evidence for all the items in the checklist? If your answers are all Evident in Section A, your organisation is likely to be ready for a CIR audit. Please note that the list of evidence suggested for each area is NOT EXHAUSTIVE. Area of Review 1 Course Quality Assurance System Criteria Interpretation of Criteria Implemented System Evidence of System Implementation Outcome of System Implementation Evident (E) / Not Evident (NE) 1.1) System to ensure courseware and assessment are aligned to WSQ, pedagogically sound and contextualised System in place to manage developers Process in place to conduct internal evaluation and validation of courseware to ensure alignment to WSQ standards WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 28

29 System in place to facilitate contextualisation of courseware to ensure its currency and relevancy, etc 1.2) System to manage profiling of learners to ensure that the course(s) is fit for purpose Process in place to ensure systematic screening/profiling of learners, e.g. did the learner meet the prerequisite of the course: 1. literacy level 2. numeracy level 3. required assumed knowledge/skills, etc Process in place ensure to only learners with the appropriate aptitude/attitude are enrolled into the course 1.3) System to manage training activities 1.4) System to manage assessment activities System in place to continuously monitor and manage the training processes to ensure: 1. it is in accordance to industry practices i.e. ensuring quality and currency of training System in place to continuously monitor and validate the assessment processes to ensure it is in accordance to principles of assessment i.e., valid, reliable, fair, and flexible, and rules of evidence WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 29

30 Area of Review 2 - Plan-Do-Check-Act (PDCA) System Criteria Interpretation of Criteria Implemented System Evidence of System Implementation Evident (E) / Not Evident (NE) Outcome of System Implementation 2.1) System to ensure PDCA is implemented regularly for the comprehensiveness of training, assessment and management system Demonstration of establishment in PDCA. This may include: 1. Organization management 2. Training management 3. Assessment management Area of Review 3 Outcome Evaluation System Criteria Interpretation of Criteria Implemented System Evidence of System Implementation Outcome of System Implementation Evident (E) / Not Evident (NE) 3.1) System to monitor and analyse L1 outcomes Clear articulation of indicators, this is to include trainee satisfaction rate. Trainee satisfaction rate minimally on these 4 areas: 1. Training Programme 2. Course Material 3. Trainers 4. Facilities and Logistical support Analysis of obtained data Follow up action(s) on analyzed data WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 30

31 3.2) System to monitor and analyse L2 outcomes 3.3) System to monitor and analyse L3 outcomes Clear articulation of indicators, this is to include 1. Passing rate 2. Attrition rate Analysis of obtained data Follow up action(s) on analyzed data Clear articulation of indicators on impact of transfer of learning, this is to include: 1. Extent of candidate s behaviour and/ or capability improvement as indicated by candidate 2. Extent of candidate s behaviour and/or capability improvement as indicated by employer Analysis of obtained data Follow up action(s)on analyzed data 3.4) System to monitor and analyse L4 outcomes Clear articulation of indicators on impact of training on business, which may include: 1. volumes, 2. values, 3. percentages, 4. timescales, 5. return on investment, 6. other quantifiable aspects of organisational performance, for instance; numbers of complaints, staff turnover, wastage, noncompliance, quality ratings, accident ratings, zero-defect ratings, growth, retention, etc. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 31

32 Analysis of obtained data Follow up action(s)on analyzed data Area of Review 4 Adult Educator Management System Criteria Interpretation of Criteria Implemented System Evidence of System Implementation Outcome of System Implementation Evident (E) / Not Evident (NE) 4.1) HR system to ensure most qualified adult educators are recruited and inducted System in place to recruit and induct qualified adult educators 4.2) HR system to monitor, appraise and manage performance of adult educators System in place to manage the performance of adult educators 4.3) HR system to train and develop adult educators System in place to facilitate adult educators ongoing professional development Area of Review 5 WSQ Related Administrative System Criteria Interpretation of Criteria Implemented System Evidence of System Implementation Outcome of System Implementation Evident (E) / Not Evident (NE) WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 32

33 5.1) Learner Support System in place 5.2) System to ensure accurate and timely submission of assessment results 5.3) Prior to 3 Nov 2014, WSQ certificate printing and distribution system Learner support system that covers: 1. Course advisory 2. Progression pathway 3. Course schedule 4. Fees and refund policy 5. Retest/Appeal procedures 6. Career advisory (if applicable) 7. Placement support (If applicable) Assessment data are submitted in accordance to prescribed timeline within 2 weeks from the date of the assessment/within 1 month from the date of assessment*, unless otherwise approved Submission of results must be accurate *applicable only to In-house ATO Prior to 3 Nov 2014, WSQ cert is printed and distributed according to agreed service timeline: 1. Cert must be printed and distributed to trainees within 3 months from date of assessment, unless otherwise approved 2. Distribution of original certificate must be accurate - NA - NA - NA - NA - NA - NA With effect from 3 Nov 2014, System to ensure the availability of WSQ e-cert to the learners and is distributed (if applicable) With effect from 3 Nov 2014, WSQ e-cert is available according to agreed service timeline: - NA - NA - NA WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 33

34 1. e-cert must be available to trainees within 3 weeks from the date of the assessment/within 5 weeks from the date of assessment* 2. Upon request from trainee (after assessment result has been uploaded), WSQ e-cert will be printed and distributed. 3. Follow up with trainee upon receipt of notification on unsuccessful attempt to reach trainee. 4. Full qualifications (including transcripts) are printed and distributed within 3 months from date of assessment *applicable only to In-house ATO 5.4) Maintain up to date profile in Skillsconnect Profile information in Skillsconnect must maintain the most up-to-date information. This include: 1. Organization profile 2. Course profile 3. Adult educators profile WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 34

35 Section B If answer to any question below is Yes, please elaborate under Remarks column. S/N Questions Yes No Remarks 1 Are there changes in the management level personnel for the past one year? 2 Is there any major instability in the organisation s financial health for the past one year? 3 Has your organisation violated any WSQ terms and conditions? WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 35

36 Annex C: 2 Year Plan (Applicable to only ATO offering public courses) ATO is required to submit proposal on the training and assessment delivery on a 2-year timeline, this is to ensure that the ATO have a clear mission and vision on their involvement in WSQ. The proposal must indicate the following: i) Identified one specific WSQ framework, e.g., Security WSQ; ii) Intended scope of delivery within the framework, e.g. bundling of SOAs for the 2 licensing modules at the Certificate in Security Operations (CSO) level, full qualification of the CSO, etc; iii) Target market and proposed clientele mixed iv) Existing clientele and new clientele over the past year who dropped out, eg., trainees who had attained at least 3 GCE O Levels including English Language or its equivalent; and trainees with language difficulties and cultural adjustment issues (ascertained through screening); v) Proposed marketing and outreach strategies, eg. advertising in print media and newspapers, course previews and career fairs at recruitment centres, etc; vi) Intended annual headcount to be trained, eg. at least 25 trainees for the 2 licensing modules at the CSO level, (or 50 Statements of Attainments [SOAs]) per month; and vii) Proposed innovative learning and assessment strategies viii)risk assessments by the ATO. These include: Financial Risk Operations Risk Strategic Risk To facilitate the submission of the proposal by the ATO, ATO can use the provided template in the following page. ATO is advised to provide between 1000 and 1250 words for the proposal. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 36

37 Template for 2-year plan Applicants are advised to provide between 1000 and 1250 words for the proposal. i. Specific WSQ framework Identified existing WSQ framework, and/or new WSQ framework(s) with intention of embarking into. ii. Scope of delivery Intended scope of delivery within the framework (existing and/or new). Indicate the WSQ modules e.g. interact with guest, deliver service excellence, workplace safety and health. iii. Target market and proposed new clientele-mix The trainees and the intended proportion Eg., training only Singaporean and local PRs. Target Corporate Clients (indicate clienteles names). iv. Existing clientele The existing ratio of Singaporeans/PRs trainees in relation to foreign trainees. Existing Corporate Clients. v. Proposed outreach strategies Proposed marketing and outreach strategies, eg. advertising in the print media and newspapers, course previews and career fairs, etc. vi. Estimated annual headcount to be trained and projected number of SOAs Eg. at least 25 trainees for the 2 licensing modules at the Customer Service Officer level, (or 40 Statements of Attainments [SOAs]) per month. vii. Proposed learning and assessment strategies Eg. Small group tutorials cum buddy scheme for learners with low learning retention ability and compact disks (CD) would be deployed for intensive trainee revisions on the practical performances before assessments at the centralised assessment centre. viii. Risk assessments by the applicant Eg. Security ATOs would face a declining no. of trainees should their clients convert to in-house training or install the following IT infrastructure: - Bio-metric access and manpower scheduling system; - Remote CCTV surveillance system; and - Visitor management system. The ATO would hence face a challenge to meet the initial proposed training target/s. ATO to suggest and elaborate how to mitigate this risk. ix. Remark/Other Comments WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 37

38 Annex D: CIR Presentation by ATO The Continuous Improvement Review (CIR) will commence with a presentation by the ATO outlined as follows. The presentation can be in the form of a powerpoint presentation or briefing notes. The Management Representative (MR) must be present to conduct the presentation or provide clarifications. The presentation or brief notes will be part of the documentary evidences of the CIR. Evidences can include the following: 1. Course Quality Assurance i) Course files (course materials submitted at point of course application, includes Assessment Plan, Slides, Participants Handouts and Trainer Guide) ii) Summary page of courses with updated changes and what were the specific affected materials. iii) E.g. 1 Interact with Guests - Lesson plan has been contextualised. iv) E.g.2 Prepare for Service Assessment plan has been amended. v) Tracking mechanism used by ATO 2. Plan-Do-Check-Act (PDCA) i) Demonstration of establishment in PDCA. 3. Outcome Evaluation i) Update on actions taken based on organisation audit or previous CIR ii) Continuous review and feedback mechanisms in place and implementation iii) System to monitor and analyse L1 to L4 outcomes with detailed analyzed of collected data 4. Adult Educator Management i) Performance management of trainers/assessors ii) Training and development of trainers/assessors iii) Description of supervision mechanism for trainers and assessors under supervision 5. WSQ Related Administrative System i) Learner support system that covers course advisory, progression pathway, course schedule, fees and refund policy, retest/appeal procedures, career advisory, placement support (if applicable) ii) System in place to ensure accurate and timely submission of assessment results WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 38

39 iii) Prior to 3 November 2014, WSQ certificate printing (if applicable) and distribution system. Post 3 November 2014, system in place to ensure availability of WSQ e-cert and distribution (if applicable). iv) Course training and assessment records 6. Viability of Organisation i) Update on governance and management of ATO ii) ATO s financial health for the past one year 7. Free from Breach of Terms and Conditions i) Records of ATO for complaints or compliments received by ATO or on ATO *Presentation should be kept to 15 mins. WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 39

40 Annex E: CIR Outcome Measures Result of CIR Outcome A) Cleared CIR with no action required Continuation of approval status with new validity period, with or without new applications Ad-hoc CIR (outside of CIR schedule) Not Addressed Within Advised Timeline Not Addressed After Reminder B) Triggered by Other Factors i) Complaints with action required ii) Misrepresentation iii) Inaccurate or late report submission iv) Breach of WDA Terms and Conditions Reminder letter with rectification within 5 working days Revoke of WSQ ATO status. Issued by: Training Partners Development Division Singapore Workforce Development Agency Version: 1.14 Effective: 30 October 2014 WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 40

41 Version Control No Date Changes Made Version No May 07 Fee revised to $315 (pg. 6) Sep 07 Fee revised to $321 (pg. 6) Jan 09 Revision of areas of review Aug July Sep Nov 11 Added Essential Elements of Review (pg.12) Added System-Evidence -Outcome Approach (pg. 13) Added Annex B: Self-assessment Checklist (pg. 25) 1.5 Removed NSRS Added Example of S-E-O (pg.13) Added Annex C: 2 year plan (pg.35) Added CIR Grade (pg.40) 1.6 Revised submission timeline for self assessment checklist and 2 year plan (p.19) Added criterion 1.2 into Course Quality assurance system Revised Outcome Measures, i.e. Revoke of WSQ ATO status under column Not Addressed After Reminder (p.39) 1.7 Revised timeline for issuance of certificates Added Appeal Process (p.21) Sep 12 Revised webpage link (p.6) Oct 13 Fee revised from 1 October 2013 to $ and $1,605 on a yearly basis (page 22) 1.10 Based on Internal Quality Assurance (IQA) report, corrected grammatical and typo errors: 10 5 Nov 13 Page 9 From one of the extremely important process to one of the extremely important processes 1.11 WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 41

42 Page 9 From Identify best practices and role models amongst the ATO to Identify best practices and role models amongst the ATOs Page 10 From All ATO is subjected to to All ATOs are subjected to Page 11 From free breach of WDA terms and conditions to free from breach of WDA terms and conditions Page 18 From building you quality to building your quality Page 19 From the office-in-charged to officer-in-charge Page 23 From ATO is to responding to WDA to ATO is to respond to WDA Page 25 From All evidence to All evidences Based on Internal Quality Assurance (IQA) report, the irrelevant examples are removed: Page 35 Part iii Removed training only Malaysian learners Page 35 Part iv and v Removed Malaysian July 2014 Page 35 Part vii Removed Compact Disks (CD) would be deployed for intensive trainee revisions on the practical performances before assessments at the centralized assessment centre, Polwel Cooperative Society Limited. Page 21, under Continuation of Approval Statuses, changed from Director (Quality Assurance Division) to Deputy Director (Training Partners Development Division) 1.12 WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 42

43 12 25 September 2014 Page 39, under Issued by changed from Quality Assurance Division to Training Partners Development Division Fee revised to $1,605 with effective from 1 October 2014 (page 22) October 2014 Added the Post 3 November 2014 requirements for WSQ e-cert to the following pages: 11, 20, 25 and 33. Update to the Annex E, page 40: Removed the word grading from the title and removed CIR outcome B WDA-WSQ-CIR-GUIDE Effective date: 11 November 2014 version 1.14 Page 43