Procedure for MSC Sampling

Transcription

1 `1 Procedure for MSC Sampling ASI-PRO MSC Sampling v1.0 Approved by: ASI Managing Director Date of approval: 20 September PURPOSE AND SCOPE 1.1 The purpose of this document is to set forth the ASI procedure for assessment sampling of CABs in the MSC Accreditation Program. 1.2 The scope of this document covers the development of ASI s annual assessment plans for evaluation of CABs in the MSC Accreditation Program. Scope includes planning of initial accreditation assessments, surveillance assessments, and reassessments of CABs. Scope does not include certification schemes other than MSC (i.e. the FSC Accreditation Program is not within scope). 1.3 The specific objective of the ASI sampling design is to ensure that ASI s initial assessments of applicant CABs and ASI s periodic surveillance assessments and re-assessments of accredited CABs are based on representative samples which are appropriate to the CAB s scope of accreditation and which are taken at sufficiently close intervals so as to enable ASI to monitor the continued fulfilment of accreditation requirements by the CAB (ref. clauses 7.5.6, , and from ISO 17011). 1.4 This specific objective is achieved by providing a detailed description of how ASI integrates its sampling design into an annual assessment plan. 2 TERMS AND DEFINITIONS 2.1 All terms and definitions are provided in the ASI Vocabulary (ASI-PRO ). 3 ASSOCIATED DOCUMENTATION 3.1 Standards and Policies ISO 17011: Procedures, Guidelines ASI-PRO Accreditation v3.0 ASI-PRO Surveillance v3.0 ASI-PRO FSC Sampling v2.1 ASI-GUI Nonconformities (NCs) v Checklists, Templates, etc. None ASI-PRO Version September 2010 Page 1 of 8

2 4 PROCEDURE All APMs and lead assessors in the MSC Accreditation Program shall follow this procedure to develop annual assessment plans for CABs unless exceptional circumstances dictate the development and use of another plan. In such cases, any deviation from the sampling design shall be approved by the ASI Managing Director prior to implementation of a revised assessment plan. 4.1 General Before the end of each calendar year, ASI staff members of the MSC Accreditation Program shall prepare an annual assessment plan for the forthcoming year. The annual assessment plan shall be developed following the sampling design described in this procedure and it shall encompass the assessment of all CABs in the Program ASI shall implement its annual assessment plan through the use of three different types of assessment activities: office assessments, witness assessments, and desk studies (the latter are sometimes also called desktop reviews or document reviews) While ASI does perform 'compliance audits' of CABs within the FSC Accreditation Program (see clause of the ASI-PRO FSC Sampling), ASI does not perform compliance audits of CABs in the MSC Accreditation Program. Therefore compliance audits are not specified in the annual assessment plan for MSC ASI does perform verification assessments of CABs within the MSC Accreditation Program (clause 8.3, ASI-GUI Nonconformities). However verification assessments are unscheduled ad hoc activities that occur in response to CAB nonconformity. Thus verification assessments are not specified in the annual assessment plan The annual assessment plan shall be developed following the methods described here for sample design which specify: Calculation of the total number of samples that ASI will assess from each CAB (= Sample Number); Identification of the criteria by which ASI will select the specific samples for assessment (= Sample Selection); and Estimation of the amount of time that ASI will require for each assessment (= Sample Duration) Results for sample number, sample selection, and sample duration may be further adjusted in order to take account of any identified risk factors of relevance to the scope and operation of a CAB. Adjustments should follow the risk tables presented below and all adjustments are recorded in the annual assessment plan The results from annual assessment sampling are analyzed using a method termed Informed Judgment wherein assessors review and synthesize the objective evidence derived from different types of independent ASI assessment activities ASI-PRO Version September 2010 Page 2 of 8

3 (office, witness, desk study) in order to arrive at a generalization about the competency of a CAB to operate a certification program in conformity with accreditation requirements ASI chooses the method of informed judgment over alternate methods which rely on statistical analyses because: 1) prohibitive costs would be incurred if ASI were to collect samples of sufficiently large size for applying rigorous statistical analyses, and 2) there is a need for ASI to develop its assessment plans responsibly, so that the monitoring program identifies and directly evaluates known risks [but such directed sampling violates a critical assumption in statistics that the samples are drawn from a population at random] ASI implements the informed judgment method in a 'PDCA' cycle. Each year, APMs formulate an annual assessment plan that is based on clearly defined sampling objectives [Plan], the plan is implemented for one year [Do], the results are reviewed for adequacy and completeness [Check], and the results are used to modify the assessment plan for the forthcoming year capturing any lessons learned [Act]. Figure 1. The PDCA cycle used by ASI for annual assessment of CABs. 4.2 Basal Sampling Rate ASI shall use a basal sampling rate (BSR) as a default assumption in setting the number of annual assessments of each accredited CAB in the Program Annually, at a minimum, the BSR specifies for each accredited CAB: One office assessment of the accredited head office of the CAB; and One witness assessment of CAB conformity assessment services that as representative and appropriate to the CAB s scope of accreditation (e.g. one witnessed CoC audit and one witnessed fishery assessment activity). ASI-PRO Version September 2010 Page 3 of 8

4 4.2.3 As described in the following sections (see below), the APM shall increase the sampling rate above BSR following review of risks which includes consideration of the size of each CAB's certification program, evaluation of known risk factors, and the degree of certification activity by CAB affiliate offices. 4.3 Sample Number - Office Assessments For office assessments, the ASI sampling objective is to perform one assessment of the accredited head office per annum (i.e. once every 12 months). The ASI Managing Director may approve an increased or decreased assessment frequency, but the time between successive office assessments shall not exceed 18 months In addition to assessments of the accredited head office, ASI shall perform office assessments at CAB affiliate offices In order to identify which offices will be considered 'CAB affiliate offices', APMs shall use the definition from ASI-PRO ASI Vocabulary v1.0 which states: "An affiliate office is a partner organization or subcontractor which reviews CAB s applications, issues proposals, and reviews and enters into certification contracts independently, selects and trains auditors, but does not make certification decisions." The ASI sampling objective is to perform at least one office assessment of each CAB affiliate office during the course of the five year accreditation period (i.e. 20% of CAB affiliate offices are sampled each year). Table 1. Summary of ASI Sampling Objectives for Office Assessments. Site CAB Accredited Head Office CAB Affiliate Offices Total No. Sample Size (% per year) Accreditation Surveillance Re-Accreditation 1 100% 100% 100% Variable n/a 20% 20% Individual CAB affiliate offices may be subject to increased sampling frequency according to a risk analysis (Table 2) The ASI Managing Director may approve the use of remote means (e.g. teleconference or other electronic media forms) for performing office or witness assessments but only following due consideration of past CAB compliance with accreditation requirements. Table 2. Selection Criteria for Increased Surveillance of a CAB Affiliate Office. ASI-PRO Version September 2010 Page 4 of 8

5 No Risk Factor / Selection Criterion The affiliate office was identified by decision of the ASI Managing Director or a senior executive of the certification scheme owner. The affiliate office is subject to a formal complaint or other form of dispute that is supported by documentary evidence. The affiliate office is part of any kind of official investigation. The affiliate office operates in a country or region that ASI has identified as representing a significant risk area. The affiliate office had > 2 major nonconformities raised by ASI during the previous office assessment. The affiliate office manages more than 50 certificates. The affiliate office has a recent history of poorly managed audit teams. 4.4 Sample Number - Witness Assessments For witness assessments, the ASI sampling objective is to perform at least one witness assessment representing each scope of accreditation per annum (i.e. once every 12 months). This is the Basal Sampling Rate (BSR). The ASI Managing Director may approve an increased or decreased assessment frequency, but the time between successive witnessed assessments shall not exceed 18 months The APM shall increase the number of witness assessments based on the number of CAB certification clients according to the following formulas: For the scope of CoC certification, sample rate is 1% of CoC certificates, with values rounded up to the nearest whole number; and For the scope of Fishery certification, sample rate is 5% of fishery assessments (certified fisheries plus those under assessments), with values rounded up to the nearest whole number. Table 3. Summary of ASI Sampling Objectives for Witness Assessments. Scope CoC Fishery Accreditation 1 (first CoC audit) 2 (first fishery assessments) Sample Size Surveillance 1% of all CoC certificates 5% of all Fishery Assessments Re- Accreditation 1% of all CoC certificates 5% of all Fishery Assessments The APM shall record the adjusted number of witness assessments in the annual assessment plan. ASI-PRO Version September 2010 Page 5 of 8

6 4.4.4 ASI may substitute desk studies for witness assessments in some instances (e.g. when the competency of CAB auditors can be adequately evaluated through review of fishery assessment reports). In such cases, those assessments conducted by desk study shall not count against attaining the BSR and they shall be approved of the ASI managing Director in advance. 4.5 Sample Selection The designated APM will use discretion to select the specific sites and/or CAB activities for witnessing In selecting samples for witness assessments, the APM will consider the risk factors and selection criteria listed in Table 4. Table 4. Criteria for Selecting Samples for Witness Assessment. No Risk Factor / Selection Criterion The certificate was identified by decision of the ASI Managing Director or a senior executive of the certification scheme owner. The certificate is subject to a formal complaint or other form of dispute that is supported by documentary evidence. The certificate was issued in a country or region that ASI has identified as representing a significant risk area. The certificate was identified for further follow-up during an ASI office assessment or through desk study. The certificate was issued in a country or region where the CAB does not normally operate (non-native country) or in which the CAB has only recently started. The certificate is for a product or service that was not offered by a CAB in previous years in a certain country or region. The certificate represents an opportunity to sample from a CAB lead auditor that has not been witnessed previously by ASI. The certificate represents an opportunity to sample the work of a CAB's subcontractor, affiliate, or partner that has not been witnessed previously by ASI. Sampling of the certificate is reasonable based on logistical considerations including combination of trips and availability of ASI assessors and experts. The certificate is selected at random Planning of assessments of affiliate offices should follow a regular schedule so that all of the affiliates are assessed during the five year period of accreditation (see above). However the APM should use the criteria shown in Table 2 to identify highrisk affiliate offices. Affiliate offices identified as high risk shall be moved forward in the annual assessment plan, or subject to increased surveillance frequency if deemed necessary by the ASI managing Director. 4.6 Sample Duration ASI-PRO Version September 2010 Page 6 of 8

7 4.6.1 Sample duration is the number of days on-site plus preparation time, time spent reviewing CAB documentation, and time spent preparing ASI reports. Sample duration is measured in assessor-days. Table 5 shows how to calculate sample duration for different assessment types and provides default (approximate) values. Sample duration may be adjusted upwards or downwards based on the actual time required for each assessment activity. Table 5. Guidance for Calculating Sample Duration. No. of Assessor-Days Assessment Activity Office Assessment CoC Witness Fishery Witness Desk Study Preparation 0.5 to Travel Time Time On Site* 1 to to 3 - Document Review to 1 1 ASI Report Writing 1 to to to 1 Total 2.5 to to to 2 * As previously agreed with the MSC, ASI does not invoice CAB's for assessor travel time. This policy may be subject to change in the future Office assessments: The APM will estimate time required on site following consideration of the size of the CAB's certification program and any identified risk factors. Generally, office assessments will require 1 to 4 assessor-days on site Witness assessments: The APM will estimate the time required on site following consideration of the actual time assessment duration following the guidance shown in Table 5. Generally, witness assessments of CABs conducting CoC audits will require 2.5 assessor-days. Witness assessments of CABs performing fishery assessments will require 3.5 to 5.5 days Desk studies: Desk studies do not involve on-site assessment. Therefore the sample duration of a desk study is generally based on the number of days required to complete the document review(s) and prepare the ASI report (see Table 5) Verification assessments: Verification assessments may or may not require that the ASI assessor spends time on-site. Sample duration of verification assessments is based on the relevant categories for assessment activity shown in Table Annual Review of ASI Assessment Results Before the start of the annual assessment planning meeting, APMs from the MSC Accreditation Program will perform an annual review of all assessment results obtained for each CAB during the past year This annual review shall specifically include a synthesis of results from office assessments and witness assessments as appropriate to the CAB's scope of accreditation. ASI-PRO Version September 2010 Page 7 of 8

8 4.7.3 During the annual review, APMs will consider all objective evidence derived from different types of ASI assessment (e.g. from office, witness, desk study, and/or verification assessments). The review shall identify where the datasets tend to reenforce conclusions about CAB competency, and the review shall identify any data limitation or gaps Where the results from sampling were found to be inadequate or insufficient, the APMs shall make recommendations to improve ASI's sampling of CABs in order to address risks. Improvements shall be incorporated into the annual assessment plan for the forthcoming year (see Section 4.1) When the annual assessment plan for the forthcoming year has been approved by the ASI managing Director, the PDCA cycle is considered complete and a new cycle is initiated. 4.8 Communication of ASI Annual Assessment Plan The number of selected sample(s) to be assessed for surveillance assessments should be communicated to the CAB at the beginning of the assessment year, or at least three months before the assessment is scheduled by the appointed APM. The APM may choose to inform the CAB of the identity of individual samples or may choose not to inform the CAB until as close to the date of the assessment as is practicable to allow for necessary logistical arrangements If a CAB does not agree with the selection of one of the certificates, the issue shall be referred to the ASI Managing Director for final decision End of Document Accreditation Services International GmbH This work is copyright to ASI. It may be reproduced or copied for fair use by those seeking accreditation, or those wishing to learn more about accreditation, but may not be used by others without the written permission of ASI. ASI-PRO Version September 2010 Page 8 of 8