Changes to UK NEQAS Leucocyte Immunophenotyping Chimerism Performance Monitoring Systems From April Uncontrolled Copy

Transcription

1 Changes to UK NEQAS Leucocyte Immunophenotyping Chimerism Performance Monitoring Systems From April 2014

2 Contents 1. The need for change 2. Current systems 3. Proposed z-score system 4. Comparison of z-score analysis and original trial analysis 5. Supplementary data 2

3 1. The need for change UK NEQAS for Leucocyte Immunophenotyping (UKNEQAS LI) currently operates 19 different External Quality Assessment (EQA) programmes covering various aspects of flow cytometry and molecular haematooncology. To examine laboratory performance there are 10 different performance monitoring systems in use by this centre, each based on different analytes, statistical methodologies and with different criteria to classify unsatisfactory performance. Percentage donor chimerism results returned by UKNEQAS LI participants produce a tight data set with a number of gross outliers representing erroneously reported % recipient results and can be skewed by 100% or 0% donor chimerism results representing participants whose assays do not have the required sensitivity to detect small proportions of recipient or donor cells respectively. This unusual distribution of data has lead to a number of iterations of the chimerism performance monitoring systems as traditional statistical methods for identifying outliers have proved too stringent (>15% of participants being identified as outliers) or too lenient (deviations from the consensus median of >10% not being identified as an outliers) Another drawback of the current chimerism performance monitoring system is that is inherently competitive. The nature of a centiles based approach means that it will always produce outliers regardless of the variance of the data set. For the reasons outlined above, UKNEQAS LI have identified the need for a new, more robust, simple chimerism performance monitoring system. 3

4 2. Current Chimerism scoring system One or two post stem cell transplantation (SCT) samples are issued at each trial with varying levels of donor engraftment chimerism. There are three trials per annum. Chimerism is measured using molecular techniques and is a quantitative response expressed as a percentage of the engraftment (maximum 100% engraftment), which allows for the serial monitoring of patients post transplant. The scoring system is a quantitative approach for which participants are asked to produce a percentage engraftment using their normal laboratory technique. From the data submitted the scoring system is applied in 3 parts. Part 1: Results of 100% and 0% will be removed from the data set as a failure to detect either donor or recipient DNA in a sample for which the consensus shows that this DNA was present. This will be deemed unsatisfactory performance and the participant will be awarded an amber performance status. Part 2: Uses the formula; median +/-(Q3-Q1 x 1.5) to give the upper and lower adjacent points. This allows for the data set to be trimmed so outliers do not skew the data set, resulting in inappropriate scoring criteria. When part 2 is performed, participants with results that fall outside of the upper or lower adjacent points will be given an amber performance status. Part 3: Using the consensus median from the remaining data, the 2.5th and 97.5th centile will be calculated. Participants with results that fall outside of the 2.5 th and 97.5th centile will be given an amber performance status. Initially all three parts of the scoring system will be applied. If too many participants are awarded amber performance (e.g. greater that 15% of returned results) then the scoring system will be reapplied to the data set omitting part two. 4

5 Additionally, where two post SCT samples are issued and a participant fails to identify the correct trend (e.g. one sample being lower than the other), and where the first and third quartiles of the two samples do not overlap, that participant will be given an amber performance status. 3. Proposed z-score analysis One or two samples are issued each trial. A participant's submitted % donor chimerism result for each sample is then used in conjunction with the robust mean and robust standard deviation to calculate a z score using the following formula: where x is the result returned by the testing laboratory, X is the assigned value (robust mean) and is the standard deviation for proficiency assessment (robust SD). The robust mean and robust SD are derived from participant data using Algorithm A (ISO ) that ensures that all data is included in the generation of the robust mean and robust SD but also minimizes the effect of outliers upon the final values. Interpretation of z-scores is as follows: A result between 2.0 and -2.0 would be classed as satisfactory A result between 3.0 and 2.0 or -2.0 and -3.0 is seen as an 'action result, that highlights a potential issue to the laboratory. Two action results in a period of 3 samples would result in classification as a critical A result above 3.0 or below -3.0 is considered to be a critical result requiring immediate investigation by the laboratory 5

6 Due to the nature of how z-scores are generated a positive z-score highlights a positive bias in a laboratory s results whereas a negative z- score shows a negative bias. As such, this adds value to the performance monitoring information provided to laboratories because the z-score immediately highlights to the participating centre if their result is above or below the expected consensus value. In addition to the z-score all methodological data featured on reports will be in the format of robust mean and robust SD. This will give participants the option to use the extra provided data to calculate additional in-house z-scores based on machine types, methodologies etc and allow them to monitor if there are any in-house technical biases. However, it is important to stress that the z-score issued by UK NEQAS for Leucocyte Immunophenotyping based on all methods will remain the only parameter that is used for performance monitoring. Any laboratory who fails to return a result by the closing date will be regarded as an action for each sample. As such any laboratories that do not return results for both samples within a trial will be classified as critical. Unsatisfactory performance in this programme is defined as any occurrence of critical performance and this will be initially communicated to participants on their trial report. This will be followed up with a letter on each occurrence of unsatisfactory performance highlighting that performance on the last sample(s) was out of consensus and offering support and guidance to assist in returning to satisfactory performance. This may take the form of repeat/additional samples, communications by , telephone conversations or face to face communications. If a participant s status is elevated to persistent unsatisfactory performance (defined as a critical classification on 3 or more occasions within a 12 month period) then a further letter will be issued and the Haematology National Quality Assurance Advisory Panel informed (for UK participants only). 6

7 4. Comparison of z-score analysis and original trial analysis In order to assess the performance of z-scores on the UKNEQAS LI chimerism data set we compared it to the current methodology for identifying outliers (median and centiles). We assessed retrospectively 3 data sets across one calendar year as this reflects the period over which we assess persistent unsatisfactory performance. We also assessed the effect of the number of d.p. to which % chimerism is reported and simulated several data sets to look at the effect of skewing of the data which may occur within chimerism data sets around the limits chimerism detection by STR analysis (100% donor or 100% recipient). When the general performance of z-score based scoring method is compared to a centiles based method it can immediately be seen that the identification of outliers is much more reproducible between samples. Using z-scores a critical deviation from the consensus robust mean (>3 s.d.) showed a range of between 4.14%-5.37% across 6 samples compared to a range of 2.3%-56.5% using the centiles based approach (see tables 2, 3 and 4). The inherent nature of a centiles based approach means that it will detect a consistent number of outliers regardless of the variance of the data set. Z-scores, however, will increase and decrease the outliers identified based on the relative variance of the data and will not produce any outliers when the dataset shows little variance. As such the z-score boundaries imposed are a reflection of the variation and limitations of the analytical techniques used. Currently UKNEQAS LI allows participants to return % donor chimerism results to as many decimal places (d.p.) as they would report them clinically. We consistently see a range of % donor chimersim results returned to us from integers to results to 3 d.p. UKNEQAS LI s specialist 7

8 advisory group has advised that the reporting of chimerism data to 1 d.p. is not clinically relevant, as it infers a precision that is unwarranted in what is, at best, a semi quantitative assay. When assessing z-scores participant returns were analysed raw (as provided to us by participants) and converted to integers to assess the potential impact on scoring. The impact we observed was negligible with a range of adjustments (with no positive or negative bias) to the critical deviation from the mean between 0.03% and 1.29% with an average of <1% (see tables 2, 3 and 4). This led to between 0 and 3 participants falling into different scoring categories per sample. As such, we recommend that participant results are processed as integers in order to drive best practise within the profession. The impact of calculating the robust mean and robust SD as integers or to 1 d.p. or 2 d.p. was also assessed. Calculating the robust mean and SD as integers produced a critical deviation from the mean in our data set with the smallest SD of only +/- 3% which we propose is not a clinically relevant deviation. Calculating to 1 d.p. ( +/- 4.2%-5.4%) and 2 d.p. ( +/- 4.14%-5.37%) provided more clinically relevant critical deviations from the robust mean and as calculating to 2 d.p. provided little extra discrimination we propose calculating the robust means and SD to 1 d.p. Whilst we acknowledge that only allowing participants to report an integer and calculating a s.d. to 1 d.p. seems inconsistent this is necessary as it provides an extra degree of discrimination allowing for the clinically relevant identification of outliers. Finally the impact of the skewing of the data sets was assessed. As post SCT samples approach 100% donor or 100% recipient UKNEQAS LI see an increase in participants returning results as 100% or 0% donor chimerism due to insufficient sensitivity within their assay. We simulated a number of data sets to assess the effect of this potential positive and a negative skew on our data. The z-scores proved robust with minimal adjustment to the critical deviation from the mean, and the scoring of virtually all participants remaining unchanged (see tables 2-4). 8

9 Following analysis of a range of different data sets, both real and mocked, we conclude that the z-scored based method for performance monitoring of the chimerism programme is a more robust, consistent and simple method of identifying clinically relevant outliers. 9

10 5. Supplementary Data Table 1: Summary of current median and centiles based approach Trial Chim Chim (Trimmed Data) Chim Sample Median th centile th centile Acceptable +ve deviation from the median Acceptable -ve deviation from the median Ambers

11 Table 2: Summary of z-score data for chimerism trial Sample Sample 128 Sample 129 Robust Mean Robust SD Raw Data Integers 0% Skew 100% Skew Raw Data Integers 0% Skew 100% Skew Action Critical Action Window Critical Window >+/-3.82 >+/-3.50 >+/-3.58 >+/-3.62 >+/-4.44 >+/-3.58 >+/-5.08 >+/-5.22 >+/-5.73 >+/-5.25 >+/-5.37 >+/-5.43 >+/-6.66 >+/-5.37 >+/-7.62 >+/-7.83 Table 3: Summary of z-score data for chimerism trial Sample Sample 132 Sample 133 Robust Mean Robust SD Raw Data Integers 0% Skew 100% Skew Raw Data Integers 0% Skew 100% Skew Action Critical Action Window Critical Window >+/ >+/ >+/ >+/ >+/ >+/ >+/ >+/ >+/-4.77 >+/-4.74 >+/-4.86 >+/-4.98 >+/-4.71 >+/-4.68 >+/-4.92 >+/

12 Table 4: Summary of z-score data for chimerism trial Sample Sample 136 Sample 137 Robust Mean Robust SD Raw Data Integers 0% Skew 100% Skew Raw Data Integers 0% Skew 100% Skew Action Critical Action Window Critical Window >+/-2.74 >+/-2.80 >+/-2.86 >+/-3.14 >+/-2.46 >+/-2.76 >+/-3.14 >+/-2.82 >+/-4.11 >+/-4.20 >+/-4.29 >+/-4.71 >+/-3.69 >+/-4.14 >+/-4.71 >+/

13 Table 5: Discordant outlier data Chim Sample 128 (robust mean = 85.96%) Participant ID % reported original Status z-score z-score status 1 82 Amber Action Amber 2.84 Action Green 2.26 Action Amber Action 13

14 Original Analysis Amber Green Amber Figure 1: original scoring versus z scores for all participants for sample 128 Chim Original Data Analysis 2.5 and 97.5th percentile Chim Sample 128 Participant Number Z score Data Analysis Critical Critical Action Satisfactory Action Z Score analysis 14

15 Table 6: Discordant outlier data Chim Sample 129 (robust mean = 69.19%) Participant ID % reported original Status z-score z-score status 1 74 Green 2.14 Action Green 2.68 Action Amber 2.90 Action 4 74 Green 2.14 Action Amber Action 6 74 Green 2.14 Action Amber Action 15

16 Original Analysis Amber Green Amber Figure 2: original scoring versus z scores for all participants for sample Chim Chim Sample Original Data Analysis 2.5 and 97.5th percentile Participant Number Z score Data Analysis Critical Satisfactory Action Critical Action Z Score analysis 16

17 Table 7: Discordant outlier data Chim Sample 132 (robust mean = 88.95%) Participant ID % reported original Status z-score z-score status Amber Satisfactory 2 85 Amber Action 3 84 Amber Action 4 85 Amber Action 5 92 Amber Action Amber Satisfactory 17

18 Original Analysis Amber Green Amber 100 Figure 3: original scoring versus z scores for all participants for sample Chim Original Data Analysis 2.5 and 97.5th percentile Chim Sample 132 Participant Number Z score Data Analysis 4 2 Critical Action 0 Satisfactory -2 Action Critical Z Score analysis 18

19 Table 8: Discordant outlier data Chim Sample 133 (robust mean = 10.00%) Participant ID % reported original Status z-score z-score status 1 13 Amber 1.79 Satisfactory Amber 2.26 Action 3 7 Amber Action Amber Satisfactory Amber Satisfactory Amber 2.93 Action Amber 1.34 Satisfactory Amber 1.36 Satisfactory Amber 1.91 Satisfactory 19

20 Original Analysis Figure 4: original scoring versus z scores for all participants for sample Chim Original Data Analysis 2.5 and 97.5th percentile Chim Sample 133 Participant Number Z score Data Analysis Z Score analysis 20

21 Table 9: Discordant outlier data Chim Sample 136 (robust mean = 85.60%) Participant ID % reported original Status z-score z-score status 1 75 Green Critical Green Critical Green 2.85 Action 21

22 Original Analysis Figure 5: original scoring versus z scores for all participants for sample Chim Original Data Analysis 2.5 and 97.5th percentile Chim Sample 136 Participant Number Z score Data Analysis Z Score analysis 22

23 Table 10: Discordant outlier data Chim Sample 137 (robust mean = 85.30%) Participant ID % reported original Status z-score z-score status Green Action 2 88 Green 2.13 Action 3 70 Green Critical Green Critical Green Critical 6 79 Green Critical 23

24 Original Analysis 100 Figure 6: original scoring versus z scores for all participants for sample Chim Original Data Analysis 2.5 and 97.5th percentile Chim Sample 137 Participant Number Z score Data Analysis Z Score analysis 24

25 25