Traditionally, surgical skills were acquired in vivo with one-on-one

Transcription

1 RANDOMIZED CONTROLLED TRIAL Prospective, Randomized Assessment of Transfer of Training (ToT) and Transfer Effectiveness Ratio (TER) of Virtual Reality Simulation Training for Laparoscopic Skill Acquisition Anthony G. Gallagher, PhD, DSc, Neal E. Seymour, MD, Julie-Anne Jordan-Black, PhD, Brendan P. Bunting, PhD, Kieran McGlade, MD, and Richard Martin Satava, MD Objectives: We assessed the effectiveness of ToT from VR laparoscopic simulation training in 2 studies. In a second study, we also assessed the TER. ToT is a detectable performance improvement between equivalent groups, and TER is the observed percentage performance differences between 2 matched groups carrying out the same task but with 1 group pretrained on VR simulation. Concordance between simulated and in-vivo procedure performance was also assessed. Design: Prospective, randomized, and blinded. Participants: In Study 1, experienced laparoscopic surgeons (n = 195) and in Study 2 laparoscopic novices (n = 30) were randomized to either train on VR simulation before completing an equivalent real-world task or complete the real-world task only. Results: Experienced laparoscopic surgeons and novices who trained on the simulator performed significantly better than their controls, thus demonstrating ToT. Their performance showed a TER between 7% and 42% from the virtual to the real tasks. Simulation training impacted most on procedural error reduction in both studies (32-42%). The correlation observed between the VR and real-world task performance was r > 0 96 (Study 2). Conclusions: VR simulation training offers a powerful and effective platform for training safer skills. Keywords: Operating room (OR), simulation, transfer of training (ToT), transfer effectiveness ratio (TER), virtual reality (VR) (Ann Surg 2013;257: ) Traditionally, surgical skills were acquired in vivo with one-on-one learning in the operating room under the supervision of a master surgeon. This approach to learning has been considerably challenged 1 by a sea-change in resident duty hours in the United States 2 and in the European Union. 3 Satava 4 proposed virtual reality (VR) simulation as an alternative strategy for the acquisition of skills that would also minimize risk to patients, particularly during the early part of the learning process. Although widely used in the aviation, 5 nuclear, 6 military 7, and space sectors 8 for training high-risk, high-skilled performance characteristics, procedural-based interventional medicine From the School of Medicine, University College Cork, Cork, Ireland; Department of Surgery, Tufts University School of Medicine, Springfield, MA; School of Psychology, Queen s University, Belfast, Northern Ireland, United Kingdom; School of Psychology, University of Ulster, Derry, Northern Ireland, United Kingdom; School of Medicine, Dentistry and Biomedical Sciences, Queen s University, Belfast, Northern Ireland, United Kingdom; and Department of Surgery, University of Washington Medical Center, Seattle, WA. Disclosure: The authors declare no conflicts of interest. Reprints: Anthony G. Gallagher, PhD, DSc, School of Medicine, Brookfield Health Sciences Complex, College Road, University College Cork, Ireland. anthonyg.gallagher@btinternet.com. Copyright C 2013 by Lippincott Williams & Wilkins ISSN: /13/ DOI: /SLA.0b013e318284f658 had almost no experience of this approach to training. Indeed, one of the major problems that interventional medicine had/has to overcome is the belief that surgical/procedural skills can only be learned while working on real patients. Other high-risk, high-skills sectors have developed an approach to learning practical skills that is based on an approach to learning developed by Edward L. Thorndike and Robert S. Woodworth 9 at the start of the 20th century. Thorndike and Woodworth proposed that what individuals learned in one context, they transferred it to another context that shared similar characteristics. Their theory implies that transfer of training (ToT) depends on the proportion to which the learning task and the transfer task are similar. In the aviation industry, to measure ToT from a flight simulator to an airplane, at least 2 groups of trainees are required. Speed of learning in an airplane by a group first trained in a simulator is compared with the speed of learning by a control group trained only in the airplane. 10 Quantitative measures of transfer include transfer effectiveness ratio (TER), which is the proportion of the time or trials saved in the airplane to the time or trials spent in the simulator. Transfer of training research (independent of simulation) has developed into a multibillion-dollar industry, producing a wealth of empirical findings and theoretical interpretations, which are often contradictory and frequently based on survey data. 11,12 Despite VR simulation having a presence in surgery for almost 2 decades, its acceptance by the medical profession as a legitimate and powerful training technology is still tenuous. A small number of studies have shown that VR training improves operating room performance However, medicine has been critical of these studies for a variety of reasons: the small number of subjects used, the fact that they have usually been conducted within a single institution, that the trainees in these studies were relatively junior (ie, surgical residents), and that the surgical procedure they were being prepared for was relatively low-risk, ie, laparoscopic cholecystectomy. 15 The first aim of the studies reported here is to assess a number of aspects of the ToT effect to replicate the ToT effect but with a group of very experienced laparoscopic surgeons. The second aim is to quantify the proportion of skills transferred for performance of a relatively straightforward but novel laparoscopic task from a VR simulation to an equivalent novel real-world task, ie, TER. In Study 2, the real-world task was built specifically to closely resemble the VRsimulated training task. The VR task is a component task of one of the best-validated VR simulators in medicine, ie, minimally invasive surgical trainer virtual reality (MIST VR). This simulator has been shown to have robust psychometric assessments that demonstrate construct, concurrent, discriminative, and internal validity It also demonstrated predictive validity in prospective, randomized clinical trials that showed that training on MIST VR improved intraoperative performance. 13,14,16,22 We will assess the concordance between performance on the simulation and the corresponding performance characteristics of performance on the real-world task. These data are important to give better insight into the relationship between the trainee s performance on the VR simulation and his/her performance on the equivalent real-world task. A study examining equivalent Annals of Surgery Volume 257, Number 6, June

2 Gallagher et al Annals of Surgery Volume 257, Number 6, June 2013 performance in virtual and real-world environments is absent from the surgical literature. Inferences have been drawn from clinical studies using VR to train performance for the operating room, but the nature of these studies makes it almost impossible to make direct and specific comparisons. METHODS Randomization and Recruitment In Study 1, numbers and in Study 2, numbers 1 30 were assigned to one of two groups using a research number randomizer (by AGG). In each study, the number and their group allocation were kept in an envelope and retrieved (from an independent third party) just before subject participation in the study. Subjects in Study 1 were recruited by NES and RMS and in Study 2 by JAJ. Participants were assigned and trained by support researchers. Outcomes assessments and analysis were conducted by individuals blinded to participant identity and training group status. The flow of subjects through the study procedures are shown in Figure 1. It is hypothesized that training in the VR environment transfers to real-world tasks and that this ToT is greater than 10%. In both studies, training transfer is assessed by having the 2 matched groups complete very similar real-world tasks. Before completing the realworld task, one group participated in a period of training on a VR simulator. Assessment of the TER effect was the difference between the objectively assessed performances of the VR-trained subjects in comparison with the non-vr-trained subjects on the real-world task. H 0 was that there would be no difference in performances between the 2 groups in either of the 2 studies. STATISTICAL ANALYSIS Differences between groups were compared for statistical significance with 1-factor analysis of variance (ANOVA). Study 1 Subjects One hundred ninety-five experienced laparoscopic surgeons (ie, had performed more than 50 minimally invasive surgical procedures) from 13 countries participated in this study. Their demographic details are shown in Table 1. Apparatus Box-Trainer. A standard laparoscopic cutting task was used, as previously reported (Fig. 2A) Briefly, a cut is made in 26 spaces clearly demarcated by black lines along the long edge of a sheet of paper. The task was placed in a portable laparoscopic trainer (3-D Technical Services, Franklin, OH). Ethicon nontoothed laparoscopic forceps (Ethicon Endosurgery, Cincinnati, OH) and laparoscopic curved scissors were used to hold and cut the paper. This task has been shown to be sensitive to differences between experienced and junior surgeons, is sensitive to learning, measures correct responses and errors, has low variability, and has been demonstrated to conform to the Gaussian distribution and a high inter-rater reliability MIST VR The MIST-VR (Fig. 2B) system (Mentice AB, Gothenburg, Sweden) has been previously described by Wilson et al and others. 27,28 It offers 6 tasks of incrementally greater complexity with 3 difficulty levels. Each one is based on a key surgical technique employed in laparoscopic cholecystectomy. Performance metrics are generated for each task formatively and summatively. A novel task based on the box-trainer task described above was developed for this study and is shown in Figure 2C. The task required the subject to hold a virtual card with the instrument in one hand and excise virtual spheres with the instrument in his/her other hand. Procedure Subjects completed a short interview about their surgical experience and training. After confirming that they had completed more than 50 laparoscopic procedures, they were given a unique identifier number and then they watched a video recording explaining how to perform the laparoscopic tasks, including what constituted an error. Subjects were prospectively randomized to complete the VR task first and then the box-trainer cutting task or they completed the boxtrainer task only. The VR task required subjects to hold the virtual card with one instrument and excise the spheres by grasping them in the center with the other instrument. When this was done correctly, the sphere disappeared. After all 5 spheres had been excised, a new card appeared. All subjects were required to complete 4 cards with a total of 20 spheres. Timing was stopped between all the spheres being excised for 1 card and a new card appearing. The laparoscopic cutting task was placed horizontally in a conventional box-trainer under standardized laparoscopic conditions. Subjects were required to make 1 incision between 26 spaces clearly demarcated by black lines along the long edge of the sheet of paper (US Letter). Subjects grasped the paper with their nondominant hand and cut with their dominant hand. An error was judged to have been made if a subject s incision cut across one of the black lines or touched it. If an incision was so close to the line that the experimenter found it difficult to make a decision, it was judged to be an error. Twenty percent of sheets were rechecked by another investigator. The interrater reliability was 98% and the Cronbach alpha was Subjects were asked to make as many incisions as they could in 2 minutes. The laparoscopic task was removed from the box-trainer after each subject was finished. Timing commenced when subjects placed the paper between the jaws of the scissors for their first incision. Results There were no significant differences in demographic characteristics between the 2 groups. The vast majority (in both groups) of surgeons who were assessed were from the United States. Both groups were very experienced laparoscopic surgeons and reported having performed more than 900 minimal invasive surgical (MIS) procedures during their career and more than 120 annually. The surgeons who practiced the simple cutting task on the VR simulator before performing the real-world task, on average, made significantly more correct incisions on the real-world task (20.73 vs ; F (df = 1, 193) = 4.717, P = 0.031), ie, 7% more correct incisions (Fig. 3A). They also made a significantly lower percentage of incorrect incisions (1.79 vs. 2.62; F (df = 1, 193) = 4.491, P = 0.035), ie, 32% fewer incorrect incisions (Fig. 3B). Study 2 Subjects Thirty participants (15 female and 15 male) with no previous experience of laparoscopic surgery were recruited to the study. The mean age was years (range years). Fourteen of the participants reported corrected vision and all were right-hand-dominant. Apparatus MIST VR. The MIST VR system is as described above and shown in Figure 2B. The VR task is shown in Figure 4A and required subjects to grasp a virtual sphere with one instrument, transfer it to the instrument in the other hand and then place the sphere in the virtual box without touching the outer edges of the wire frame or releasing it before being accurately placed in the wire frame/box. The C 2013 Lippincott Williams & Wilkins

3 Annals of Surgery Volume 257, Number 6, June 2013 Transfer of Training (ToT) and Transfer Effectiveness Ratio (TER) FIGURE 1. Flow of trainees through Studies 1 and 2. TABLE 1. Demographic Details of the Surgeons Tested Age % of Surgeons VR 1st, N BT, N Country of Practice N %ofsample years 24.6% India years 40.0% Brazil years 23.0% Canada years 12.4% Colombia years 0.0% 0 0 France Total no. surgeons Germany Iceland % n n Mexico Right-hand dominant 92.3% South Africa Male 86.0% USA Venezuela Mean no. career MIS procedures 903 (SD = 1107) 1057 (SD = 1003) UK Mean no. annual MIS procedures 129 (SD = 111) 152 (SD = 89) Pakistan Total: 195 BT = box trainer, VR = virtual reality. FIGURE 2. The box-trainer cutting task (A) used as the test task; the minimally invasive surgical trainer virtual reality (MIST VR) setup (B), and the VR task (C) that was used to train subjects. training required subjects to perform the task with alternate hands, thus training performance ambidexterity. Box-Trainer. A cellular target area (170 mm 170 mm 28 mm) containing 12 boxes with a 4 3 matrix (each box 35 mm 52 mm 28 mm) was placed at a 45-degree angle in a conventional box-trainer shown in Figure 4B. The target area was viewed at a right angle through a Panasonic miniature camera (model no. WV KS 152) that was capable of producing on a Panasonic 14-inch color monitor (model no. TC-14SIR). The task was maintained in the center of the visual field. Ethicon nontoothed laparoscopic forceps were used C 2013 Lippincott Williams & Wilkins

4 Gallagher et al Annals of Surgery Volume 257, Number 6, June 2013 FIGURE 3. The mean (and standard error) number of correct (A) and the percentage of incorrect incisions (B) made by the untrained (n = 97) and the VR-trained (n = 98) surgeons. FIGURE 4. The experimental setup for performing the VR acquire, transfer, and place task (A), and the real-world box-trainer task that duplicated the VR task (B). Figure 4C demonstrates that the goal of the task in setup A (VR) and B (real world) were the same, ie, transfer the sphere from one instrument to the other and then place it in the box. An error was scored if the instruments holding the sphere touched the outer edges of the box. In the VR version, an error was scored if the sphere was not released by the instrument when placed in the box, and in the real-world version, if the sphere was dropped outside the target box. to grasp and manipulate circular sponge targets (diameter 12 mm). The forceps were linked through a processed timer to a digital timer counter (made by Electronic Developments). The digital counter recorded the number of errors made (the number of contacts made by the instrument on the target area). Time was recorded using a stopwatch. For both parts of the experiment, the monitor was positioned at eye level with the tools at standard surgical height between the participant and the monitor, as shown in Figure 2B. Procedure Each participant was randomly assigned to 1 of 2 conditions. Condition 1 Participants completed a simulated task on MIST VR shown in Figure 4A. The VR task required subjects to acquire a virtual sphere with one instrument, transfer it to the instrument in their other hand and then place it in a virtual wire frame, without dropping the sphere or touching the outer edges of the wire frame with the sphere as it was released. This was repeated 5 times for each hand. Time and error scores were recorded by the simulator. Participants then completed a real-life task on a box-trainer that was similar to the simulated task on MIST VR (Fig. 4B). The task involved grasping a sponge sphere, transferring it between instrument tips and placing it in one of the boxes contained on the cellular target area. The experimenter indicated which box the sphere was to be placed in. Like the MIST VR task, this also was repeated 5 times for each hand and similarly an error was scored and recorded by the digital counter if the subjects touched the walls or edge of the box. A stopwatch was used to record time and a digital timer counter recorded the number of errors made by the subject. The performance requirements for both the virtual and the real-world tasks in this study were very similar as demonstrated in Figure 4C. Condition 2 Participants completed only the real-life task on the box-trainer (as described above). No previous training was given to this group. Results Figure 5A shows the mean time taken to complete the laparoscopic task in the box-trainer by subjects who had previous training on MIST VR (Condition 1) and by subjects who only completed the task on the box-trainer and received no previous VR training (Condition2). Differences between the group means for time were examined using a 1-way ANOVA for independent groups design. Participants who had previously trained on MIST VR took significantly less time (ie, 26%) in completing the task on the box-trainer than those who had received no previous training (F (1, 28) = 12.71, P = 0.001). They also made 42% fewer errors (Fig. 5B) on the real-world task than subjects who had received no previous VR training (F (1, 28) = 7.69, P = 0.010) C 2013 Lippincott Williams & Wilkins

5 Annals of Surgery Volume 257, Number 6, June 2013 Transfer of Training (ToT) and Transfer Effectiveness Ratio (TER) FIGURE 5. Mean and standard error scores for the number of seconds it took for the standard (n = 15) and the VR-trained (n = 15) groups to perform the real-world task (A) and the mean number of errors made while performing the task (B). FIGURE 6. The time scores of the simulationtrained subjects (n = 15) on the VR task plotted against their time scores on the real-world boxtrainer task (A), their VR and real-world boxtrainer task errors scores plotted against each other (B), and their regression curves with 95% individual prediction intervals. The time taken by subjects in Condition 1 to complete the VR task and then the box-trainer task are shown on a scatterplot in Figure 6A. VR performance time was highly predictive of time to complete the real-world task, the Pearson product moment correlation coefficient of r = 0.964, which accounted for 92.4% (ie, adjusted r- squared = 0.924) of the variance (F (1, 13) = , P = ). The number of errors enacted during VR performance of the task was also highly predictive of real-world task performance (Fig. 6B) where r = 0.974, which accounted for 94.6% of the variance (F (1, 13) = , P = ). DISCUSSION Both studies show that VR simulation training transfers to improved performance on real-world tasks. The lowest proportion of performance improvement was observed in Study 1. Experienced surgeons made 7% more correct incisions after VR training. The largest proportion of performance improvement was observed for the laparoscopic novices (42%) in Study 2. Might subjects who had been trained on the box-trainer assessment task have performed equivalent or better than the VR-trained group in Study 2? In a previous prospective, randomized, and blinded study, we addressed this question. 20 The first group (the controls) received only the assessment (as in Study 2), the second-group subjects were trained on the task they were to subsequently be assessed on, and the third-group subjects were trained on the MIST VR. The amount of training that the second-group subjects had practicing the box-trainer test task was case-matched to the amount of time that the VR-trained group subjects took to complete their training. We found that the VR-trained group significantly outperformed the control and standard trained groups. Even though the performance improvement of the experienced surgeons in Study 1 was only 7% better than the control group, it was a statistically significant improvement. In contrast, the same group of surgeons demonstrated a 31% lower error rate in comparison with the control group. Both of these findings could indicate a ToT effect and/or an effect of procedure warm-up that has been shown to improve procedure performance. 29 The same pattern was also observed in Study 2, ie, VR training appeared to have its greatest impact by lowering the actual number of errors on the real-world task. This finding may be explained by the fact that both VR simulations specifically targeted the enactment of intraoperative errors during training. In both VR training simulations, trainees received immediate notification by the simulator after each error. This type of (formative) feedback allows for optimal learning due to the configuration of performance feedback during practice versus after a task is completed. Furthermore, data from Study 2 showed that performance on the VR task was highly predictive of performance on the real-world tasks. Study 2 probably gives the most robust estimate of TER for skill acquisition by novices. Although the task experienced surgeons were required to perform in Study 1 was novel to them, they had already acquired and honed their MIS surgical skills over many in-vivo procedures. Thus an improvement in their performance of only 7%, even though statistically significant, was not surprising. What was surprising was the considerable decrease that previous VR training had on error enactment (ie, 32%). For the novices, a rate of 26 42% TER 30 is considerable, particularly for skills at the start of a trainee surgeon s C 2013 Lippincott Williams & Wilkins

6 Gallagher et al Annals of Surgery Volume 257, Number 6, June 2013 FIGURE 7. Graph showing the probability of a bile duct injury as a function of surgeon experience with laparoscopic cholecystectomy (LC) [30]. It shows that the probability of a bile duct injury during the first 10 cases performed by the trainee was approximately 1 in 50, which had decreased to approximately 1 in 500 by their 50th case. The shaded area indicates the proportion of the learning curve that would be eliminated based on the data presented in this article (ie, from Study 2). learning curve where the patient is at the greatest risk of an error from the trainee s performance. Figure 7 shows the real-world implications of 26 40% TER to clinical performance. It shows extrapolated data from the Southern Surgeons Club, 31 which was one of the earliest studies to quantitatively demonstrate that the risk of a serious complication (ie, bile duct injury) during a laparoscopic cholecystectomy was directly associated with the operative experience of the trainee. The graph shows that during the first 10 laparoscopic cholecystectomies performed by the trainee, the risk of a bile duct injury is almost 1 in 50, whereas the risk at their 50th procedure had decreased to about 1 in 500. The graph also shows the impact of a modest 26% TER and a 42% ToT; both rates of transfer (if our hypothesis is correct) would eliminate the early part of the learning curve when the risk is the greatest to the patient. Thus, training on the simulator supplants this early learning experience, immaterial of procedure, assuming simulation training effectiveness. It should also be noted that the results reported here refer purely to a ToT effect, and whether the same findings would be observed for training to proficiency now needs to be investigated. We would however expect similar positive results. Simulation models for skills training are effective because they provide a context, an organizational structure, and focus to apply information retrieved from long-term memory Trainees can also practice the sequencing of psychomotor skills to complete the task effectively, efficiently, and safely with proximate highly structured formative performance feedback. Thus, trainees can learn and rehearse what to do and with which instruments. It is also important that they learn what not to do. From the evidence presented here and reports from clinical studies on VR simulation training, 13,14,16,17,35 37 it would appear that simulation training has its greatest impact on facilitating a reduction in performances that deviate from optimal task execution. Furthermore, the simulation training does not have to be virtual. In a study on advanced laparoscopic skill acquisition (ie, intracorporeal suturing for Nissen fundoplication), it was shown that training surgical residents to proficiency 38 on improvised simulation models significantly improved intraoperative performance on Nissen fundoplication. 39 Simulation models provide a vehicle for the configuration and delivery of a teaching and training curriculum and an essential part of this curriculum is performance feedback. Metricbased performance feedback is preferable because it is objective, transparent, reliable, and precise. 33 Although the simulation training can be undertaken on a bespoke model, it is preferable that the simulation model is virtual because of the automated nature of performance assessments that this affords. It has been demonstrated that bespoke simulation models are effective training platforms; however, the investment required from trainers to provide proximal formative performance feedback over an extended period of time to trainees is considerable Indeed this is one of the limitations of this study. Although we have demonstrated ToT and provided estimates of TER based on prospective, randomized experimental data, we have not provided an estimate of the cost implications. This is another metric for ToT that the surgical (and medical) community will require to help them make a business case for investment in VR simulation technology for training the next generation of surgeons. These trainee surgeons do not have the quantity of case exposure that previous generations of trainee surgeons had. 42 This means that they must be optimally prepared for the case exposure that they do get, which requires efficient and effective skills laboratory training to a level of proficiency before operating in vivo under the supervision of a master surgeon. CONCLUSIONS VR simulation training transfers to improved performance on real-world tasks and the transfer effect appears to be the greatest on the reduction in errors made. These results mean that VR simulation training can be used to supplant the early part of the procedure learning curve, which means better and safer operations. REFERENCES 1. Pellegrini CA. Surgical education in the United States: navigating the white waters. Ann Surg. 2006;244: Beall DP. The ACGME Institutional Requirements: What Residents Need to Know. J Am Med Assoc. 1999;281: Pickersgill T. The European Working Time Directive for doctors in training. BMJ. 2001;323: Satava RM. Virtual reality surgical simulator. The first steps. Surg Endosc. 1993;7: Salas E, Bowers CA, Rhodenizer L. It is not how much you have but how you use it: toward a rational use of simulation to support aviation training. Int J Aviat Psychol. 1998;8: Webster CS. The nuclear power industry as an alternative analogy for safety in anaesthesia and a novel approach for the conceptualisation of safety goals. Anaesthesia. 2005;60: King DR, Patel MB, Feinstein AJ, et al. Simulation training for a mass casualty incident: two-year experience at the Army Trauma Training Center. J Trauma. 2006;61: Rolfe JM, Staples KJ. Flight Simulation: Cambridge: Cambridge University Press; C 2013 Lippincott Williams & Wilkins

7 Annals of Surgery Volume 257, Number 6, June 2013 Transfer of Training (ToT) and Transfer Effectiveness Ratio (TER) 9. Thorndike EL, Woodworth RS. The influence of improvement in one mental function upon the efficiency of other functions. Psychol Rev. 1901;8: Roscoe SN, Williges BH, eds. Chapter 16: Measurement of Transfer of Training. Ames, IA: Iowa State University Press; 1980: Saks AM, Belcourt M. An investigation of training activities and transfer of training in organizations. Hum Resour Manage. 2006;45: Baldwin TT, Ford JK. Transfer of training: a review and directions for future research. Personnel Psychol. 1988;41: Ahlberg G, Enochsson L, Gallagher AG, et al. Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg. 2007;193: Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002;236: ; discussion Sutherland LM, Middleton PF, Anthony A, et al. Surgical simulation: a systematic review. Ann Surg. 2006;243: Grantcharov TP, Kristiansen VB, Bendix J, et al. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg. 2004;91: Larsen CR, Soerensen JL, Grantcharov TP, et al. Effect of virtual reality training on laparoscopic surgery: randomised controlled trial. BMJ. 2009;338:b Gallagher AG, Lederman AB, McGlade K, et al. Discriminative validity of the Minimally Invasive Surgical Trainer in Virtual Reality (MIST-VR) using criteria levels based on expert performance. Surg Endosc. 2004;18: Gallagher AG, Satava RM. Virtual reality as a metric for the assessment of laparoscopic psychomotor skills. Learning curves and reliability measures. Surg Endosc. 2002;16: Gallagher AG, Hughes C, Reinhardt-Rutland AH, et al. A case-control comparison of traditional and virtual-reality training in laparoscopic psychomotor performance. Minim Invasive Therapy Allied Technol. 2000;9: Pearson AM, Gallagher AG, Rosser JC, et al. Evaluation of structured and quantitative training methods for teaching intracorporeal knot tying. Surg Endosc. 2002;16: Seymour NE, Gallagher AG, Roman SA, et al. Analysis of errors in laparoscopic surgical procedures. Surg Endosc. 2004;18: Gallagher AG, McClure N, McGuigan J, et al. An ergonomic analysis of the fulcrum effect in the acquisition of endoscopic skills.endoscopy. 1998;30: Crothers IR, Gallagher AG, McClure N, et al. Experienced laparoscopic surgeons are automated to the fulcrum effect : an ergonomic demonstration. Endoscopy. 1999;31: Gallagher AG, Jordan-Black JA, O Sullivan GC. Prospective, randomized assessment of the acquisition, maintenance and loss of laparoscopic skills. Ann Surg. 2012;256: Gallagher AG, Smith CD, Bowers SP, et al. Psychomotor skills assessment in practicing surgeons experienced in performing advanced laparoscopic procedures. J Am Coll Surg. 2003;197: Wilson MS, Middlebrook A, Sutton C, et al. MIST VR: a virtual reality trainer for laparoscopic surgery assesses performance. Ann R Coll Surg Engl. 1997;79: Sutton C, McCloy R, Middlebrook A, et al. MIST VR. A laparoscopic surgery procedures trainer and evaluator. Stud Health Technol Inform. 1997;39: Kahol K, Satava RM, Ferrara J, et al. Effect of short-term pretrial practice on surgical proficiency in simulated environments.randomized trial JAmColl Surg. 2009;208: Taylor HL, Lintern G, Hulin CL, et al. Transfer of training effectiveness of a personal computer aviation training device. IntJAviatPsychol. 1999;9: The Southern Surgeons Club. A prospective analysis of 1518 laparoscopic cholecystectomies. N Engl J Med. 1991;324: Anderson JR. Acquisition of cognitive skill. Psychol Rev. 1982;89: Gallagher AG, O Sullivan GC. Fundamentals of Surgical Simulation; Principles & Practices London: Springer-Verlag; Van Sickle K, Gallagher AG, Smith CD. The effect of escalating feedback on the acquisition of psychomotor skills for laparoscopy. Surg Endosc. 2007;21: Palter VN, Grantcharov T, Harvey A, et al. Ex vivo technical skills training transfers to the operating room and enhances cognitive learning: a randomized controlled trial. Ann Surg. 2011;253: Sroka G, Feldman LS, Vassiliou MC, et al. Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room a randomized controlled trial.am J Surg. 2010;199: Prinssen M, Verhoeven ELG, Buth J, et al. A randomized trial comparing conventional and endovascular repair of abdominal aortic aneurysms. N Engl JMed. 2004;351: Gallagher AG, Ritter EM, Champion H, et al. Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg. 2005;241: Van Sickle K, Ritter EM, Baghai M, et al. Prospective, randomized, doubleblind trial of curriculum-based training for intracorporeal suturing and knot tying. J Am Coll Surg. 2008;207: Van Sickle K, Baghai M, Huang IP, et al. Construct validity of an objective assessment method for laparoscopic intracorporeal suturing and knot tying. Am J Surg. 2008;196: Van Sickle K, Smith B, McClusky DA, III, et al. Evaluation of a tensiometer to provide objective feedback in knot-tying performance. Am Surg. 2005;71: Bell RH, Jr., Biester TW, Tabuenca A, et al. Operative experience of residents in US general surgery programs: a gap between expectation and experience. Ann Surg. 2009;249: C 2013 Lippincott Williams & Wilkins