Development of Simulation Tools Software

Transcription

1 Development of Simulation Tools Software Vincent Luboz Department of Biosurgery and Surgical Technology Imperial College London BSc VR Surgical Simulation Software Slide 1 Contents Virtual Reality VR Surgical Simulation Modelling Collision detection and collision response Performance metrics Examples Summary and Conclusions BSc VR Surgical Simulation Software Slide 2

2 Contents Virtual Reality VR Surgical Simulation Modelling Collision detection and collision response Performance metrics Examples Summary and Conclusions BSc VR Surgical Simulation Software Slide 3 Classic VR Examples? BSc VR Surgical Simulation Software Slide 4

3 Classic VR Examples Flight Simulators (here from Airbus company) BSc VR Surgical Simulation Software Slide 5 Classic VR Examples Video Games (here from Medal of Honor 3 and 4) BSc VR Surgical Simulation Software Slide 6

4 What is Virtual Reality? Virtual reality is a way for humans to visualize, manipulate and interact with computers and complex data. The user interacts with the virtual world and can directly manipulate objects in it. The technology attempts to immerse the user into a computer generated world representing the reality. BSc VR Surgical Simulation Software Slide 7 What is Immersion? Immersion in an environment (real or virtual) is related to the quantity and quality of sensory data from that environment. Immersion in a virtual environment can be gauged by the extent to which the computer shuts out sensations from the real world and accommodates different sensory modalities (vision, audio, haptic). BSc VR Surgical Simulation Software Slide 8

5 Examples Immersa Desk Window on world Desktop VR System Immersive systems (HMD) BSc VR Surgical Simulation Software Slide 9 Immersive systems (CAVE) Telepresence Mixed reality Architecture of a VR system Rendering Engine (easy) Graphical Display (easy) Computer Models Geometric (anatomical) (easy) Physical (biomechanical) (very difficult) Tactile Feedback (very difficult) Haptics Mechanical Interface (moderately easy) User Audio interfaces can also be included BSc VR Surgical Simulation Software Slide 10

6 Contents Virtual Reality VR Surgical Simulation Modelling Haptics Collision detection and collision response Examples Summary and Conclusions Hands-on session BSc VR Surgical Simulation Software Slide 11 Elements of a VR Surgical Simulation? Rendering Engine (easy) Graphical Display (easy) Computer Models Geometric (anatomical) (easy) Physical (biomechanical) (very difficult) Tactile Feedback (very difficult) Haptics Mechanical Interface (moderately easy) User Audio interfaces can also be included BSc VR Surgical Simulation Software Slide 12

7 Elements of a VR Surgical Simulation Computer Model (geometrical or physical) Liver Hips Vessels BSc VR Surgical Simulation Software Slide 13 Elements of a VR Surgical Simulation Computer Model (geometrical or physical) Rendering Engine (interactions, collision detection and collision response) Biopsy Catheterization

8 Elements of a VR Surgical Simulation Computer Model (geometrical or physical) Rendering Engine (interactions, collision detection and collision response) Interface (haptics) BSc VR Surgical Simulation Slide 15 VSP for catheterization Phantom for biopsy Elements of a VR Surgical Simulation Computer Model (geometrical or physical) Rendering Engine (interactions, collision detection and collision response) Interface (haptics) Performance metrics (measuring the outcome of a virtual surgery and the performance of the trainee) BSc VR Surgical Simulation Software Slide 16

9 Simulating Minimally Invasive Procedures Involves interaction of several systems Hardest to achieve accuracy. A topic of great interest due to the potential commercial value. Example: Gall Bladder (producing bile) and Surrounding organs procedures (Laparoscopic Cholecystectomy) BSc VR Surgical Simulation Software Slide 17 Sample Commercial Simulators Lapsim Laparoscopic Simulator BSc VR Surgical Simulation Software Slide 18 Simbionix Lap Mentor

10 BSc VR Surgical Simulation Software Slide 19 Immersion Medical Sigmoidoscopy Simulator Patient Specific Simulation BSc VR Surgical Simulation Software Slide 20 Original CTA

11 Patient Specific Simulation BSc VR Surgical Simulation Software Slide 21 Segmentation of the vessels Patient Specific Simulation R. CC L. CC R. SC EIA Brach AA IA Fem IA Fem L. SC EIA RA CA SMA RA Arch of the aorta (AA), Brachiocephalic (brach) artery, Right and left common carotid (RCC and LCC), Right and left subclavian artery (RSC and LCC), Renal arteries (RA), Celiac artery (CA), Superior mesenteric artery (SMA), Internal (IA) and external (EA) iliac arteries, And femoral artery (fem). BSc VR Surgical Simulation Software Slide 22

12 Patient Specific Simulation Non-pathological Aorta Aortic Aneurysm Aortic Dissection BSc VR Surgical Simulation Software Slide 23 Patient Specific Simulation Model of liver vasculature from fluoroscopy BSc VR Surgical Simulation Software Slide 24

13 Simulation Complexity - + No deformation - easy - Bronchoscopy - Colonoscopy - Upper GI endoscopy -ERCP - Laryngoscopy - Laparascopic Cholesectomy Large and Complex Deformations BSc VR Surgical Simulation Software Slide 25 Contents Virtual Reality VR Surgical Simulation Modelling Collision detection and collision response Performance metrics Examples Summary and Conclusions BSc VR Surgical Simulation Software Slide 26

14 Geometric Modelling Geometric modelling can be used to reproduce anatomy Geometric models can be derived from scan data (CT, MR, US ) and be patient specific Applications: walk through the peritoneum, lungs, colon BSc VR Surgical Simulation Software Slide 27 Extracting Geometric Models Segmentation CT Data Set Reconstruction Mesh Generation BSc VR Surgical Simulation Software Slide 28

15 Creating Walk through Render Mesh Model Texture BSc VR Surgical Simulation Software Slide 29 Walk Through Virtual Colonoscopy University of Hamburg BSc VR Surgical Simulation Software Slide 30

16 Spherical model of the Colon Easy to create effects such as - spasm, - inflation of the colon Anatomy can be constrained to be representative of a real colon. BSc VR Surgical Simulation Software Slide 31 Limitation of Geometric Model Triangulated models are easy to obtain, but are difficult to deform. (large number of triangles) They are suitable for rigid structures (bones, larynx, lung, trachea etc) They are less efficient for deformable organs (colon, gall bladder, skin ) and instruments BSc VR Surgical Simulation Software Slide 32

17 Modelling Problem - Colonoscopy Turn View Direction Turn View Direction Turn Rigid Co llar BSc VR Surgical Simulation Software Slide 33 View Direct ion In a real colonoscopy we might find If we twist does the loop tighten or the image rotate? BSc VR Surgical Simulation Software Slide 34

18 There are many possibilities BSc VR Surgical Simulation Software Slide 35 Biomechanical Modelling One solution to the previous problem is biomechanical modelling - however this is fraught with problems. - What happens when you push the colon wall? - What dynamic properties does the endoscope have? - What frictional forces are acting? BSc VR Surgical Simulation Software Slide 36

19 Behavioural Models For many endoscopic procedures we cannot see the deformation that is caused. Thus simulators can get away by modelling only the behaviour of the instrument. BSc VR Surgical Simulation Software Slide 37 Can we simulate real deformations? Answer Almost there but not quite yet! Certainly not in real time with a high degree of accuracy BSc VR Surgical Simulation Software Slide 38

20 Modelling deformations The most accurate modelling tool uses the finite element method. In the finite element method a surface or volume is divided into small parts called elements. Element BSc VR Surgical Simulation Software Slide 39 Node Face Typical 3D Elements BSc VR Surgical Simulation Software Slide 40

21 Finite Element Model of the Liver Geometry extracted from scan data. Boundary conditions defined initially. Material properties specified physically. BSc VR Surgical Simulation Software Slide 41 Problems Real time finite element solutions are possible for small numbers of isotropic linear elastic elements. Muscle tissue is not isotropic or linear elastic, and large numbers of elements are required. BSc VR Surgical Simulation Software Slide 42

22 Possible solutions Precomputation and encoding of a large number of accurate solutions using statistical shape modelling. Restriction of the number of degrees of freedom (only appropriate for certain techniques). Use of hierarchical approach to solve the local deformation and propagate accordingly. BSc VR Surgical Simulation Software Slide 43 Basic Interactions Grabbing and pulling Pressing and dragging BSc VR Surgical Simulation Software Slide 44

23 Contents Virtual Reality VR Surgical Simulation Modelling Collision detection and collision response Performance metrics Examples Summary and Conclusions BSc VR Surgical Simulation Software Slide 45 Collision Detection/Response? BSc VR Surgical Simulation Software Slide 46

24 Collision Detection/Response? BSc VR Surgical Simulation Software Slide 47 Collision Detection/Response Position Orientation Contact Information Collision Detection Object Database Geometry Force Torque Collision Response Material BSc VR Surgical Simulation Software Slide 48

25 Collision Detection The algorithms combine the information to obtain the positions in Cartesian inside the virtual environment. The algorithm uses position information to find collisions. The algorithm reports the resulting degree of penetration or indentation. BSc VR Surgical Simulation Software Slide 49 Collision Response The algorithm computes interaction forces between objects involved in a collision. The algorithm sends interaction forces to the control algorithms. BSc VR Surgical Simulation Software Slide 50

26 Contents Virtual Reality VR Surgical Simulation Modelling Collision detection and collision response Performance metrics Examples Summary and Conclusions BSc VR Surgical Simulation Software Slide 51 Performance metrics Evaluate the performance of the user. Give feedback on the task. Point to areas that the user needs to improve. Derived from a task analysis. BSc VR Surgical Simulation Software Slide 52

27 Performance metrics Common metrics: - Outcome of the procedure. - Time spent on the procedure. - Accuracy of the task. - Respect of the scenario. - Comparison with an ideal procedure. - BSc VR Surgical Simulation Software Slide 53 Contents Virtual Reality VR Surgical Simulation Modelling Collision detection and collision response Performance metrics Examples Summary and Conclusions BSc VR Surgical Simulation Software Slide 54

28 VR Simulator for Liver Biopsy DICOM DATA Interactive Segmentation Framework SIMULATOR Raw image data Labelled Segmentation Surface mesh Tetrahedral mesh BSc VR Surgical Simulation Software Slide 55 Deformable Modelling Mesh Generation Pipeline BSc VR Surgical Simulation Software Slide 56

29 VR Simulator for Endovascular operation BSc VR Surgical Simulation Software Slide 57 Contents Virtual Reality VR Surgical Simulation Modelling Collision detection and collision response Performance metrics Examples Summary and Conclusions BSc VR Surgical Simulation Software Slide 58

30 Summary We have seen: The principles of VR surgical simulator software design The concepts of modelling with regard to surgical simulation programs How the mechanism of collision is handled by VR simulators The basic knowledge and examples of software platforms used for VR simulator BSc VR Surgical Simulation Software Slide 59 Conclusion VR is a powerful technique for visualisation, interaction and planning. Surgical simulation systems for training and assessment are increasingly relevant. Still improving: - In accuracy (e.g.: real organs elasticity) - In computing time (e.g.: using computer graphic card) - In realism (e.g.: quality of the texture and the rendering) - The haptic rendering allows adding sense of touch to the visualisation immersion BSc VR Surgical Simulation Software Slide 60

31 Development of Simulation Tools Software Vincent Luboz Department of Biosurgery and Surgical Technology Imperial College London BSc VR Surgical Simulation Software Slide 61