Autonomous Diagnostic Imaging Performed by Untrained Operators using Augmented Reality as a Form of Just in Time Training PROPOSAL TEAM PI: David S. Martin, MS, Wyle Science, Technology, and Engineering (STE), Johnson Space Center (JSC) Cardiovascular and Vision Laboratory Co I: Lui Wang, PhD, Software Robotics and Simulation Division, NASA, Johnson Space Center Co I: Steven S. Laurie, PhD, Wyle STE, JSC Cardiovascular and Vision Laboratory Co I: Michael B. Stenger, PhD, Wyle STE, JSC Cardiovascular and Vision Laboratory Co I: Stuart M. C. Lee, PhD, Wyle STE, JSC Cardiovascular and Vision Laboratory Statistical Consultant: Alan Feiveson
Ultrasound Imaging Versatile, safe, reproducible, relatively small, real time Second most widely used imaging modality in U.S. Highly operator dependent
Research Studies Using Ultrasound on the International Space Station (ISS) Technology demonstration ADUM Braslet Integrated Cardiovascular Sprint Ocular Health Cardio Ox Fluid Shifts Vessel Imaging (French) Vascular Echo (Canadian) Intervertebral Disc Medical Operations ocular
Diagnostic Applications Analog Populations In nuclear submarine crews and Antarctic crews that winter over, the liklihood of requiring emergency medical care is between 5% and 7% chance per person per year
Space Medicine Care Provider Crew medical officer Usually not a physician Ultrasound training measured in hours as opposed to days Minimum 2 years needed in the US + 600 scans Questionable retention of training Lack of trained assistants
Ultrasound Remote Guidance Remote guidance uses real time two way audio communication combined with down linked video from the ultrasound system to allow a ground based expert to guide an inexperienced operator through an ultrasound study
Research Studies Using Ultrasound on ISS Using Remote Guidance Technology demonstration ADUM Braslet Integrated Cardiovascular Sprint Ocular health Cardio Ox Fluid Shifts Vessel Imaging (French) Vascular Echo (Canadian) Intervertebral Disc Med Ops (eye) Scans guided by Cardiovascular and Vision Laboratory Cardiac 219 Stress echo 14 Vascular 66 Transcranial 75 Musculoskeletal 61 Abdominal 17 Total: 452
Exploration Class Missions
Types of Just In Time Training (JIT) Needed to facilitate autonomous ultrasound during exploration class missions operation Current paradigm: Review of material prior to exam Usually on a laptop Need to provide more timely, information rich guidance Augmented reality (AR)
Research Studies Using Ultrasound on ISS Using Remote Guidance We have applied these same remote guidance techniques to remotely guide crew members in the collection of data with other specialized equipment including Optical Coherence Tomography.
Objectives Develop AR program to provide audio/video instructions for ultrasound and OCT procedures Test the performance of inexperienced operators using two modalities with specific reference to medical operations and crew health care Compare performance using traditional training vs. AR guidance Compare performance of inexperienced vs. experienced operators
Specific Aims Specific Aim 1 Develop an audio visual tutorial using augmented reality glasses that guides non experts through an abdominal trauma ultrasound protocol. Specific Aim 2 Develop an audio visual tutorial using augmented reality glasses to guide an untrained operator through the setup and acquisition of an OCT protocol.
Specific Aims Specific Aim 3 Evaluate the quality of abdominal ultrasound and OCT images acquired by untrained operators using augmented reality guidance during ground based testing with images acquired using current JIT techniques. Hypothesis: Ultrasound and OCT images acquired using augmented reality guidance will be superior to those acquired using current JIT techniques, as determined by independent expert observers, and require less time to perform. Specific Aim 4 Compare the time and costs required to develop tutorials and complete imaging studies using augmented reality tutorials with the time and costs of to current JIT practices to determine whether time efficiencies exist. Hypothesis: Although the time required to develop the two types of tutorials will be similar, images acquired by untrained operators using augmented reality guidance will require less time to acquire as compared to untrained operators using current JIT training (reviewing teaching aides immediately prior to imaging).
Previous Experience in Facilitating Autonomous Ultrasound with Inexperienced Operators Virtual guidance replaces the remote guider with a digitally stored tutorial that can be viewed in real time through video glasses when remote guidance is not feasible. Innovation grants 2010 Virtual Ultrasound Guidance for Inexperienced Operators 2011 IR&D Virtual Guidance for Medical Ultrasound Procedures on Exploration Class 2013 Augmented Reality Aided by Image Recognition Software to Assist with Autonomous Ultrasound Exams
Pulsed Wave Doppler of Common Carotid Experienced Operator Inexperienced Operator
Transverse Carotid and Jugular Experienced Operator Inexperienced Operator Images obtained by experienced and inexperienced operators were scored by a cardiologist 8/10 inexperienced operators captured diagnostically adequate images. Martin et al. 2012
Ophthalmic Virtual Guidance
Augmented Reality Microsoft HoloLens is the first fully untethered, holographic computer, enabling you to interact with high definition holograms in your world
Augmented Reality Guidance Audio/visual program that will guide inexperienced operators through and ultrasound and OCT scans Audio instruction for scan technique and equipment settings Section of common scanning mistakes for each probe location with audio instructions to optimize Default sequence of operation for user to follow, but can be overridden with user control
Hololens Correct probe placement Anatomical image Resulting ultrasound image Step in protocol Can recognize markers on the skin and provide probe placement information Procedure progress is user driven.
Augmented Reality Guidance (OCT) AR system will provide an augmented reality overlay of the target image and recommend corrective adjustments in real time to acquire a quality image for analysis
Ultrasound FAST Exam (Focused Assessment with Sonography in Trauma) Initial imaging test of choice for trauma care in the United States and is part of the Advanced Trauma Life Support (ATLS) protocol developed by the American College of Surgeons. It is nearly perfect for recognizing intraperitoneal bleeding in hypotensive patients who need an emergent laparotomy and for diagnosing cardiac injuries from penetrating trauma. More sensitive and faster than X ray for detecting hemothorax and pneumothorax Obtain images from multiple locations: e.g., cardiac, pulmonary, abdominal Ann Surg,1998;228:557 67 J Trauma,1996;41:815 20 J Trauma,2004;57:288 95.
FAST Protocol Right coronal abdominal image of Morrison s pouch (the potential space between the right kidney and liver) Right coronal image of diaphragm and pleural space Left coronal to evaluate the space around the spleen Subcostal view of the heart to assess for pericardial fluid Parasternal long axis of the heart, also to assess for pericardial fluid Right lung to assess for pneumothorax or hemothorax
Optical Coherence Tomography (OCT) Particularly adept at imaging VIIP related change in the eye, allowing the distinctive layers of the retinal to be visualized Valuable resource for missions of longer duration to monitor eye health Has been used on ISS for both Medical Operations and research (Fluid Shifts study)
OCT Protocol Align patient pupil with center of scan Adjust distance to patient to get the best fundus image Optimize the scan signal strength and image quality using position, focus or
Hypothesis Ultrasound FAST exams performed with AR guidance will be of superior quality to those relying on review of material prior to the procedure. Ultrasound FAST exams performed with augmented reality guidance will take less overall time than those relying on review of steps and images prior to the procedure. The difference in image quality scores of the two OCT groups may not be significantly different, due to the features of the OCT system, however, the time to generate those images will be shorter when using augmented reality guidance.
25 subjects using laptop based instruction prior to scanning 25 subject/operators will use AR guidance Subjects will be gender balanced and represent national averages of race and ethnicity 4 radiologists will rate the quality of ultrasound FAST images 4 ophthalmologist will rate the quality of OCT images Methods
Schedule
Laptop Based Training Will provide optimal images and photos of probe placement Criteria of satisfactory image
Image Raters Will be blinded to the origin of the study Independent from technology development Ocular: Doheny Eye Institute, UCLA, Doheny Image Reading Center one of the largest and most active ophthalmic reading centers in the nation, and among the top three or four in the world Ultrasound: University of Vanderbilt Medical Center, Department of Radiology
Ultrasound Scoring Criteria 1 2 Represents an inability to position the probe in the correct location, or having the probe in the correct location but not pointed in the correct direction to image the anatomy in question. 3 4 Having the probe in what is typically the correct location but still unable to adequately display the anatomy due to overlying bowel gas, subject specific or some other limitation (failure to adapt to patient anatomy). 5 6 Having the probe in the correct location and providing some imagery of the required anatomy, but not of sufficient quality due to inadequate equipment settings or inadequate recording of images. 7 8 An adequate image would be of high enough quality to be diagnostically adequate. In the case of a FAST exam, an adequate image would depict the presence or absence of free fluid in the view. 9 10 An excellent image would be of clarity and quality approaching that of expertly acquired images.
OCT Scoring Criteria 1 2 The retina cannot be identified in the scanning window and thus no useful information is available. 3 4 The image is noisy and/or the retina or choroid is cut off the screen due to too high or too low positioning of the B scan in the imaging window. Due to the noise, the retinal layers cannot be clearly discerned, though the presence of the retina in the image can be identified. 5 6 The retinal layers can be seen but the external limiting membrane (ELM) cannot be visualized. The choroid is partially visible. 7 8 An adequate image would be of high enough quality to be diagnostically adequate. The retinal layers will be well seen including the external limiting membrane (ELM). The anterior portion of the choroid may be visible, but not its full extent. 9 10 An excellent image would be of clarity and quality approaching that of expertly acquired images. All retinal layers (including ELM) are clearly seen and the full extent of the choroid can be visualized.
Survey Total time required for each study Time to complete each section Degree of confidence in the training, specific to each section Ability to understand instructions Impressions on specifics of the AR or laptop based program
Summary AR will improve the capability of exploration class mission astronauts to perform research or diagnostic ultrasound studies or OCT autonomously, if needed. Other populations such as the military or remote environments without trained personel may also benefit from this technology development.
Back up slides
HERA There have been 10 missions since the start of its use at JSC with no more than 4 subjects per mission. Future missions are trending toward longer durations, meaning fewer opportunities for subjects. The schedule would make it impossible to us the HERA facility without drastic reduction in the number of subjects.
Hardware Microsoft Hololens