Vision for Vision. IEEE ICIA 2012, Shenyang, China. ter Haar Romeny, TU Eindhoven, Netherlands 1. Brain Inspired Computing.

Transcription

1 Medical workstations quickly become the super-assistants of radiologists and surgeons Vision for Vision Learning image analysis from the brain Prof. Bart ter Haar Romeny Eindhoven University of Technology 3D visualization Computer aided diagnosis Surgery planning Virtual endoscopy Tissue classification Interactive analysis etc. Millions of images, high-dimensional, anatomy / function, time Need for Computer-Aided Diagnosis (CAD). Computer-Aided Diagnosis The challenge Brain Inspired Computing How do we do it? Vision discoveries Mathematical models Applications Bev Doolittle: The forest has eyes - optical brain imaging - advanced MRI - for biological and computer vision - brain connectivity - computer-aided diagnosis The visual brain Multi-Scale Image Analysis Optical tract: retina thalamus cortex Biologically inspired computer vision bio-mimicking Netherlands 1

2 Hot issues: Brain connectivity Brain functionality Grand challenges in brain research Brain connectivity Structural (tractography) Functional (resting state fmri) Brain diseases US: Connectome China: Brainnetome Neurogenerative diseases (Parkinson, Alzheimer) Mental illness (schizophrenia, epilepsy) Brain function Neurocomputational models Large scale simulations Felleman & van Essen, 1991 The retina Rod and cone disks 150 million receptors 1 million ganglion cells Imaging the brain Rhodopsine MRI- anatomy Optogenetics Diffusion tensor imaging Functional MRI Voltage sensitive dyes Netherlands 2

3 Grand challenges in brain research Receptive fields Brain connectivity Structural (tractography) Functional (resting state fmri) Brain diseases Neurogenerative diseases (Parkinson, Alzheimer) Mental illness (schizophrenia, epilepsy) Brain function Neurocomputational models Large scale simulations Ganglion cell in retina Simple cell in cortex Mathematics of vision V1 receptive fields: These are multi-scale differential operators: The brains takes high order derivatives of incoming images, up to 4 th order. Models for V1 simple cells (Young 1991, Koenderink 1994) Automatic Polyp Detection The Gaussian derivative model for simple cells x L 0x Gx; s = L 0 x x Gx; s V1 can do high order differential geometry MIT AI Lab Rodieck, 2004 Multi-scale retina: measuring at many resolutions simultaneously toppoints graph theory Deep structure scale Retinal stack model (Lindeberg 1994, Koenderink 1990) 1. Multi-scale scale-space MR slice aorta The multi-scale stack gives the hierarchy intrinsic to images: topology Netherlands 3

4 Top-points generate a hierarchy in multi-scale computer vision Where is this sub-image? Forget about the pixels forget the pixels The translations we find correspond to the location of the objects in the scene. Robust subscene retrieval despite scaling, rotation and occlusion. C2 C1 Image content-based retrieval: image/video Google? Crucial property: invertibility From pixels to toppoints, from toppoints to pixels Multi-orientation Cells in the primary visual cortex are strongly orientation-specific Drastically improved reconstruction by reproducing kernel theory and Gelfand triples (Duits et al, TU/e, 2009). Now we can calculate with the top-points just as well as with the pixels. 2. Multi-orientation From Bonhoeffer and Grinvald Nature 353, , 1991 Netherlands 4

5 Optical dye response at different orientations (monkey V1) Calcium fluorescence microscopy Orientation sensitive cells are organized in a pin-wheel fashion Cortical column Pinwheel centers are extremely well organized Kenichi Ohki et al., Fukuoka, Japan Nature, Vol 442, , 2006 Connections exist between similar orientations to far away columns image rotating kernel orientation space Inverse orientation transform Fitzpatrick, Duke University, Nature 2002 Alexander & van Leeuwen, 2010 Fourier Transform / Inverse Fourier Transform: Sin / Cos Orientation Space 2D: New wavelet family Orientation Space 3D: Mathieu functions Exactly invertible orientation transform: z z n 2 n ( z, ) an ( ) e, z z n 2 n ( z, ) an ( ) e ; n 0 image rotating kernel orientation space Denoising of crossing fibers (collagen, tissue engineered heart valve) Different orientations are disentangled in the orientation space image orientation score Franken, Duits et al., TU/e, 2010 Netherlands 5

6 Perceptual grouping (Gestalt) from orientations: robust detection Gestalt laws Grand challenges in brain research Brain connectivity Structural (tractography) Functional (resting state fmri) Brain diseases Neurogenerative diseases (Parkinson, Alzheimer) Mental illness (schizophrenia, epilepsy) Brain function Neurocomputational models Large scale simulations Diffusion tensor : 3x3 matrix Image Science & Technology Eindhoven is specialized in the analysis and visualization of complex, multivalued medical data. Seven Images / Slice I 0 I xx I xy I xz Diffusion Tensor Imaging brain tractography [Prckovska, Peeters, Vilanova, ter Haar Romeny 2010] G D xx D xy D xz D D xy D yy D yz D xz D yz D zz Basser, PJ. et al., J Magn Reson 1994;103: I yy I yz I zz Diffusion MRI - tractography Grand challenges in brain research Brain connectivity studies: exploding field Prof. Denis Le Bihan TU/e Holst Medal 2010 Brain connectivity Structural (tractography) Functional (resting state fmri) Brain diseases Neurogenerative diseases (Parkinson, Alzheimer) Mental illness (schizophrenia, epilepsy) Brain function Neurocomputational models Large scale simulations Netherlands 6

7 DTI-based navigation For Deep Brain Stimulation Netherlands 7

8 Automatic electrode trajectory planning for deep brain stimulation Taking DBS path planning to a higher dimension Ellen Brunenberg Bram Platel Anna Vilanova Hyperdirect pathway? Structural connectivity of the STN (Brunenberg et al. 2011) JSTP project with China, Birgit Plantinga Structural connectivity of the STN (Brunenberg et al., PloSOne 2012) MRI: k-space: 3D spatial sampling, q-space: 3D orientational The problem is crossings, kissings, splittings, endings, etc. Glyphs: 8th order spherical harmonics We are developing the complex mathematics and interactive visualizations vist/e: Visualization tool for DW MRI (A. Vilanova - project leader) Netherlands 8

9 image rotating kernel orientation space image rotating kernel orientation space Different orientations are disentangled in the orientation space image orientation score Fourier Transform / Inverse Fourier Transform: Sin / Cos Orientation Space 2D: New wavelet family Orientation Space 3D: Mathieu functions Inverse orientation transform Exactly invertible orientation transform: z z n 2 n ( z, ) an ( ) e, z z n 2 n ( z, ) an ( ) e ; n 0 Temporal lobe epilepsy surgery Optic fibers running from the Lateral Geniculate Nucleus (LGN) to the primary visual cortex in the occipital lobe % of the patients seizure free after surgery (Uijl et al. 2008) However, a common complication is a partial / complete upper quadrantanopsia. Uijl et al. Prognosis after temporal lobe epilepsy surgery: The value of combining predictors, Epilepsia, 49(8): , Functional MRI SPM Analysis ConTrack Illustration of the probabilistic ConTrack algorithm Netherlands 9

10 Results of fmri-seeded DTI tractography Central (yellow), upper (blue)and lower (purple) retinal quadrant tracts of the optic radiation (Tax, Jacobs, Duits, Vilanova, Ossenblok et al. 2011). Gabor receptive fields Multi-spatial frequency stack Matthan Cahn, Multi-spatial frequency Analogy you might be familiar with Multi-spatial frequency Orientation space / score Franken, Duits, ter Haar Romeny 2008 Frequency space / score Non-invasive heart infarct quantification (Florack, van Assen, Bruurmijn, ter Haar Romeny 2011) Netherlands 10

11 IEEE ICIA 2012, Shenyang, China Counter-intuitive: add dimensions Innovative theoretical model Context, Gestalt Quantitative computer vision Huge application range Massively parallel implementation Brain vision: Lie Group Vision Philips 4 km Multiscale Multiorientation Multicolor Multivelocity Multi-spatial frequency Neusoft Eindhoven TU/e Shenyang NEU 4 km NEU Hospital Maastricht Hospital : TU/e NEU Neusoft - Philips: Multidisparity Contract for 20 years EU grant application 2012: 2.5 M 5 partners, 10 PhD Vision for Vision Philips Healthcare 4 km A large reason for blindness comes from diabetes: Philips High Tech Campus 4 km Blood vessels start to leak Eindhoven: 2011 World s Most Intelligent Region Eindhoven TU/e New vessels, angiogenesis Aneurysms, stenoses, drusen 100 km Maastricht University Hospital The retina gives an excellent view on the blood vessels Fundus Camera Diabetes in China 92.4 million adults in China age 20 or older (almost 10% of the country's total population) have diabetes, and million adults have pre-diabetes. The majority of diabetes cases are undiagnosed and untreated. China has edged ahead of India to become the country with the highest number of diabetes-afflicted people. Estimated medical costs for diabetes and its complications accounted for 18.2 percent of China's total health expenditure in Netherlands Screening for cataract and making retina fundus images is needed on a large scale. 11

12 Problem of Conventional Vessel Tracking Methods Animation Zoomed OS response Using Orientation Scores we can filter out certain orientations?! Erik Bekkers, BME, 2012 Animation Zoomed OS response Gabor Orientation Score Erik Bekkers, BME, 2012 Gabor Orientation Score employees Eye Care: He University and He Shi industry Screening province of Liaoning: 24 million people. BMIE School TU/e Netherlands 12

13 Qi Gong Prof. Kang Prof. He Prof. Romeny Fuli Zhang Han v Triest R. Evers, Focal Mengmeng Tong (granted) Jiong Zhang (granted) Chinese PhD candidates applying for CSC support 2012 Tos Berendschot, AZM Han JW van Triest, BMIE TU/e: Precision eye surgery robot Vision for Vision Postdoc / PhD application TU/e BME MSc students: May Wong Erik Bekkers Rick Philipsen TU/e-UM team in Maastricht VR eye surgeon prof. Marc de Smet Cannulation of a blood vessel in a 12-days incubated chicken egg Collaborations, interest, grant applications Genome Connectome USA: prof. David van Essen Xiangshan Meeting, Beijing 2011 China: prof. Tianzi Jiang Brains Unlimited Maastricht - prof. Rainer Goebel 30 million euro investment 3T, 7T, 9.4T whole body Utrecht: 7T - IDII -prof. Nick Ramsey - prof Peter Luijten Jülich Aachen -prof. Jon Shah - prof. Katrin Amunts 9.4 T whole body Micrometer tractography Netherlands: Institute for Diagnostic & Interventional Imaging (TU/e, Philips, Utrecht) Netherlands 13

14 IEEE ICIA 2012, Shenyang, China International Conference: Organization for Human Brain mapping Beijing, June 2012, Chair: Tianzi Jiang, Katrin Amunts Lie Group Vision Counter-intuitive: add dimensions Innovative theoretical model Context, Gestalt Quantitative computer vision Huge application range Massively parallel implementation Multiscale Multiorientation Multi-spatial frequency Multicolor Multivelocity Multidisparity EU Future Emerging Technology Flagship 2012 Human Brain Project 1 billion euro/10 years Chair: Henry Makram Brain Inspired Computing Conclusions - We have just begun with the understanding of the brain - Enormous potential for research Vision discoveries optical brain imaging Mathematical models - for biological and computer vision Brain &Vision Imaging Institute - TU/e, NEU, He - 1. Retina, 2. Connectivity, 3. Function Netherlands - Applications brain connectivity retina analysis heart deformation a new computer vision Chain: - Learning from the brain - Build the bio-computer - Apply in medical image diagnosis 14