Analysis of the Tumour Microenvironment with Computer Vision

Transcription

1 Analysis of the Tumour Microenvironment with Computer Vision Johan Lundin, MD, PhD Research Director Institute for Molecular Medicine Finland FIMM University of Helsinki and Guest Professor, Karolinska Institutet FIMM - Institute for Molecular Medicine Finland

2 Need for more personalized prediction of outcome and tailored treatment 2

3 Patient with breast cancer Patient profile Age 55 years No cancer in lymph nodes Tumor size 1.5 cm Ductal carcinoma Grade 3 Estrogen receptor positive Progesterone receptor positive HER2 expression borderline HER2 in situ show amplification Gene expression data: Luminal B Pathway: proliferative Data from different sources 3

4 Patient with breast cancer Patient profile Age 55 years No cancer in lymph nodes Tumor size 1.5 cm Ductal carcinoma Grade 3 Estrogen receptor positive Progesterone receptor positive HER2 expression borderline HER2 in situ show amplification Gene expression data: Luminal B Pathway: proliferative What do these factors have in common? 4

5 Patient with breast cancer Patient profile Age 55 years No cancer in lymph nodes Tumor size 1.5 cm Ductal carcinoma Grade 3 Estrogen receptor positive Progesterone receptor positive HER2 expression borderline HER2 in situ show amplification Gene expression data: Luminal B Pathway: proliferative lobular? moderate? +? weak? ++? 5%? Human observer based readout by microscopy! 5

6 Digital Microscopy Revolution lobular? moderate? +? weak? ++? 5%? No of cells: IHC pos: Percentage: 22.9 % Tissue types: Epithelium: 34% Stroma: 60% Descriptive microscopy Quantitative microscopy

7 What would be needed? Tissue analysis that is Quantitative Repeatable Consistent Robust Able to capture spatial information Allows multiplexed analysis Capable of high-throughput

8 Time for next generation of microscopy

9 Digitization of microscopy samples Images captured at high magnification Up to image tiles Glued or stitched digitally and compressed Up to terapixel sized images

10 Digital Microscopy Slides - Size Comparison Football field 100 x 70 m Tissue sample 10 x 7 mm Football 25 cm Small blood vessel 25 um

11 Web- and cloud microscopy WebMicroscope Network Disclaimer: Fimmic Oy, J Lundin, founder and co-owner

12 The Multitouch Microscope Disclaimer: Spin-off Fimmic Oy, J Lundin, founder and co-owner

13 Digital microscopy platform in the cloud Available Soon (Q2/2015) Disclaimer: Fimmic Oy, J Lundin, founder and co-owner

14 Webmicroscopy for Cancer Analytics: Readout, analysis and prediction modelling IHC, FISH, CISH Visual scoring Morphology Image analysis Intergrated online tools Reports and models 14

15 Image analysis methods for pathology Quantification of stained cells E.g. immunohistochemistry, in situ hybridization Multiplexed analysis of protein and gene expression E.g. multiplexed IHC, analysis co-localization Classification based on pattern recognition E.g. segmentation of tissue into clinically meaningful entities 15

16 Quantitative image analysis of single colour immunohistochemical staining (CD8)

17 Automated assessment of Ki-67 protein expression in breast cancer Juho Konsti HR 1.77, P< HR 2.34, P< Konsti J, BMC Clin Pathol. 2011;11:3.

18 Image analysis of immune response to oncolytic viral therapy Lymphocyte infiltration defined for biopsies from ovarian cancer before and after oncolytic viral treatment. Riku Turkki After therapy Before therapy FIMM National Network Project. Collaboration with Dr. A Hemminki and Oncos Theraputics

19 Fluorescence signal counting Juho Konsti Ratio (Red/Green): 222/38= Konsti et al. J Clin Pathol. 2008;61:

20 Digital multiplexed analysis ER Ki-67 H&E phh3 PR ER H&E Ki-67 Digital Re-alignment Yinhai Wang Hybrid image 20

21 Digital multiplexed analysis H&E Hybrid image Riku Turkki IF Image registration Staining detection Digital overlay Ck8 (epithelium) CD45 (leukocytes) Hoechst (nuclei) Stroma Epithelium Leukocytes in stroma Leukocytes in epithelium

22 Multiplexed IHC + HE in the same section H&E CD8 CD4 PanCK Nuc Case study - Breast cancer with lymphocyte infiltration multiplexing + H&E Teijo Pellinen and Sami Blom 22

23 Multiplexed IHC + HE in the same section H&E CD8 CD4 PanCK Nuc Case study - Breast cancer with lymphocyte infiltration multiplexing + H&E Teijo Pellinen and Sami Blom 23

24 Tumor microenvironment Hanahan and Weinberg. Cell 2011;144:

25 Automated morphological segmentation Tools from geoinformatics... adapted for tissue analysis..and run server-side for high-troughput analysis

26 Progress in Deep Learning of Visual Scenes Karpathy et al, Computer Science, Stanford University 26

27 Aims: Deep Learning applied to Tumor Tissue Samples Tumor cells Dividing cell Blood vessel Fibroblasts Endothelial cell 27

28 Study on texture analysis of tumor microenvironment in colorectal cancer To develop an algorithm, based on texture analysis, that can differentiate stroma from epithelium Epithelium Stroma

29 Differentation of tumor epithelium from stroma by a texture classifier A B C Nina Linder Riku Turkki LBP LBP A morphological classifier was trained using 80 samples Texture features were extracted and a statistical classifier optimized 29

30 Stroma epithelium segmentation Nina Linder Riku Turkki Stroma Colorectal cancer

31 Stroma epithelium segmentation Nina Linder Riku Turkki Epithelium Stroma Colorectal cancer

32 Tumor epithelium vs stroma classification Nina Linder Riku Turkki The accuracy of the LBP classifier was in 576 test cases The agreement percentage was 95 % The sensitivity was 95% and the specificity was 94% Linder N et al Diagn Pathol 2012;7:22 32

33 Tumor viability classifier Riku Turkki Necrosis

34 Tumor viability classifier Riku Turkki Necrosis Vital epith

35 Ongoing study: Tumor viability classifier Nina Linder Riku Turkki Necrosis Vital epith Collaboration with Prof. K Alitalo Lung cancer xenografts in mouse 35

36 Tumor viability classifier Vital Necrotic = borderline between live and necrotic area drawn by expert

37 Tumor viability classifier Necrosis Vital cells = borderline between live and necrotic area drawn by Presentation name or name of the guest expert

38 Tumor viability classifier Necrosis Vital Necrotic Vital cells = borderline between live and necrotic area drawn by expert

39 Assessment of tumor viability

40 Scan Unsupervised clustering of tissue based on texture features

41 Scan Ville Ojansivu Unsupervised texture clusters: Preliminary results

42 Unsupervised texture-based clustering: Preliminary results 42

43 Preliminary results: Unsupervised texturebased clustering of breast cancer tissue Topic 9 Topic 11 Breast Cancer Specific Survival HR 5.01 P = 0.01 Topic 9 Topic 11 ER Topic 9 Topic 11 HER2 Bag-of-words analysis of SIFT features in 1,934 cases with breast cancer from the FinProg Series 43

44 Scan Analyze whole tumor sample for heterogeneity

45 Challenges in tissue image analysis Sample variability Ischemic time, fixation, sectioning, staining protocol etc Calibration of algorithms Need for carefully annotated reference samples and internal controls Virtual quality assurance samples needed Computational resources Pre-processing after scanning, compute cloud 45

46 Who decides what is a cell/nuclei? = agree? = agree? 46

47 Blur Partial Overlapping 47

48 Tissue image analysis in the cloud Image size 20,000 x 30,000 pixels 48

49 Future: Image matching with large databases Data Transfer Internet Oncoline database Secure Compute cloud Scanner Tissue Sample Decision support Case-match

50 Conclusions 1 Feasible to perform readout of immunostaining and in-situ hybridization with image analysis Also possible to classify and segment tissue according to morphology, e.g. epitheliumstroma, non-viable tissue 50

51 Conclusions 2 Multiplexed analysis likely to improve in the near future Reference samples (e.g. ground truth) needed, both for development and assessment of image analysis algorithms Expert annotations (with controlled vocabularies and terminologies) important for development of image analysis algorithms 51

52 Supported by: Lundin Group

53 23/03/2015 FIMM - Institute for Molecular Medicine Finland