Mesh Learning for Classifying Cognitive Processes

Transcription

1 Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman Vural* * Middle East Technical University, Ankara ** Koc University, Istanbul *** Google

2 Motivation: Can we model the brain activities measured by fmri as a machine learning system? Algorithms that can learn Human Machine 2

3 Focus: Design a classifier to model the distributed patterns of activity in memory Machine Human _ 1.Training 2.Test 1. Encoding: subject studies objects from a category 2. Retrieval: subject is asked to recognize a test object 3

4 (Öztekin & McElree, 2007; Öztekin et al., 2009; Öztekin & Badre, 2011) fmri Data Acquisition 4

5 Image samples 5

6 fmri dataset 10 semantic categories: animals, colors, furniture, body parts, fruits, herbs, clothes, chemical elements, vegetables and tools. Dataset: 24 samples /category 240 training test samples from the encoding and retrieval phase and Number of voxels: Memory: 8142 Whole Brain:

7 fmri intensity values of a voxel 7

8 Spatial Distribution of Voxel Intensities for a time instant t 8

9 fmri intensity values as a function of time State of the art performance ~25-30% (e.g. Öztekin & Badre, 2011) 9

10 Spatio-temporal Distribution of Voxel Intensities 10

11 Voxel intensity values for 10 class at 10 neighboring voxels 11

12 Difference of intensity values between two neighboring voxels 12

13 Neurons are massively interconnected Relationships among the voxels are more discriminative then the individual voxel intensity values to represent a certain category Need to model the relationships among the voxels 13

14 A Local Mesh Model: A voxel is represented in a neighborhood system M p v ti, sj v ti, sj p, ai, j, k A,, i j i, j, k i k i, j s v t s a v t s k p i j i k j k j l i l 1 v t, s { v t, s : s s s s, v t, s D } 1 1 p v ti, sj { v ti, sk p v ti, sj : sj sk sj sl, v ti, sl p v ti, sj } c 14

15 Local Relational Features: LRF a a a... a i, j i, j,1 i, j,2 i, j, p ε 2 ij A a a... a j 1, j 2, j N, j T

16 Mesh Learning with Spatial Neighborhood 16

17 Performance of Mesh Learner with Spatial Neighborhood 10 class classification performances using 8142 voxels. ICA: Independent component analysis PCA: Principal Component Analysis KPCA: Kernel Principal Component Analysis 17

18 Experiments on Mesh Learner with Spatial Neighborhood Single voxel performance for 10 classes. 18

19 19

20 20

21 Discussion on Mesh Learning with Spatial Neighborhoods Spatial neighborhood with L2-norm implies anatomical surroundings of a voxel; which may not be the case in cognitive process. Selecting the optimal value of p is not validated and introduced as a user parameter. Employ functional connectivity. Find voxels which are highly correlated to other voxel. p changes for each voxel. 21

22 Mesh Learning with Functional Neighborhood 1. Need to define functional connectivity among the voxels 2. Define Functional Neighborhood 3. Apply functional neighborhood to k-nn to and select k-functionally-closest neighbors which implies coupled-activation in cognitive process 22

23 Functional Connectivity Statistical association or dependency among the time series of voxels 23

24 Modeling Methods of Brain Using Connectivity 1. Find correlation between time series of two voxels, using a correlation metric. 2. Construct correlation matrix by using correlation measure of each pair of voxels. 24

25 Cross Correlation Metric Cross-correlation of any two individual timeseries (i,j), at lag h, ρ ij (h), is defined as 25

26 Scalability of Functional Connections Connectivity matrices are expensive in voxel level, when no approximations are made Considering functional relations of a voxel with all other voxels; 8142 voxels makes 33M functional relations 26

27 Design and Use of Functional Connectivity Cluster voxels by their locations Measure correlation metric within clusters to generate connectivity matrices Use connectivity matrices to find functionallynearest neighbors 27

28 Functional connectivity map among the clusters k=256 sub-regions Functional connectivity for each cluster, for a given class..... s 35 voxels k=256 of each sub-regions 28

29 Design of New Neighborhood System Rather then selecting p-spatially closest points by L2-norm; select p-functionally closest points Select p-functionally closest points analyzing rows of within-cluster connectivity matrix Construct neighborhood-set with p-functionally closest voxels and calculate LRF accordingly 29

30 Given a voxel Select p-functionally Closest Voxel(s) Selecting p=4 functionally closest voxels v j For voxel v i in cluster c k where i=1 and k=26; Resulting neighbor indexes by considering highly correlated voxels in the cluster: j={3,6,9,14} 30

31 Classification Performances (%) with Spatial and Functional connectivity % LRF Order Values with Functional Connectivity (p) Without FC (raw LRF) k-nn 61, , , , , , , , ,

32 Performance of Mesh Learning with Functional Connectivities, p=10 32

33 Performance (%) Knn Classification for Each Class with Best FC Aware LRF 0,9 0,8 0,7 0,6250 0,7500 0,7917 0,6957 0,6250 0,6667 0,7083 0,7083 0,6 0,5 0,5417 0,5417 0,4 0,3 0,2 0, Class Label Her sınıf için elde edilen en yüksek performans değeri

34 Conclusion Mesh Learning model allows us to identify and differentiate classes of information represented in the brain during memory encoding and retrieval processes Functional connectivity represents the mesh better than the spatial connectivity 34

35 Implications We ultimately aim to read minds Better understand intention Better interpret feedback... Although we are not there yet, we are as close as we can get! 35

36 Thanks to Google Project Website: neuro.ceng.metu.edu.tr 36

37 Open Issues Estimating the true number of clusters Hierarchical neighbor selection Network measures will be incorporated Combination and use of between cluster metrics 37

38

39