Chapter 6. Machine Learning Classifiers for Protein- Protein Interaction Prediction

Transcription

1 Chapter 6 Machine Learning Classifiers for Protein- Protein Interaction Prediction

2 6.1 Introduction In the previous chapter, I reported that the PPIs predicted by GN, GC, ES, PP and GM methods show marginal overlap and hence they can complement each other. Therefore, integration of these methods is likely to provide better information than the single method alone. The integrative approach requires the tools and methods capable of transforming the heterogeneous scores generated by these prediction methods into meaningful combination. On the basis of rich mathematical/statistical foundation, Machine Learning Classifiers (MLCs) allow us to go beyond a mere linear score cutoff based predictions and provide hidden trends of the data in the form of testable models using gold standard data. Then, one can use these models to predict whether unknown protein pairs belong to positive or negative category. There are a number of studies showing feasibility of PPI prediction task using MLCs [54, 80, 88, 90, 91, 167]. It was suggested that performance of MLCs can be improved by appropriate integration and selection of features rather than by adding as many as features possible [80-82, 143, 168, 169]. However, previous reports lack critical comparative assessment of various MLCs and data dependency. Furthermore, the combination of MLCs for classification problems has been widely used in non-biological sciences but few studies have evaluated multiple classifiers for PPI predictions [86, 88, 91, 170]. A majority of these studies have been performed in Yeast as model organism [ ]. Some of the issues worth to be explored for E. coli encoded proteins is are: 1) which MLC is best suited for PPI predictions in E. coli, 2) the feasibility of multiple MLC based PPI predictions and, 3) choice of a appropriate Gold Standard (GS) dataset for learning. In this chapter, I have investigated aforementioned issues by predicting genome-wide PPI networks using seven different MLCs trained on four GS datasets. The resulting PPI networks provide highly accurate protein-protein functional interaction map generated to date, which can be used for system level analysis of Escherichia coli proteome.

3 6.2 Material and Methods Gold standard datasets All MLCs applied to the PPI prediction task in this work are supervised learning approaches. Therefore, these algorithms require GS dataset for training and testing. The analysis was carried out using positive GS datasets composed of Operon, DIP and Complex datasets. Because, PPI prediction methods were able to strongly discriminate these datasets from the negative examples (Chapter 4, section ). Considering the highest classification accuracy of Operon and Complex datasets, it was decided to combine them together to understand the discriminative power of MLCs on mixture of two different types of PPIs i.e. functional (co-operonic PPIs) and physical (co-complex PPIs). The negative examples were randomly chosen from the high confidence negative dataset generated for testing PPI prediction methods (Chapter 2, section 2.2.2). Each positive dataset was combined with a negative subset of size five times greater than the number of positive examples. This was done to predict physiologically meaningful PPIs whereas performance evaluation of MLCs was carried out using positive and negative dataset of equal size to get fair judgment of prediction accuracy Data feature encoding The interaction scores (i.e. features) for all possible pairs of E. coli proteins (including gold standard protein pairs) were calculated by GN, GC, ES, PP and GM methods. There are two possible ways for encoding these scores for machine learning. First, the scores for a particular protein pair generated by the five PPI prediction methods is represented by a single value called as Summary [88]. The same information source is described by the five values called as Detailed [88]. For the present analysis, Detailed method of feature encoding was used due to less number of available features.

4 6.2.3 Machine learning classifiers A set of seven MLCs was used in this study which includes Support Vector Machine (SVM), Decision tress (DT), Random Forest (RF), Naïve Bayes (NB), Bayesian Network (BN), Neural Network (NN) and Logistic Regression (LR). LibSVM package ( was used for SVM based predictions and WEKA toolbox ( for other machine learning algorithms. The cost and gamma functions of SVM were optimized using grid.py script provided in LibSVM package. J48 variant of decision tree was used because it incorporates numerical attributes, allows post pruning after induction of trees. The RF classifier is based on multiple DTs. A total of 100 DTs were grown simultaneously where each node uses a random subset of the features. In order to classify a new instance from an input vector, each input vector is subjected to analyzed by each of the trees in the forest. The output decision is based on majority vote over all the trees in the forest. For the Naïve Bayes classifier, a kernel estimation parameter was switched on. All other classifiers were applied with default parameters in the WEKA Cross-validation and performance assessment of MLCs The positive and negative GS datasets were randomly divided into five equal sized datasets. The protein pairs present in positive and negative dataset are referred as true positives (i.e. interacting) and true negatives (i.e. noninteracting). We performed five-fold cross-validation on all the MLCs using the above mentioned gold standard datasets. In each round of cross-validation, we used a random combination of four datasets for training the MLC and the remaining dataset was used for testing the model. This process was repeated for five times by using different combinations of training and testing datasets. For each round, we calculated the sensitivity (or TPR), specificity, and positive predictive value (or PPV or precision) as a measure of the quality of binary (twoclass) classifications using each MLC. The equations for sensitivity, PPV measures

5 have been described in the chapter 2, section Specificit y TN TNFP For all performance measures, TP is the number of positive dataset protein pairs that are correctly predicted as interacting by given MLC. FN is the number of positive dataset protein pairs that are incorrectly predicted as not interacting. Similarly TN is the number of negative dataset protein pairs that are correctly predicted as not interacting. FP is the number of negative dataset protein pairs that are incorrectly predicted as interacting. The average performance of five rounds of cross-validation was taken as a measure of prediction performance. The TPR and FPR were calculated using the decision values for protein pairs generated by SVM. For other MLCs, TPR and FPR values generated by WEKA were obtained. The TPR and FPR values were used to plot ROC curves Genome-wide PPI prediction The models generated by each of the seven MLCs using aforementioned four gold standard datasets were used for genome-wide PPI predictions in E. coli. The resulting 28 PPI networks were used to further analysis. The topological properties of the networks were calculated as described in the chapter 5, section Comparison of predicted PPI networks with experimental and functional PPI datasets PPINet GS ( PPINet, ) 100* PPINet GS JC GS

6 . The gold standard representing experimentally characterized and functional PPIs are same as described in Table Results and Discussion The 28 whole genome PPI maps of E. coli were reconstructed using models generated by seven MLCs on five data features that are interaction scores generated by GN, GC, ES, PP and GM. In the chapter 4, it was found that these prediction methods were able discriminate Operon, Complex and DIP PPIs from negatives with very high accuracy (Figure 4.1 & 4.2). Hence, these three datasets were used as gold standard (GS) positive datasets to generate MLC models along with a fourth GS called as Complex_Operon, which was a union of Operon and Complex PPIs. The performance of Operon and Complex dataset was observed relatively better than that of other datasets. It was assumed that their combination would provide enough training examples of functional (Operon) and physical (Complex) PPIs to generate better MLC model for predictions MLCs have predicted PPI networks with very high accuracy A combination of five data features was used to train and test seven MLCs using four GS datasets. Performance measures were averaged over five-fold crossvalidation for each MLC. As shown in the Figure 6.1, ROC curves for MLCs trained on four GS have reported extremely good sensitivities/tprs at the cost of less than 0.05 FPR (i.e. 5%). It suggests that only 5% predictions are likely to be false predictions. At the cost of 5% FPR, MLCs were able to discriminate 65, 80, 95 and 85 percent of PPIs (True positives) of DIP, Complex, Operon and Complex_Operon from negative training examples (Figure 6.1). One of the previous studies on PPI predictions in Yeast have reported that RF consistently ranked as one of the top two classifiers [88]. Figure 6.1 depicts that the performance of RF indeed better than other six classifiers using three out of four

7 GSs evaluated in this study. NB is the best performer when DIP GS was used for training MLCs. Figure 6.1 Performance accuracy measured as ROC curves for seven machine learning classifiers trained on four gold standard datasets. Each solid line on plots depicts the machine learning classifier (MLC). The colors of the lines correspond to the Bayesian Network (BN), Decision tress (DT), Logistic Regression (LR), Naïve Bayes (NB), Neural Network (NN), Random Forest (RF), and Support Vector Machine (SVM) classifier. MLCs were trained on gold standard dataset from A) DIP database B) cocomplex PPIs C) Co-operonic PPIs D) union of (B) and (C) PPIs. Performance of all MLCs using four gold standard dataset is elegant. RF classifier outperformed on (B), (C) and (D) gold standard datasets whereas NB on DIP. Note that y-axis starts at 0.4 TPR while x-axis ends at 0.05 FPR. generated by ES and GM in addition to above mentioned three features. However, Another independent study had been performed on SVM based PPI predictions in E. coli by Yellabiona and coworkers [54]. Authors have reported average fivefold cross-validation accuracy of SVM trained on GN, GC and PP features using Complex dataset of 0.89 (i.e. average of sensitivity and specificity). The present study have performed on five data features which include scores

8 the accuracy, which is obtained in the present study, is very similar to the reported by these authors. It was expected that the performance accuracy of classifiers would excel by two features additional features as compared to the three used by them but it didn t. One possible explanation for the similar results could be the composition of negatives and positives in their GS dataset. They have used 13 times higher numbers of negative examples as compared to the positives for predictions, which raised specificity of their cross-validation analysis to one as compared to the 0.97 achieved in this study. The sensitivity of 0.79 was achieved by them was preceded by 0.81 achieved in this study [54]. To a large extent ROCs are insensitive to absolute number of false positives since the calculations are solely based on ranks of positives relative to those of negatives [167]. So the ROCs alone can be misleading if absolute numbers of false positives become relevant. The possible number of all protein pairs among the proteins of any organism is too high as compared to the estimated PPIs. Out of 8 million possible pairs of E. coli proteins, the estimated number would be close to 40,000 probable interacting candidates only, thereby the absolute number of false positives equally matter. In order to ensure the quality of the final PPI predictions, PPV or precision and specificity of fivefold cross-validation was calculated and has shown in Figure 6.2. The analysis of Figure 6.2 A, reflects the outperformance of NB on all GS datasets and only two percent of negative examples were predicted as potential interacting by NB. The second best performance is observed for BN followed by SVM classifier. Specificities of NB trained on four GS lead to observations similar to PPV plot (Figure 6.2 B). Although, the performance of RF classifier measured as ROC is better than other classifiers, ability to detect negatives is substantially higher for the NB. Hence, NB predictions are biologically more significant than RF by considering the total space of possible protein pairs among proteins of an organism as compared to the estimated numbers of actual PPI.

9 Figure 6.2 Performance accuracy measured as specificity and positive predictive value for seven machine learning classifiers trained on four gold standard datasets. Each bar on plots depicts the average of five-fold crossvalidation accuracy of machine learning classifier trained on corresponding gold standard dataset. The machine learning classifiers are Bayesian Network (BN), Decision tress (DT), Logistic Regression (LR), Naïve Bayes (NB), Neural Network (NN), Random Forest (RF), and Support Vector Machine (SVM) classifier. A) Accuracy measured as the specificity B) Accuracy measured as the positive predictive value. Performance of NB classifier trained on four gold standard datasets is highest as compared to the other classifiers. Note that y-axis starts at Numbers of PPIs in the networks predicted by various MLCs varied greatly A total of 28 PPI networks were predicted using model generated by seven MLCs trained on four datasets consists of. The number of PPIs predicted by each classifier and overlap has shown in the Table 6.1. The numbers of PPIs predicted have shown substantial differences. A total number of interactions among the proteins present in an organism are difficult to determine by experimental method if not impossible. The range of 16,000 to 37,000 PPIs for Yeast proteome of ~6300 proteins was estimated by two independent computational analyses. To best of our knowledge such analysis has not been carried out in E. coli. Unlike Yeast, E. coli lacks sub-cellular compartments and the number of proteins is 4132 suggests that the number of interacting proteins may be in the same range. However, as shown in Table 6.1, 15 out of 28 predicted PPI networks by seven MLCs using four gold standard datasets contains number of interactions more than 100 thousands whereas numbers of interactions in remaining networks range from ~44,004 to 87,277.

10 Table 6.1 A statistics on the numbers of protein-protein interactions predicted by seven Machine Learning Classifiers (MLCs) trained on four different gold standard datasets DIP BN DT LR NB NN SVM RF BN DT LR NB NN SVM RF Complex BN DT LR NB NN SVM RF BN DT LR NB NN SVM RF Operon BN DT LR NB NN SVM RF BN DT LR NB NN SVM RF Complex_Operon BN DT LR NB NN SVM RF BN DT LR NB NN SVM RF Notes: The MLCs are Bayesian Network (BN), Decision tress (DT), Logistic Regression (LR), Naïve Bayes (NB). DIP, Complex, Operon and Complex_Operon are gold standard proteinprotein interaction datasets. The numbers with bold face represent a total number of PPI

11 predicted by corresponding classifier. NB predicts least numbers of PPIs whereas RF predicts higher numbers. various types of PPIs is also not reasonable since the coverage of known Majority of MLCs predicted more than 100 thousand interactions using Complex and DIP gold standard datasets. Though RF classifier has outperformed using three GS datasets but the predicted number of interactions using each model was more than 200 thousand. These numbers are far higher than the estimated numbers and hence might be consist of many false predictions. On the other hand, NB has predicted 78956, 57806, and PPIs using DIP, Complex, Operon and Complex_Operon GS datasets respectively. These numbers are quite close to the estimated PPIs in an organism and considering their small numbers likely to have many potential interactions Functional coverage of the predicted networks is very less The 28 PPI networks were reconstructed using classifiers that have given fivefold cross-validation PPV of more than 85%. It means that the 85% of the predicted PPIs using these models/classifiers are expected to be true interactions. The coverage of various known PPIs in these networks is one of the options to validate training accuracies of the MLCs. Plots in the Figure 6.3 show the coverage of 12 datasets representing known experimental (six sets) and functional (six sets) PPIs in the form of boxplots/wisker s plots. Boxplots represent differences between groups without making any assumptions of the underlying statistical distribution and hence they are non-parametric. The spacing between the different parts of the box indicates the degree of dispersion and skewness in the data. Here groups are various MLCs and each box on the plot represents distribution of Jaccard coefficients (JC) between the predicted PPIs by corresponding MLC and every known PPI datasets. JC represents the coefficient (scaled in between 0-100) of similarity between two datasets taking into account the size difference. Even though MLCs have shown very high sensitivity values, coverage of known PPIs in the predicted networks is very less (Figure 6.3). The training on

12 experimental (Figure 6.3 A) and functional (Figure 6.3 B) PPIs in the networks predicted using DIP, Complex and Operon GS is almost similar. Every known PPI dataset is underrepresented in the predicted networks, irrespective of the training gold standard datasets and MLC used for prediction. JC have not risen above 5 for a single classifier which was scaled in between zero and 100 (Figure 6.3). JC score of zero here reflects no overlap between two PPI networks while Figure 6.3 Coverage of 12 known experimental and functional proteinprotein interaction datasets in the networks, predicted by seven machine learning classifiers trained on four different gold standard datasets. Each boxplot on plots depicts the distribution of Jaccard similarity Coefficients calculated between network predicted by corresponding machine learning classifier and 6 datasets of known physical and functional protein-protein interactions (PPIs). The machine learning classifiers are Bayesian Network (BN), Decision tress (DT), Logistic Regression (LR), Naïve Bayes (NB), Neural Network (NN), Random Forest (RF), and Support Vector Machine (SVM). A) Coverage of experimentally characterized PPI datasets which include His-tagged bait, TAP-tagged baits (2 sets), co-presence of proteins in the same protein complex, DIP database and synthetic lethality analysis. B) Coverage of functional PPI datasets which include protein pairs that belong to the same KEGG pathway, GO term, COG functional category, Operon, EcoCyc functional category and transcriptional regulatory associations. Coverage of experimental as well as functional datasets is higher in the networks predicted by NB classifier using all gold standard training datasets. Every known PPI dataset is underrepresented in the network predicted by different MLCs.

13 score 100 means complete overlap. It is consistent with the previous observations that a computational PPI prediction complements the existing knowledge of PPIs [54, 56, 86, 173, 174]. It was also observed that coverage of known indirect interactions is better than the direct PPIs in the predicted networks [54]. Here, indirect interaction is defined as link between two proteins via its adjacent neighbors in the network rather than their direct interaction. Therefore, the marginal coverage of known PPIs in the networks predicted in present study is not an artifact but the limited available knowledge of biological systems. Unexpectedly, the coverage of experimental PPI datasets is relatively higher in the networks predicted by NB classifier (Figure 3A). These experimental PPI datasets include PPIs derived from His-tagged bait, TAP-tagged baits (2 sets), co-presence of proteins in the same protein complex, DIP database and synthetic lethality analysis. The coverage of these datasets in the NB networks has observed substantially higher than that of RF classifier. RF was the best performer in terms of ROC performance measure using three GS. It reflects the shortcoming of routinely used ROC performance measure for PPI predictions. As suggested by Park, Y, the ROC measure should be complimented with other performance measure to ensure the percentage of positives among the predicted PPIs [167]. The present study suggests the PPV and specificity plots can also be useful for such cross validation. NB has outperformed when performance was measured as PPV and specificity alone. The coverage of available functional PPI datasets also led to the similar observations (Figure 6.3 B). The coverage of functional PPI datasets is slightly higher than the experimentally derived ones but still all JC scores are below 5 (Figure 6.3 B). Furthermore, the distributions of JCs for various MLCs have differed comparably. Again, NB predicted PPI networks have higher coverage of functionally linked proteins as compared to the classifiers except BN (Figure 6.3 B). BN also have relatively better coverage of the functional PPI datasets under consideration. BN represents the probabilistic relationships between data

14 features/variables for the corresponding labels/evidence. Given data features, the Bayesian network can be used to compute the probabilities of the label associated with the data feature. NB is a special case of BN with strong independence (naïve) assumptions. Hence these results suggest that the classifiers based on the probabilistic models are better predictor of PPIs than the other classifiers used in this study such as RF, SVM etc. In the literature, such approaches have been indeed favored as compared to the other classifiers for PPI predictions [86, ] Networks predicted by NB classifier are better than the previous reports With the very limited knowledge of the total PPIs existing in the E. coli, it is challenging to compare these predictions with previously reported computational predictions. Furthermore, not only methodology differs with each report but also the GS used for evaluation. Hence, the direct comparison of various approaches that are used so far is often difficult [87]. In addition, performance of prediction often biased towards the GS, which is used for validation of predictions. Considering the combination of five PPI predictions (i.e. data features), multiple MLC models using four GS datasets, it was expected that the predicted PPIs would be of high quality with respect to false predictions and the coverage of proteins. In the light of afore mentioned concerns, it was decided to use three datasets that are not related to MLC training in the present study. At the same time, these datasets are likely to be free from experimental noise. The first two datasets consists of PPIs that are detected by the systematic large-scale tandemaffinity purifications to isolate multiprotein complexes by two independent studies [86, 109, 110]. The experimental set up, which was used by authors, minimizes the spurious non-specific protein associations and enables recovery of native protein complexes at near-endogenous levels. Third dataset was created using pairs of proteins that have been associated with each other in at least two

15 experimental analysis. Then, percentage of these datasets was calculated in the networks predicted in the present study and also for the four computational genome-scale PPI networks reported in the literature [54, 81, 82, 86]. Most of the networks predicted by various MLCs had limited coverage of these known datasets as compared to the networks reported in previous studies. Some of these networks had higher coverage, which may be likely due to the larger size of the predicted networks. Some of the networks such as, NB and BN predicted networks had much better coverage of these experimentally derived PPIs as compared to the existing networks. The comparisons of these networks have shown in the Figure 6.4. The size of these networks is comparatively less than that of existing genome-scale PPI datasets but the coverage of experimentally derived PPIs is substantially better than them (Figure 6.4). Figure 6.4 Comparison of the predicted networks with previously published PPI networks. X-axis represents computationally predicted genome-scale protein-protein interaction (PPI) networks. Networks with asterisk marks are predicted in the present study. Each bar represents percentage of a corresponding experimentally characterized PPIs in the predicted networks. Exp2 dataset represents PPIs that have been characterized by at least two different experimental analyses. The networks predicted in the present study have substantially high coverage of three PPI datasets used evaluation. Only network predicted by Yellabiona and coworkers is close to the predicted networks in this study. cnb, onb and conb stands for networks predicted by models generated by Naïve Bayes classifier using Complex, Operon and a union of complex & operon as a positive gold

16 standard dataset. Similaraly, cobn network is predicted by Bayesian Network using a union of complex & operon as a positive gold. 6.4 Summary The present study describes the prediction of genome-wide PPI networks using seven MLCs trained on four different GS datasets. It was observed that the combination of the confidence scores generated by five PPI prediction methods effectively increased classifiers accuracy. Majority of classifiers statistically show higher training accuracies but the predicted networks lack coverage of known PPIs. A probabilistic model based NB classifier, which assumes independence of data features recovers substantially higher numbers of known PPIs. These results led me the conclusion that probability based classifiers, which assume feature independence, are best suited for PPI predictions in E. coli. The reason behind their better coverage of known PPIs could be independent assumptions of the five methods for PPI predictions. Thereby, each feature have complemented others and enabled recovery of biologically meaningful interactions, when NB classifier was used. A combination of five PPI prediction methods, seven MLCs, and four GS datasets, led to the prediction of 28 networks with size no more than 3,20,000 PPIs. Majority of the networks consist of PPIs in the range between 1,00,000-2,00,000. These comparisons have encouraged me to speculate that a total number of PPIs in bacterial cells could be around 1,50,000 which is much higher than the previous estimates.