IEMS 441 Social Network Analysis Term Paper Multiple Testing Multi-theoretical, Multi-level Hypotheses

Transcription

1 IEMS 441 Social Network Analysis Term Paper Multiple Testing Multi-theoretical, Multi-level Hypotheses Jiangtao Gou Department of Statistics, Northwestern University Instructor: Prof. Noshir Contractor March 13, Introduction Exponential random graph model has been carefully reviewed in [11] and [13], and the corresponding theory of testing multi-theoretical multilevel hypotheses has been carefully discussed in [2] and [9], and a practical R package statnet has been released [4][5]. In this term paper, I consider a set of statistical inferences simultaneously. Exponential random graph (p*) statistical models (ERGM) nested variables at various levels which are simultaneously estimated. But if I want to test several hypotheses together under a preset significant level, then multiple testing problems occur. Under the significant level (type-i error) 5%, I need to apply multiple testing procedures to guarantee that the family-wise error rate (FWER). If I test the two hypotheses separately, the FWER may not be controlled by 5% which I set previously. In most of research papers, authors did not consider all hypotheses simultaneously. Consequently, if they drew conclusions simultaneously, the type-i error might not be controlled, in other words, they would face larger risk to the false positives than they originally expected. Some researchers applied multiple testing procedures, for example, Baker and Faulkner (1991) applied Bonferroni multiple testing procedure when they studied the network in the Hollywood film industry [1], although they did not specifically mention that they actually had used Bon- 1

2 ferroni multiple testing procedure. While, classic Bonferroni multiple testing procedure is very conservative [14]. It is safe, but sometimes lack of power, say, lack of the ability to discover the true positive. In 1980 s, a lot of multiple testing procedures were constructed. Here, I introduce Hochberg procedure and Hommel procedure into social network area. Hochberg procedure is slightly less powerful than Hommel procedure, but easier to apply. In Hochberg procedure [7], at first, all p-values are ordered p (1) p (2) p (n), and the corresponding hypotheses are H (1), H (2),, H (n), then in step 1, compare p (n) with significant level α, if p (n) α, reject all hypotheses then stop, otherwise go to the next step. In step 2, compare p (n 1) with significant level α/2, if p (n 1) α/2, reject all hypotheses except H (n) then stop, otherwise go to the next step. Similarly, in step j, compare p (n i+1) with significant level α/i, if p (n i+1) α/i, reject all hypotheses from H (1) to H (n i+1) then stop, otherwise go to the next step until comparing with p (1) with significant level α/n. In Hommel procedure [8], p-values and hypotheses are ordered, then in step 1, compare p (n) with significant level α, if p (n) α, reject all hypotheses then stop, otherwise go to the next step. In step j, if p (n i+1) > α(j i + 1)/j for i = 1,, j, then accept H (n j+1), go to the next stop, otherwise reject all H (i) where p (i) α/(j 1) [3]. 2 Examples In this section, I give three examples to demonstrate how to apply multiple hypothesis procedures. 2.1 Florentine Family Marriage and Business Ties Data This is a data set of marriage and business ties among Renaissance Florentine families [10]. The two relations are business ties and marriage alliances. Vertex information includes wealth, the number of seats on the civic council, and the total number ties [5]. There are 16 vertices, with 20 links in marriage network and 15 links in business network. Both net- 2

3 works are symmetric. Figure 1: Florentine family marriage (left) and business (right) network plot with vertex size proportional to wealth [4] The model is flomarriage edges + nodecov("wealth") + nodecov("priorates") + nodecov("totalties") + edgecov(flobusiness) Table 1: MLE estimates with p-values Estimate SE p-value edges nodecov.wealth nodecov.priorates nodecov.totalties edgecov.flobusiness There are five p-values, I order them from the largest to the smallest When setting the significant level α 5%, if I apply the least significant (LS) procedure, then I directly compare all p-values with Since p (3), 3

4 Table 2: Ordered p-values p (5) nodecov.totalties p (4) nodecov.wealth p (3) nodecov.priorates p (2) edgecov.flobusiness p (1) edges p (2) and p (1) are less than 5%, I may conclude that priorates, flobusiness and edges are significant. If I apply the Hommel procedure, it is a step-up procedure, I have (1) p (5) = > 0.05 = α, go to the next step, (2) p (4) = > = α/2, go to the next step, (3) p (3) = < = 2α/3, stop, reject all p (i) s which are less than α/2 = 0.025, say, p (3), p (2) and p (1). I may conclude that priorates, flobusiness and edges are significant. If I apply the Hochberg procedure, it is a step-up procedure, I have (1) p (5) = > 0.05 = α, go to the next step, (2) p (4) = > = α/2, go to the next step, (3) p (3) = > = α/3, go to the next step, (4) p (2) = < = α/4, stop, reject p (2) and p (1). I may conclude that flobusiness and edges are significant. If I apply Bonferroni procedure, which is the most conservative procedure, then I directly compare all p-values with α/5 = Since p (2) and p (1) are less than 1%, I may conclude that flobusiness and edges are significant. Table 3: Testing Results: NS (not significant) S (significant) LS Hommel Hochberg Bonferroni p (5) nodecov.totalties NS NS NS NS p (4) nodecov.wealth NS NS NS NS p (3) nodecov.priorates S S NS NS p (2) edgecov.flobusiness S S S S p (1) edges S S S S 4

5 Bonferroni procedure is more conservative than Hochberg procedure, Hochberg procedure is more conservative than Hommel procedure, Hommel procedure is more conservative than LS procedure. Boferroni, Hochberg and Hommel procedures can control the familywise error rate (FWER) under given significant level α, but LS procedure can not. 2.2 Longitudinal networks of positive affection within a monastery In this section I consider a directed network as a case in point. In this data set, Sampson recorded the social interactions among a group of monks while resident as an experimenter on vision, and collected numerous sociometric rankings [5][12]. The whole data set includes three phases, I only use the data in phase 3, called samplk3 in R. There are 18 vertices with 56 (directed) edges. The model is flomarriage samplk3 edges + mutual + gwesp(0.2, fixed = T) The estimations are shown in Table 4. Table 4: MLE estimates with p-values Estimate SE p-value edges mutual gwesp.fixed When applying significant level 5%, the testing results are 2.3 Goodreau s Faux Mesa High School This data set shows a simulation of an in-school friendship network, which is based in the rural western US, with a student body that is largely Hispanic and Native American [5]. This is a 205-vertex network with 203 connections. There are 99 female students and 106 male students. 5

6 Figure 2: Longitudinal networks of positive affection within a monastery, phase three Table 5: Testing Results: NS (not significant) S (significant) LS Hommel Hochberg Bonferroni p (3) gwesp.fixed.0.2 S S S NS p (2) mutual S S S S p (1) edges S S S S 6

7 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 Figure 3: Goodreau s Faux Mesa High School network plot 7

8 The model is mesa edges + nodematch("grade", diff = T) + nodematch("race", diff = T) The estimation results are shown in Table 6 Table 6: MLE estimates with p-values Estimate SE p-value edges < nodematch.grade < nodematch.grade < nodematch.grade < nodematch.grade < nodematch.grade < nodematch.grade < nodematch.race.black -Inf NA NA nodematch.race.hisp nodematch.race.natam < nodematch.race.other -Inf NA NA nodematch.race.white Some coefficients can not be estimated, because there are too few students in these categories. When applying significant level 5%, the testing results are If I apply the least significant (LS) procedure, then I directly compare all p-values with I may conclude that all except nodematch.race.hisp are significant. If I apply the Hommel procedure, it is a step-up procedure, I have (1) p (10) = > 0.05 = α, go to the next step, (2) p (9) = < = α/2, stop, reject all p (i) s which are less than α = I may conclude that all except nodematch.race.hisp are significant. If I apply the Hochberg procedure, it is a step-up procedure, I have (1) p (10) = > 0.05 = α, go to the next step, (2) p (9) = < = 8

9 Table 7: Testing Results: NS (not significant) S (significant) LS Hommel Hochberg Bonferroni edges S S S S nodematch.grade.7 S S S S nodematch.grade.8 S S S S nodematch.grade.9 S S S S nodematch.grade.10 S S S S nodematch.grade.11 S S S S nodematch.grade.12 S S S S nodematch.race.black nodematch.race.hisp NS NS NS NS nodematch.race.natam S S S S nodematch.race.other nodematch.race.white S S S NS α/2, stop, reject all p-values between p (9) and p (1). I may conclude that all except nodematch.race.hisp and nodematch.race.white are significant. If I apply Bonferroni procedure, which is the most conservative procedure, then I directly compare all p-values with α/5 = Since p (2) and p (1) are less than 1%, I may conclude that all except nodematch.race.hisp are significant. 3 Future Work In this paper, I only consider the uncorrelated hypotheses multiple testing at first (though this assumption about the uncorrelated hypotheses may not be true in the case of network analysis). The next step is to consider the correlated hypotheses multiple testing. I think how to estimate the correlation between different hypotheses could be a problem, but I can assume a relatively safe correlation (if the null hypotheses are favored, we can assume a relatively small correlation) to form the multiple testing procedure. After some correlation corrections, multiple testing 9

10 procedures, like Hommel procedure, Hochberg procedure, can be applied into multiple testing in network hypotheses. References [1] Wayne E. Baker, Robert R. Faulkner, Role as Resource in the Hollywood Film Industy. The American Journal of Sociology, 97, 2, p [2] N. S. Contractor, S. Wasserman, K. Faust, Testing Multitheoretical Multilevel Hypotheses About Organizational Networks: An Analytic Framework and Emprical Example. Academy of Management Review, 31, 3, p [3] Alex Dmitrienko, Ajit C. Tamhane, Frank Bretz, Multiple Testing Problems in Pharmaceutical Statistics. Chapman and Hall/CRC Press. [4] Steven M. Goodreau, joint with the rest of the Statnet Development Team, Introduction to Exponetial-family Random Graph (ERG or p*) modeling with statnet. INSNA Sunbelt - St. Pete Beach, Florida. [5] Mark S. Handcock, David R. Hunter, Carter T. Butts, Steven M. Goodreau, and Martina Morris, Software Tools for the Statistical Modeling of Network Data. Version 2.6. Project home page at http: //statnet.org, URL [6] P. D. Hoff, A. E. Raftery, M. S. Handcock, Latent space approaches to social network analysis, Journal of the American Statistical Association 97, 1090C1098. [7] Yosef Hochberg, A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75, 4, p [8] G. Hommel, A Stagewise Rejective Multiple Test Procedure Based on a Modified Bonferroni Test. Biometrika, 75, 2, p [9] P. R. Monge, N. S. Contractor, Theories of communication networks. New York: Oxford University Press. 10

11 [10] John F. Padgett, Marriage and Elite Structure in Renaissance Florence, Paper delivered to the Social Science History Association. [11] G. Robins, P. Pattison, Y. Kalish, D. Lusher, Introduction to exponential random graph (p*) models for social networks. Social Networks, 29, 2, p [12] S. F. Sampson, A novitiate in a period of change: An experimental and case study of relationships, Unpublished Ph.D. dissertation, Department of Sociology, Cornell University. [13] M. Shumate, E. T. Palazzolo, Exponential Random Graph (p*) Models as a Method for Social Network Analysis in Communication Research. Communication Methods and Measures, 4, 4, p [14] A. C. Tamhane, Statistical Analysis of Designed Experiments. John Wiley and Sons, Inc. 11