Practical statistical network analysis (with R and igraph)

Transcription

1 Practical statistical network analysis (with R and igraph) Gábor Csárdi csardi@rmki.kfki.hu Department of Biophysics, KFKI Research Institute for Nuclear and Particle Physics of the Hungarian Academy of Sciences, Budapest, Hungary Currently at Department of Medical Genetics, University of Lausanne, Lausanne, Switzerland

2 What is a network (or graph)? Practical statistical network analysis WU Wien 2

3 What is a network (or graph)? Practical statistical network analysis WU Wien 3

4 What is a network (or graph)? Practical statistical network analysis WU Wien 4

5 What is a graph? Binary relation (=edges) between elements of a set (=vertices).

6 What is a graph? Binary relation (=edges) between elements of a set (=vertices). E.g. vertices = {A, B, C, D, E} edges = ({A, B}, {A, C}, {B, C}, {C, E}).

7 What is a graph? Binary relation (=edges) between elements of a set (=vertices). E.g. vertices = {A, B, C, D, E} edges = ({A, B}, {A, C}, {B, C}, {C, E}). It is better to draw it: E D C A B

8 What is a graph? Binary relation (=edges) between elements of a set (=vertices). E.g. vertices = {A, B, C, D, E} edges = ({A, B}, {A, C}, {B, C}, {C, E}). It is better to draw it: E C A B D E C B A D Practical statistical network analysis WU Wien 5

9 Undirected and directed graphs If the pairs are unordered, then the graph is undirected: B vertices = {A, B, C, D, E} A C E edges = ({A, B}, {A, C}, {B, C}, {C, E}). D

10 Undirected and directed graphs If the pairs are unordered, then the graph is undirected: B vertices = {A, B, C, D, E} A C E edges = ({A, B}, {A, C}, {B, C}, {C, E}). D Otherwise it is directed: B vertices = {A, B, C, D, E} A C E edges = ((A, B), (A, C), (B, C), (C, E)). D Practical statistical network analysis WU Wien 6

11 The igraph package For classic graph theory and network science. Core functionality is implemented as a C library. High level interfaces from R and Python. GNU GPL. Practical statistical network analysis WU Wien 7

12 Vertex and edge ids Vertices are always numbered from zero (!). Numbering is continual, form to V.

13 Vertex and edge ids Vertices are always numbered from zero (!). Numbering is continual, form to V. We have to translate vertex names to ids: V = {A, B, C, D, E} E = ((A, B), (A, C), (B, C), (C, E)). A =, B =, C = 2, D = 3, E = 4.

14 Vertex and edge ids Vertices are always numbered from zero (!). Numbering is continual, form to V. We have to translate vertex names to ids: V = {A, B, C, D, E} E = ((A, B), (A, C), (B, C), (C, E)). A =, B =, C = 2, D = 3, E = 4. > g <- graph( c(,,,2,,2, 2,4), n=5 ) Practical statistical network analysis WU Wien 8

15 Creating igraph graphs igraph objects

16 Creating igraph graphs igraph objects print(), summary(), is.igraph()

17 Creating igraph graphs igraph objects print(), summary(), is.igraph() is.directed(), vcount(), ecount() > g <- graph( c(,,,2,,2, 2,4), n=5 ) 2 > g 3 Vertices: 5 4 Edges: 4 5 Directed: TRUE 6 Edges: 7 8 [] -> 9 [] -> 2 [2] -> 2 [3] 2 -> 4 Practical statistical network analysis WU Wien 9

18 Visualization > g <- graph.tree(4, 4) 2 > plot(g) 3 > plot(g, layout=layout.circle)

19 Visualization > g <- graph.tree(4, 4) 2 > plot(g) 3 > plot(g, layout=layout.circle) # Force directed layouts 2 > plot(g, layout=layout.fruchterman.reingold) 3 > plot(g, layout=layout.graphopt) 4 > plot(g, layout=layout.kamada.kawai)

20 Visualization > g <- graph.tree(4, 4) 2 > plot(g) 3 > plot(g, layout=layout.circle) # Force directed layouts 2 > plot(g, layout=layout.fruchterman.reingold) 3 > plot(g, layout=layout.graphopt) 4 > plot(g, layout=layout.kamada.kawai) # Interactive 2 > tkplot(g, layout=layout.kamada.kawai) 3 > l <- layout=layout.kamada.kawai(g)

21 Visualization > g <- graph.tree(4, 4) 2 > plot(g) 3 > plot(g, layout=layout.circle) # Force directed layouts 2 > plot(g, layout=layout.fruchterman.reingold) 3 > plot(g, layout=layout.graphopt) 4 > plot(g, layout=layout.kamada.kawai) # Interactive 2 > tkplot(g, layout=layout.kamada.kawai) 3 > l <- layout=layout.kamada.kawai(g) # 3D 2 > rglplot(g, layout=l)

22 Visualization > g <- graph.tree(4, 4) 2 > plot(g) 3 > plot(g, layout=layout.circle) # Force directed layouts 2 > plot(g, layout=layout.fruchterman.reingold) 3 > plot(g, layout=layout.graphopt) 4 > plot(g, layout=layout.kamada.kawai) # Interactive 2 > tkplot(g, layout=layout.kamada.kawai) 3 > l <- layout=layout.kamada.kawai(g) # 3D 2 > rglplot(g, layout=l) # Visual properties 2 > plot(g, layout=l, vertex.color="cyan") Practical statistical network analysis WU Wien

23 Simple graphs igraph can handle multi-graphs: V = {A, B, C, D, E} E = ((AB), (AB), (AC), (BC), (CE)). > g <- graph( c(,,,,,2,,2, 3,4), n=5 ) 2 > g 3 Vertices: 5 4 Edges: 5 5 Directed: TRUE 6 Edges: 7 8 [] -> 9 [] -> [2] -> 2 [3] -> 2 2 [4] 3 -> 4 Practical statistical network analysis WU Wien

24 Simple graphs igraph can handle loop-edges: V = {A, B, C, D, E} E = ((AA), (AB), (AC), (BC), (CE)). > g <- graph( c(,,,,,2,,2, 3,4), n=5 ) 2 > g 3 Vertices: 5 4 Edges: 5 5 Directed: TRUE 6 Edges: 7 8 [] -> 9 [] -> [2] -> 2 [3] -> 2 2 [4] 3 -> 4 Practical statistical network analysis WU Wien 2

25 Creating (more) igraph graphs el <- scan("lesmis.txt") 2 el <- matrix(el, byrow=true, nc=2) 3 gmis <- graph.edgelist(el, dir=false) 4 summary(gmis) Practical statistical network analysis WU Wien 3

26 Naming vertices V(gmis)$name 2 g <- graph.ring() 3 V(g)$name <- sample(letters, vcount(g)) Practical statistical network analysis WU Wien 4

27 Creating (more) igraph graphs # A simple undirected graph 2 > g <- graph.formula( Alice-Bob-Cecil-Alice, 3 Daniel-Cecil-Eugene, Cecil-Gordon )

28 Creating (more) igraph graphs # A simple undirected graph 2 > g <- graph.formula( Alice-Bob-Cecil-Alice, 3 Daniel-Cecil-Eugene, Cecil-Gordon ) # Another undirected graph, ":" notation 2 > g2 <- graph.formula( Alice-Bob:Cecil:Daniel, 3 Cecil:Daniel-Eugene:Gordon )

29 Creating (more) igraph graphs # A simple undirected graph 2 > g <- graph.formula( Alice-Bob-Cecil-Alice, 3 Daniel-Cecil-Eugene, Cecil-Gordon ) # Another undirected graph, ":" notation 2 > g2 <- graph.formula( Alice-Bob:Cecil:Daniel, 3 Cecil:Daniel-Eugene:Gordon ) # A directed graph 2 > g3 <- graph.formula( Alice +-+ Bob --+ Cecil Daniel, Eugene --+ Gordon:Helen )

30 Creating (more) igraph graphs # A simple undirected graph 2 > g <- graph.formula( Alice-Bob-Cecil-Alice, 3 Daniel-Cecil-Eugene, Cecil-Gordon ) # Another undirected graph, ":" notation 2 > g2 <- graph.formula( Alice-Bob:Cecil:Daniel, 3 Cecil:Daniel-Eugene:Gordon ) # A directed graph 2 > g3 <- graph.formula( Alice +-+ Bob --+ Cecil Daniel, Eugene --+ Gordon:Helen ) # A graph with isolate vertices 2 > g4 <- graph.formula( Alice -- Bob -- Daniel, 3 Cecil:Gordon, Helen )

31 Creating (more) igraph graphs # A simple undirected graph 2 > g <- graph.formula( Alice-Bob-Cecil-Alice, 3 Daniel-Cecil-Eugene, Cecil-Gordon ) # Another undirected graph, ":" notation 2 > g2 <- graph.formula( Alice-Bob:Cecil:Daniel, 3 Cecil:Daniel-Eugene:Gordon ) # A directed graph 2 > g3 <- graph.formula( Alice +-+ Bob --+ Cecil Daniel, Eugene --+ Gordon:Helen ) # A graph with isolate vertices 2 > g4 <- graph.formula( Alice -- Bob -- Daniel, 3 Cecil:Gordon, Helen ) # "Arrows" can be arbitrarily long 2 > g5 <- graph.formula( Alice Bob ) Practical statistical network analysis WU Wien 5

32 Vertex/Edge sets, attributes Assigning attributes: set/get.graph/vertex/edge.attribute.

33 Vertex/Edge sets, attributes Assigning attributes: set/get.graph/vertex/edge.attribute. V(g) and E(g).

34 Vertex/Edge sets, attributes Assigning attributes: set/get.graph/vertex/edge.attribute. V(g) and E(g). Smart indexing, e.g. V(g)[color=="white"]

35 Vertex/Edge sets, attributes Assigning attributes: set/get.graph/vertex/edge.attribute. V(g) and E(g). Smart indexing, e.g. V(g)[color=="white"] Easy access of attributes: > g <- erdos.renyi.game(, /) 2 > V(g)$color <- sample( c("red", "black"), 3 vcount(g), rep=true) 4 > E(g)$color <- "grey" 5 > red <- V(g)[ color == "red" ] 6 > bl <- V(g)[ color == "black" ] 7 > E(g)[ red %--% red ]$color <- "red" 8 > E(g)[ bl %--% bl ]$color <- "black" 9 > plot(g, vertex.size=5, layout= layout.fruchterman.reingold) Practical statistical network analysis WU Wien 6

36 Creating (even) more graphs E.g. from.csv files. > traits <- read.csv("traits.csv", head=f) 2 > relations <- read.csv("relations.csv", head=f) 3 > orgnet <- graph.data.frame(relations) 4 5 > traits[,] <- sapply(strsplit(as.character 6 (traits[,]), split=" "), "[[", ) 7 > idx <- match(v(orgnet)$name, traits[,]) 8 > V(orgnet)$gender <- as.character(traits[,3][idx]) 9 > V(orgnet)$age <- traits[,2][idx] > igraph.par("print.vertex.attributes", TRUE) 2 > orgnet Practical statistical network analysis WU Wien 7

37 Creating (even) more graphs From the web, e.g. Pajek files. > karate <- read.graph(" 2 format="pajek") 3 > summary(karate) 4 Vertices: 34 5 Edges: 78 6 Directed: FALSE 7 No graph attributes. 8 No vertex attributes. 9 No edge attributes. Practical statistical network analysis WU Wien 8

38 Graph representation There is no best format, everything depends on what kind of questions one wants to ask. Iannis Esmeralda Fabien Alice Diana Helen Cecil Jennifer Gigi Bob Practical statistical network analysis WU Wien 9

39 Graph representation Adjacency matrix. Good for questions like: is Alice connected to Bob? Alice Bob Cecil Diana Esmeralda Fabien Gigi Helen Iannis Jennifer Alice Bob Cecil Diana Esmeralda Fabien Gigi Helen Iannis Jennifer Practical statistical network analysis WU Wien 2

40 Graph representation Edge list. Not really good for anything. Alice Bob Cecil Alice Diana Cecil Esmeralda Bob Cecil Diana Alice Bob Diana Esmeralda Fabien Gigi Diana Esmeralda Alice Helen Bob Diana Diana Esmeralda Esmeralda Fabien Fabien Gigi Gigi Gigi Helen Helen Helen Helen Helen Helen Iannis Iannis Jennifer Jennifer Practical statistical network analysis WU Wien 2

41 Graph representation Adjacency lists. GQ: who are the neighbors of Alice? Alice Bob Cecil Diana Esmeralda Fabien Gigi Helen Iannis Jennifer Bob, Esmeralda, Helen, Jennifer Alice, Diana, Gigi, Helen Diana, Fabien, Gigi Bob, Cecil, Esmeralda, Gigi, Helen, Iannis Alice, Diana, Fabien, Helen, Iannis Cecil, Esmeralda, Helen Bob, Cecil, Diana, Helen Alice, Bob, Diana, Esmeralda, Fabien, Gigi, Jennifer Diana, Esmeralda Alice, Helen Practical statistical network analysis WU Wien 22

42 Graph representation igraph. Flat data structures, indexed edge lists. Easy to handle, good for many kind of questions. Practical statistical network analysis WU Wien 23

43 Centrality in networks degree Iannis 2 Esmeralda 5 Cecil 3 Fabien 3 Diana 6 Gigi 4 Helen 7 Bob 4 Alice 4 Jennifer 2 Practical statistical network analysis WU Wien 24

44 Centrality in networks closeness C v = V i v d vi Iannis.53 Esmeralda.69 Cecil.53 Fabien.6 Diana.75 Gigi.64 Helen.82 Bob.64 Alice.6 Jennifer.5 Practical statistical network analysis WU Wien 25

45 Centrality in networks betweenness B v = g ivj /g ij i j,i v,j v Iannis Esmeralda 4.62 Cecil.83 Fabien.45 Diana 6.76 Gigi.45 Helen. Bob.2 Alice.67 Jennifer Practical statistical network analysis WU Wien 26

46 Centrality in networks eigenvector centrality E v = λ V i= A iv E i, Ax = λx Iannis.36 Esmeralda.75 Cecil.46 Fabien.49 Diana.88 Gigi.68 Helen Bob.7 Alice.63 Jennifer.36 Practical statistical network analysis WU Wien 27

47 Centrality in networks page rank E v = d V + d V i= A iv E i Iannis.34 Esmeralda.74 Cecil.47 Fabien.47 Diana.87 Gigi.59 Helen Bob.59 Alice.6 Jennifer.34 Practical statistical network analysis WU Wien 28

48 Community structure in networks Organizing things, clustering items to see the structure. M. E. J. Newman, PNAS, 3, Practical statistical network analysis WU Wien 29

49 Community structure in networks How to define what is modular? Many proposed definitions, here is a popular one: Q = 2 E [A vw p vw ]δ(c v, c w ). vw

50 Community structure in networks How to define what is modular? Many proposed definitions, here is a popular one: Q = 2 E [A vw p vw ]δ(c v, c w ). vw Random graph null model: p vw = p = V ( V )

51 Community structure in networks How to define what is modular? Many proposed definitions, here is a popular one: Q = 2 E [A vw p vw ]δ(c v, c w ). vw Random graph null model: p vw = p = V ( V ) Degree sequence based null model: p vw = k vk w 2 E Practical statistical network analysis WU Wien 3

52 Cohesive blocks (Based on Structural Cohesion and Embeddedness: a Hierarchical Concept of Social Groups by J.Moody and D.White, Americal Sociological Review, 68, 3 27, 23) Definition : A collectivity is structurally cohesive to the extent that the social relations of its members hold it together.

53 Cohesive blocks (Based on Structural Cohesion and Embeddedness: a Hierarchical Concept of Social Groups by J.Moody and D.White, Americal Sociological Review, 68, 3 27, 23) Definition : A collectivity is structurally cohesive to the extent that the social relations of its members hold it together. Definition 2: A group is structurally cohesive to the extent that multiple independent relational paths among all pairs of members hold it together.

54 Cohesive blocks (Based on Structural Cohesion and Embeddedness: a Hierarchical Concept of Social Groups by J.Moody and D.White, Americal Sociological Review, 68, 3 27, 23) Definition : A collectivity is structurally cohesive to the extent that the social relations of its members hold it together. Definition 2: A group is structurally cohesive to the extent that multiple independent relational paths among all pairs of members hold it together. Vertex-independent paths and vertex connectivity.

55 Cohesive blocks (Based on Structural Cohesion and Embeddedness: a Hierarchical Concept of Social Groups by J.Moody and D.White, Americal Sociological Review, 68, 3 27, 23) Definition : A collectivity is structurally cohesive to the extent that the social relations of its members hold it together. Definition 2: A group is structurally cohesive to the extent that multiple independent relational paths among all pairs of members hold it together. Vertex-independent paths and vertex connectivity. Vertex connectivity and network flows. Practical statistical network analysis WU Wien 3

56 Cohesive blocks Practical statistical network analysis WU Wien 32

57 Cohesive blocks Practical statistical network analysis WU Wien 33

58 Rapid prototyping Weighted transitivity c(i) = A3 ii (AA) ii

59 Rapid prototyping Weighted transitivity c(i) = A3 ii (AA) ii c w (i) = W 3 ii (WW max W) ii

60 Rapid prototyping Weighted transitivity c(i) = A3 ii (AA) ii c w (i) = W 3 ii (WW max W) ii wtrans <- function(g) { 2 W <- get.adjacency(g, attr="weight") 3 WM <- matrix(max(w), nrow(w), ncol(w)) 4 diag(wm) <- 5 diag( W %*% W %*% W ) / 6 diag( W %*% WM %*% W) 7 } Practical statistical network analysis WU Wien 34

61 Rapid prototyping Clique percolation (Palla et al., Nature 435, 84, 25) Practical statistical network analysis WU Wien 35

62 ... and the rest Cliques and independent vertex sets. Network flows. Motifs, i.e. dyad and triad census. Random graph generators. Graph isomorphism. Vertex similarity measures, topological sorting, spanning trees, graph components, K-cores, transitivity or clustering coefficient. etc. C-level: rich data type library. Practical statistical network analysis WU Wien 36

63 Acknowledgement Tamás Nepusz All the people who contributed code, sent bug reports, suggestions The R project Hungarian Academy of Sciences The OSS community in general Practical statistical network analysis WU Wien 37