Measurement Quality in Social Networks

Measurement Quality in Social Networks LLuís Coromina Quantitative Methods, University of Girona Department of Economics, Campus Montilivi, 17071 Girona, Spain E-mail: lluis.coromina@udg.edu 1

Outline Studies on Social Network Data Quality Survey Design o Data collection o Questionnaire Design Types of hypothesis tested in other Quality studies Quality Measurement (Validity and Reliability) Measurement procedure o Multitrait-Multimethod Model (MTMM) o Multilevel MTMM o Interpretation 2

Outline INSOC PROJECT: o Social Network Data collection Design o Multitrait-Multimethod Model results o Meta-analysis Design o Meta-analysis Results Results and comparison from others studies. 3

Objective Main purpose of scientific research is the interpretation and/or prediction of phenomena Quality measurement instruments (empirical data) is of crucial importance It is important to know how well we can measure with certain measurement instruments and how much a researcher can rely on findings based on data collected. 4 4

Objective Egocentered network data are especially important. Network characteristics (network size, structure and composition) Characteristics of network members (feeling of closeness or importance; frequency of advice, collaboration, social support) collected from the respondent (ego). 5 5

Quality in Social Network Survey Data Studies on Social Network Data Quality Killworth & Bernard, 1976, 1980; Bernard et al., 1979,1980, 1982; Hammer, 1984; Sudman, 1985, 1988; Freeman and Romney, 1987; Freeman, Romney & Freeman, 1987; Corman and Bradford, 1993; Hlebec, 1993; Brewer and Webster, 1997, 6 6

Quality in Social Network Survey Data People are generally very inaccurate in reporting on their past interactions with other people. People remember long-term or typical patterns of interaction with other people rather well. Accuracy of reporting about interactions is also influenced by the frequency of interaction (more frequent contact with group members, more accurate reports) and by the reliability of an individual respondent on about actual interactions (Romney and Faust, 1982; Romney and Weller, 1984). 7 7

Quality in Social Network Survey Data Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj & Hlebec, 1999). + Emotional and informational support dimensions Reliability - Social companionship and material support dimensions Reasons for higher quality More intimate types of support more important to the respondents. Relatively small number of very important people. Less effortful for the respondent. Network members are important and close 8 8

Data Collection Methods DATA COLLECTION METHODS In Social Sciences, the most frequent measurement instrument is a SURVEY 9 9

Data Collection Methods 10 10

Data Collection Methods Paper With interviewer Personal (Face-to-face) telephone Electronic computer assisted personal interview (CAPI) computer assisted telephone interview (CATI) Self-administration paper selfadministered mail E-mail Web surveys Computer Assisted Self- Interview (CASI) ACASI Mobile phone surveys NO ONE COLLECTION METHOD IS BEST FOR ALL CIRCUMSTANCES. THE CHOICE MUST BE MADE WITHIN THE CONTEXT OF THE PARTICULAR OBJECTIVES OF THE SURVEY AND THE RESOURCES AVAILABLE. 11 11

Data Collection Methods Characteristics of the Population + Characteristics of The Sample + Types of Questions + Question Topic + Response Rate + Cost $$ + Time 12 12

Data Collection Methods PERSONAL INTERVIEWING ADVANTAGES: DISADVANTAGES: Generally: highest cooperation and lowest Most costly mode of administration refusal rates Allows for longer, more complex interviews Longer data collection period High response quality Interviewer concerns Takes advantage of interviewer presence Multi-method data collection TELEPHONE INTERVIEWING ADVANTAGES: DISADVANTAGES: Less expensive than personal interviews Samples of general population Non-response Shorter data collection period than personal Questionnaire constraints Interviewer administration (vs. mail) Difficult to administer questionnaires on sensitive or complex topics Better control and supervision of interviewers (vs. personal) Better response rate than mail for list samples Biased against households without telephones, unlisted numbers 13 13

Data Collection Methods MAIL SURVEYS ADVANTAGES: DISADVANTAGES: Generally lowest cost Most difficult to obtain cooperation Can be administered by smaller team of No interviewer involved in collection of data people (no field staff) Access to otherwise difficult to locate, busy More likely to need an incentive for populations respondents Respondents can look up information or Need good sample consult with others Slower data collection period than telephone 14 14

Data Collection Methods WEB SURVEYS ADVANTAGES: Lower cost (no paper, postage, mailing, data entry costs) DISADVANTAGES % of homes own a computer; % have home e-mail. Can reach international populations Representative samples difficult - cannot generate random samples of general population Time required for implementation reduced Differences in capabilities of people's computers and software for accessing Web surveys Complex skip patterns can be programmed Different ISPs/line speeds limits extent of graphics that can be used Sample size can be greater 15 15

Data Collection Methods MEASUREMENT QUALITY IMPLICATIONS OF MODE SELECTION Social Desirability Under-represent (illegalities) or Over-represent (voting) different population groups. Sensitive information from respondents Self-administered methods produce fewer social desirability effects. Response effects (wording, answer categories order, or questions order) Response order (Primacy-visually- and recency telephone) Data completeness (with interviewers more questions answered) Completion times 16 16

Data Collection Methods Mean completion times(in seconds) by mode (web vs Telephone) and age group 17 17

Data Collection Methods Differences in data quality for different methods???? Data collection method for ego-centered network data. Most used face-to-face data collection mode. Face to face vs. Telephone (Kogovšek et al., 2002) Validity and Reliability of web surveys (Couper, 2000; Dillman, 2000; Couper et al., 2001; Vehovar et al., 2002) Quality of Egocentered Network Data Collected via Web Marin (2002) ; Lozar Manfreda et al. (2004). 18 18

Questionnaire Design QUESTIONNAIRE DESIGN Problems in the response process Survey ERRORS Krosnick (1999); Groves (2004), Tourangeau, Rips & Rasinski (2000), Dillman, (2007) 19 19

Questionnaire Design 1) Failure to encode the information required Problem of asking people about what topics that they do not pay much attention OR are not important. Improvement: Pre-test; Filter questions (or multiple name generator) 2) Misinterpretation of the questions Hospital study: During the past 12 months, since, how many times have you seen or talked to doctor [NAME i ] about your health? Do not count any time you might have seen a doctor while you were a patient in a hospital, but count all other times your actually saw or talked to a medical doctor of any kind. 20 20

Questionnaire Design 3) Forgetting and other memory problems Example: Low options High responses Responses % Responses % <1/2 hr 7.4 < 2 ½ hr 62.5% ½ to 1hr 17.7 2 ½ to 3 hr 23.4% 1 to 1 ½ hr 26.5 3 to 3 ½ hr 7.8% 1 ½ hr to 2 hr 14.7 3 ½ hr to 4 hr 4.7% 2 to 2 ½ hr 17.7 4 to 4 ½ hr 1.6% >2 ½ hr 16.2 >4 ½ hr 0% Question about television watching can go from 16% to 37% saying that they watched more than 2.5 hours/day, depending on the set categories. 21 21

Questionnaire Design 4) Sensitive (social desirability) questions (recommendations) More time to respondents to answer in order to not misreport information (Krosnick 1999) Question example for strategy for over-reported behaviors: In talking about elections, we often find that a lot of people were not able to vote because they are not registered, they were sick, or they just didn t have the time. How about you did you vote in the elections this November? Use self-administration, computer-administered questionnaire or similar methods to improve reporting. 22 22

Questionnaire Design 5) Failure to follow instructions (mainly for self-administered respondents) If first page is half-page explaining instructions, respondent use to skip that part. 23 23

Questionnaire Design Instructions must be exactly where is needed and not in a separate section 24 24

Questionnaire Design Format of response scale (Network Questionnaires) Binary judgments (least difficult for respondents) about whether respondents have a specified relationship with each actor on the roster. Ordinal ratings of tie strength Rankings: More demanding for respondents. Majority of respondents preferred binary over ranking or rating 25 25

Questionnaire Design. BUT reliability for ratings > binary judgments (Ferligoj & Hlebec, 1999). Recommendations (not only network questionnaires) Use closed questions for measuring attitudes Use 5 point response scale and labeled every scale point. With 9 or 11 point response scale the beginning and the ending. Use ranking (preferences order) only if the respondents can see all the alternatives: otherwise use paired comparisons. 26 26

Questionnaire Design Name Generator Design Studies suggest using minimal modules of name generators: Van Der Poel (1993): 3-item: Discussing a major life change, aid with household tasks, and monthly visiting. 5-items, add: Borrowing household money and going out socially. Bernard et al. (1990): 5 items: Social activities, hobbies, personal problems, advice about important decisions, closeness Burt (1997): 3-item: GSS important matters issue, socializing, and discussion of a job change 27 27

Questionnaire Design Clusters of persons named by relationships (families, workplaces, ) than by similarity of individual features -gender, race or age- (Fiske,1995) Use Recognition rather than Recall (when possible) - Level of forgetting - (Brewer, 2000) Multiple name generators may limit forgetting because persons forgotten for one generator are often name in response to others. Single name generator maybe sufficient for core networks 28 28

Quality in Social Network Data Measurement Comparisons Questionnaire components Network ties 29 29

Quality in Social Network Data In summary, Validity and Reliability of survey data, social networks, can be affected by many characteristics of the measurement instrument ( Name Interpreters). a) Response scale (binary; 5 ordinal ending; 5 ordinal all; line drawing scale; 11 ordinal) b) Response category labels( all categories or end points). c) Name Generator method (Recognition vs free recall). c) Data collection method (face-to-face interview, telephone, web survey, ) e) Question wording (after we obtain the list of alters with name generators, we can ask name interpreter questions by alters or by questions ). f) Layout of the questionnaire: Plain vs Graphical 30 30

Quality in Social Network Data Types of networks analyzed: Social support (Weiss, 1974; Hirsch, 1980; Wills, 1985:, vaux, 1988) 1) instrumental support (aid, material support, or tangible support) is the provision of financial aid, material support, and necessary services), 2) informational support (advice support, appraisal support, or cognitive guidance) involves help in defining, understanding, and coping with problematic events) 3) emotional support (or close support), 4) social companionship (diffuse support and belongingness) relates to time spent with others in leisure activities). 31 31

Quality in Social Network Data I) Frequency of Scientific advice II) Frequency of Collaboration III)Asking for crucial Information IV) Social activities outside the work Tie Characteristics: Demographic variables a) Measures of tie strength Education Frequency of contact Gender Feeling of closeness Age b) Feeling of importance c) Frequency of the alter upsetting the ego 32 32

Hypothesis and Research Questions from Studies Hypothesis and Research Questions from Studies 33 33

Hypothesis and Research Questions from Studies To test whether the recognition data collection technique is more stable than that of free recall (Ferligoj & Hlebec,1999) The size of the social network and age of the respondent should not affect the similarity of free-recall and recognition methods. Reliability and validity of the measurement of tie characteristics for emotional and informational support alters would be higher compared to that for social companionship and material support alters. Kogovsek & Ferligoj (2004) The stability of social support : Emotional support (close and important ties) more stable than Material support (provided by specialized sources). Closeness is not required for providers of material support and informational support. Hlebec & Ferligoj (2002) 34 34

Hypothesis and Research Questions from Studies Hypothesis Predictors Kogovsek & Ferligoj (2005) Measurement instrument Higher quality of measurement with by alters than by questions data collection technique Higher quality of measurement of cognitively demanding questions with the personal interview/by alters Higher quality of measurement of sensitive questions with the telephone interview/by alters Network size Higher quality of measurement in small networks Personal characteristics Higher quality of measurement among female respondents Higher quality of measurement among younger respondents Higher quality of measurement among more educated respondents Personality characteristics Higher quality of measurement among extraverts 35 35

Hypothesis and Research Questions from Studies We might expect that every tie or alter characteristic would be more reliably and more validly measured when the question is posed by alters than by questions ( Kogovšek, Ferligoj, Coenders, Saris, 2002) We therefore expect that cognitively more demanding questions would be more prone to measurement errors in the telephone than in the faceto-face mode, therefore: It is expected that demanding questions would be more reliably and more validly measured in face-to-face interviews than by telephone 36 36

Quality of measurement How to Measure 37 37

Quality of measurement Reliability: The extent to which any questionnaire, test or measure produces the same results on repeated experiments. Random error. Validity: The extent to which any measure measures what is intended to measure (Carmines & Zeller, 1979:12). Systematic error, bias. Convergent validity refers to common trait variance and is inferred from large and statistically significant correlations between measures of the same trait using different methods. Discriminant validity refers to the distinctiveness of the different traits; it is inferred when correlations among different traits are less than one. 38 38

Quality of measurement Test retest stability of networks comparing data collection techniques of network data (Example: Free recall vs Recognition method) The test is performed twice. Giving a group of participants the same questionnaire on two different occasions, the test-retest reliability is the correlation between separate administrations of the test is high. 39 39

Quality of measurement Q U A L I T Y Multilevel Multitrait Multimethod Data collection Questionnaire Design Reliability and validity estimates by country Meta-analysis design (MCA) Meta Analysis results 40 40

Multitrait Multimethod model The MTMM model has rarely been used for measurement quality assessment in social network analysis. Hlebec (1999), Ferligoj and Hlebec (1999), Kogovšek et al. (2002), egocentered networks (TS model). For model identification purposes the MTMM approach usually requires at least 3 repeated measurements of the same variable, using three different methods (Kenny, 1976) Burden on the respondent and increases the cost of data collection. To reduce these problems: Split ballot MTMM experimental design (Saris, 1999; Saris & Coenders, 2000) combinations of just two repetitions (methods). 41 41

Multitrait Multimethod model Reliability based on the classical test theory (Lord & Novick, 1968): Y ij = S ij + eij e Y S Y ij is the response of variable i measured by method j. S ij is called the True Score (Saris & Andrews, 1991) e ij is the random error, related to lack of reliability. True Score is the result of Trait and Method: S ij = m ij M j + t ij T i 42 42

Multitrait Multimethod model T 1 = Trait / variable of interest M = reaction to S= True Score the method Y = observed response e = random error The strength of the relationship between T and S is called Validity Validity = 1- var (M) 43 43

Multitrait Multimethod model Confirmatory Factor Analysis (CFA) specification of the MTMM model: Y ij = m ij M j + t ij T i + e ij Traits (Coromina, Coenders, Ferligoj, 2006): Trait 1: Ask for scientific advice Trait 2: Collaboration with your colleagues concerning research Trait 3: Ask for information /data/software Trait 4: Social activities outside the work 44 44

Multitrait Multimethod model Assumptions (Andrews, 1984): Cov(T i,e ij )=0 ij Cov(M j,e ij )=0 ij Cov(T i,m j )=0 ij It makes possible to decompose the variance of Y ij Trait variance t 2 ij Var(T i ) method variance m 2 ij Var(M j ) random error variance Var(e ij ) 45 45

Multitrait Multimethod model M 1 M 2 e 11 Y 11 Y 21 Y 31 Y 41 Y 12 Y 22 Y 32 Y 42 e 42 e 21 e 31 e 41 e 12 e 22 e 32 T 1 T 2 T 3 T 4 46 46

Multitrait Multimethod model Reliability increases not only with true or trait variance, but also with method variance, which also belongs to the stable or repeatable part of the measurements. Var( S Var( Y ij ij ) ) 2 2 m ijvar( M j ) + tijvar( Ti ) = Var( Y ) Validity, % of variance of the True Score explained by the Trait: ij t 2 ij Var( T Var( S ij i ) ) = 2 ij 2 mvar( M ij t Var( T j i ) + t ) 2 ij Var( T i ) QUALITY, strength of the relationship between Y and T = RELIABILITY * VALIDITY 47 47

Multilevel MTMM Egocentered network data Two-level MTMM models. The mean centred individual scores for group g and individual k can be decomposed into: Y Tgk = Y gk Y Ego (g) Between group component (g) Y Bg = Y g Y Alter 1 Alter 2 Alter n Within group component (k) Y Wgk = Y gk Y g 48 48

Multilevel MTMM WITHIN BETWEEN Between level: Units of measurement are egocentered networks as a whole, average across all egocentric networks Quality of averages studied. Within level: Individual ego-alter 49 49

Multilevel MTMM Decomposition of the sample covariance matrix. Total population covariance matrix Σ T Σ T = Σ B + Σ W Analysis of each component separately: Y ij = m ij M j + t ij T i + e ij Y ij = m Bij M Bj + t Bij T Bi + e Bij + m wij M wj + t wij T wi + e wij Y Bij Y Wij 50 50

Interpretation We obtain two reliabilities and two validities for each trait-method combination: Between and within groups Groups are respondents Different interpretation Individuals are stimuli evaluated by respondents from standard way The between-group reliabilities and validities Quality of the measurement of the egocentered network as a whole (average). If the ego is the focus of interest and these averages are used as data instead of the raw responses regarding individual alters Between-group measurement quality is the relevant one to look at. 51 51

Interpretation The within-group reliabilities and validities Each subject is a separate unit of analysis and thus variance is defined across stimuli presented to the same subject, not across subjects. If the relationship is the focus of interest within group measurement quality. We know the between and within scores add to a total score: Y ij = m Bij M Bj + t Bij T Bi + e Bij + m wij M wj + t wij T wi + e wij Y Bij Y Wij Then 52 52

Interpretation It is possible to compute percentages of variance. Decomposition of variance: Var(Y ij ) = m 2 wij Var(M Wj ) + m 2 Bij Var (M Bj ) + t 2 wij Var(T Wi ) + t 2 Bij Var(T Bi )+ Var(e wij ) + Var(e Bij ) From the decomposition one can: compute overall reliabilities and validities: By aggregating all trait, method, and error components. compute overall percentages of within and between variance: By aggregating all within components and all between components. 53 53

Interpretation compute the percentage of between and within trait variance over the total trait variance. A higher within percentage shows a higher alter variability in the tie characteristics within a network and a lower variability of average values of the tie characteristics between networks. compute a percentage of pure random error variance (i.e., within error variance) over the total variance of the observed variables (grand total, i.e., including all 6 components). The percentage of total variance explained by any of the other 5 components can be computed in a similar way. 54 54

INSOC Project INSOC project International Network on Social Capital and Performance 55 55

INSOC Project Explain the academic PhD students success based on social capital. Develop and refine data collection methods. Develop social network analysis methods. 56 56

INSOC Project Web Survey design Verbal (words and numbers) + rich visual. Special navigational features (progress indicators), animations Lower costs Faster data collection and data analysis process Easy and fast questionnaire modification. Response when convenient Piping: Assigning questionnaire items based on earlier answers from the respondent 57 57

INSOC Project Coverage error!!! Population with nearly universal internet access (PhD students) 58 58

INSOC Project INSOC Questionnaire Design Focus groups Comparable versions (Catalan, Slovenian and Dutch) Two independent translations Pre-test 59 59

Population, sample and data collection PhD students at the Universities of Girona, Ljubljana and Ghent April 2003: To know PhD students promoters Personal interview to promoters name generator questions 1. Name all the teaching assistants (or doctoral assistants) whose research is mainly under your supervision. 2. Name all the researchers of whom you are formally the mentor and who work on or participate in a research project. 3. Name your colleague professors, senior researchers, junior researchers or people working in the private sector with whom you substantially work together on those research projects in which PhD student X [name PhD student] is involved. 60 60

Population, sample and data collection November 2003: e-mail invitations (86 in Girona, 191 in Ljubljana and 233 in Ghent). Follow-up: after 1, 4 and 8, weeks PhD responses Method 1 2weeks PhD responses Method 2 Girona 67 78% 61 91% Ljubljana 118 62% 81 69% Ghent 198 85% 55* 60% Method 2: Different question order, different style of response category labels and different graphical display and lay-out of the questions. 61 61

Population, sample and data collection Responses 80% 70% 60% 50% 40% 30% 20% 10% 0% 1st reminder (2/12/03) srv 2nd reminder (23/12/03) 3rd reminder (23/01/04 to 26/01/04) Supervisors PhD Students 62 62

Traits & Methods Frequency Traits Trait 1: Ask for scientific advice Trait 2: Collaboration with your colleagues concerning research Trait 3: Ask for information/data/software Trait 4: Social activities outside the work sample repetition Factor 1 Factor 2 Factor 3 Girona Only one Main questionnaire by questions all labels plain Girona Only one Follow-up by alters end labels plain Kogovsek & Ferligoj (2004, 2005) Group N First interview Second interview 1 320 Face-to-face/by alters Telephone/by alters 2 311 Face-to-face/by alters Telephone/by questions 3 402 Telephone/by alters Telephone/by questions 63 63

Traits & Methods 64 64

Traits & Methods 65 65

Traits & Methods Example of Method 2 in Girona 66 66

MTMM Results -- University of Girona Advice Collaboration Info Social act within trait variance M 1 67.5% 65.3% 55.6% 67.0% M 2 66.1% 59.3% 47.7% 42.4% within method variance M 1 0.0% 0.0% 0.0% 0.0% M 2 9.6% 9.2% 12.3% 13.1% within error variance M 1 13.3% 17.2% 17.3% 14.5% M 2 7.0% 12.3% 16.3% 25.2% between trait variance M 1 12.2% 13.1% 18.5% 11.4% M 2 12.0% 11.9% 15.9% 7.2% between method variance M 1 0.4% 0.4% 0.6% 0.9% M 2 5.2% 5.0% 6.7% 7.1% between error variance M 1 6.6% 3.9% 8.0% 6.3% M 2 0.0% 2.3% 1.2% 4.9% Goodness of fit: Yuan-Bentler χ2= 103.1 with 33df. TLI:.945 67 67

MTMM Results -- University of Girona % of within trait variance over all trait variance. The results show that most of the error free variance corresponds to the within level. This means that egos really discriminate between the different alters. T 1 T 2 T 3 T 4 t wij 2 Var(T Wi )/ [ t wij 2 Var(T wi ) + t Bij 2 Var(T Bi )] 84.6% 83.3% 75.0% 85.4% Within level Between level Overall level T 1 T 2 T 3 T 4 T 1 T 2 T 3 T 4 T 1 T 2 T 3 T 4 Reliability coef. M 1.91.89.87.91.81.88.84.81.90.89.86.89 M 2.96.92.89.83 1.00.94.98.86.96.92.91.84 Validity coef. M 1 1.00 1.00 1.00 1.00.98.98.98.96 1.001.001.00.99 M 2.93.93.89.87.83.84.84.71.92.91.88.84 Quality coef. M1.91.89.87.91.79.86.82.78.90.89.86.88 M2.89.86.79.72.83.79.82.61.88.83.80.71 68 68

Meta analysis Design Meta-Analysis Design and Results 69 69

Meta analysis Design Summarizing the results of studies carried out by the universities of Girona, Ljubljana and Gent. University sample repetition Factor 1 Factor 2 Factor 3 Girona (Spain) Only one Main questionnaire by questions all labels plain Girona (Spain) Only one Follow-up by alters end labels plain Ljubljana (Slovenia) 1 and 2 Main questionnaire by questions all labels plain Ljubljana (Slovenia) 1 Follow-up by questions end labels Graphical Ljubljana (Slovenia) 2 Follow-up by alters all labels Graphical Ghent (Belgium) Only one Main questionnaire by questions all labels plain Ghent (Belgium) Only one Follow-up by alters end labels plain Categorical predictors (country, trait and factor 1 to factor 3) Multiple classification analysis (MCA) was used Estimation of the contribution of each factor on reliability and validity. 70 70

Meta analysis Design Question order: by alters or by questions. 71 71

Meta analysis Design Formulation by alters, with end labels and with a plain lay-out Kogovšek et al. (2002) by alters seems to be more reliable for telephone interviews. 72 72

Meta analysis Design Response category labels: for all categories or for the end points of the response scale. Revilla & Saris (2011) find more quality in all labeled. Költringer (1995) reports no effect of the way of labeling on either reliability or validity. Personal and telephone interviews and mostly related to attitudinal (i.e., not network) variables using vague category labels of the type rather satisfied, completely agree Our study: Category labels are not vague quantifiers. Precise actual frequencies of behavior and thus additional labels may help respondents give precise answers about the frequency of contact with their social network. 73 73

Meta analysis Design Lay-out of the questions and the web page: plain or graphical display. Dillman et al. (1998b), who suggest that using a plain questionnaire provided better results (response rate, less time ) than a graphical display version. Deutskens et al. (2004) found that visual effects actually increase response quality. 74 74

Meta-analysis results Means of each level of all factors, corrected by the levels of other factors country trait Girona Ljubl. Ghent trait 1 trait 2 trait 3 trait 4 % of between trait variance.133.088.117.079.125.118.111 % of within trait variance.610.612.555.674.654.552.508 % of between method variance.024.025.002.017.015.022.030 % of within method variance.049.074.077.052.047.072.114 % of between error variance.049.064.057.053.039.063.078 % of within error variance.152.166.189.137.132.186.219 Between reliability.874.735.839.794.875.804.698 Between validity.921.907.943.916.945.919.876 Within reliability.899.895.875.916.917.877.856 Within validity.974.932.931.963.965.936.887 Overall reliability.885.882.869.892.913.867.849 Overall validity.963.923.935.955.960.932.867 % of trait variance at within level.823.864.826.895.840.823.804 75 75

Meta-analysis results factor 1 factor 2 factor 3 by questions by alters all labels end labels plain lay-out graph. lay-out % of between trait variance.125.082.103.116.105.117 % of within trait variance.650.508.595.600.593.608 % of between method variance.013.031.006.042.026.008 % of within method variance.035.120.070.074.082.045 % of between error variance.065.046.061.054.059.058 % of within error variance.136.215.181.147.163.183 Between reliability.785.809.777.825.789.806 Between validity.948.880.961.856.902.958 Within reliability.912.857.884.903.894.883 Within validity.975.888.942.932.927.965 Overall reliability.895.852.873.893.884.868 Overall validity.967.893.947.914.922.962 % of trait variance at within level.835.855.850.830.844.841 76 76

Meta-analysis results Overall reliabilities and validities are acceptable: around or above 0.85 By traits (pattern is consistent at different levels): Collaboration (T2) Advice (T1) Asking for crucial information (T3) Socializing (T4) Overall reliability and validity is higher when: Social network questions in a survey are organized by questions All labeled categories Graphical display questionnaire design (not within validities) 77 77

Meta-analysis results Country: No significant effect on reliability or validity. % of trait variance at the within level: Advice Collaboration Ask for information Socializing Scientific advice tends to be asked rather often to some group members and not often to other group members by the same ego, but all egos have a rather similar frequency average, while this occurs to a lesser extent for socializing. 78 78

Meta-analysis results from Other Studies Meta-Analysis Results from other Studies 79 79

Meta-analysis results from Other Studies Kogovšek, Ferligoj, Coenders, Saris (2002) Coromina, Coenders, Kogovsek (2004) By type of networks: +: Frequency of contact is measured with the highest reliability Feeling of importance of the alter Feeling of closeness between ego and the alter - : Frequency of being upset. Frequency of contact is less reliably measured over the telephone (speed of telephone communication) compared with the degree of closeness between the ego and the alter or importance of the alter ) 80 80

Meta-analysis results from Other Studies Answering the question about frequency of contact is probably more cognitively demanding than answering about the degree of closeness for which the answer may be more readily available Cognitively more demanding name interpreter questions would be more reliably measured by face-to-face than by telephone mode. Kogovsek & Ferligoj (2005) Reliability Results : Method +reliable: Telephone/by alters (except contact) Personal/by alters -reliable: Telephone/by questions 81 81

Meta-analysis results from Other Studies When Telephone mode only by alters (higher quality). Telephone speed of the mode smaller elicited networks (would reduce respondent burden and therefore lead to data of better quality) BUT mean network size was actually smaller in the face-to-face (6.84) than in the telephone mode (7.24). Method: +reliable: face to face +valid: telephone (anonymity of phone) -reliable: telephone -valid: face to face Cognitively more demanding questions would be measured more reliably by faceto-face mode, and that questions that are potentially more sensitive would be measured more reliably in the telephone by alters condition. 82 82

Meta-analysis results from Other Studies It is quite possible that context effects are stronger when the data is collected by questions, since the respondent answers the same question for all alters first and not vice versa. If we consider a question relating to feelings of closeness, it is more likely that the respondent would compare the alters while answering the question. In that case, it would not really be the actual feelings of closeness towards each individual alter that would be measured, but the feelings of closeness relative to previous alters on the list. Coromina et al. (2004): Not so clear that telephone method is better than face-to -face method at the between level. Telephone produces better quality data at the within level, and an analysis of the S B such as the one done by Kogovšek et al (2002) is inevitably contaminated by the within level structure according to E(S B )= Σ W + cσ B 83 83

Meta-analysis results from Other Studies Social support (reliability) Ferligoj & Hlebec (1999) +: Emotional support (discussion of important matters) +: Informational support (Illness ) Social Companionship (birthday party) -: Instrumental support (material) (importance of the topic to the respondents). Response scale (the most important predictor of reliability estimates) Hlebec & Ferligoj (2002) +: 5 point category all labels. +: 11 point category scale ending 5 point category scale ending labels Line drawing scale - : Binary scale 84 84

Meta-analysis results from Other Studies Response scale +reliable: all labelled -reliable: ending labels Name Generator: Recognition data collection enhances the reporting of both strong and weak ties, especially when the measurement scales employed can also measure the strength of ties. Ferligoj & Hlebec (1999) ( Recognition / Free recall ) no differences Interview design (2nd most powerful predictor for reliability estimates) +reliable : Two measures (when two measures of the same trait are presented in the same interview) -reliable: One measure (a question is presented alone in a questionnaire) 85 85

Meta-analysis results from Other Studies Network size (not significant) 1 5 +valid: 1-5 6+ -valid: 6+ (larger proportion of weak ties and measurement error more pronounced) With smaller networks (important alters, consistent answers) the data collection technique (by alters/by questions) is more important than the data collection mode (telephone/face-to-face). In larger networks: (seems that context effects are no longer so prominent). +valid: telephone -valid: face-to-face 86 86

Meta-analysis results from Other Studies Type of question +reliable: Behavior +valid: behaviour.975 -reliable Emotional, but not very different -valid: emotional.958 Education Up to compl. second. (no effect) College or more Age +reliable: 40 years or less +valid: 40 years or less -reliable: 41+ -valid: +41 (memory or hearing problems; more weak ties...) 87 87

Meta-analysis results from Other Studies Gender +reliable: female +valid: male -reliable: male -validity: female (maybe because women have larger networks and more weak ties and might increase method effect) Interaction Age / Gender -valid: older women +valid: other categories Interaction Method / Gender Female/method Male/method +valid: face-to-face (.957) +valid: telephone (.999) - valid: telephone (.930) -valid: face-to-face (.928) Being interviewed about personal relationships by telephone suits both genders better, but face-to-face interviews suit women better than men. 88 88

Project Conclusions Considering egocentered network data as hierarchical, Multilevel recommended. Qualities at Within, Between and Overall level. Combination of % from decomposed variance. Egocentered Network data quality collected by Websurvey: ordered by Questions All categories of the scale labeled Overall and Within Graphical display validity 89 89