C2 Training: June 8 9, 2010 Data Evaluation: Initial screening and Coding Adapted from David B. Wilson and Mark W. Lipsey The Campbell Collaboration www.campbellcollaboration.org
Overview Coding protocol: essential feature of systematic review Goal: transparent and replicable description of studies extraction of findings Forms should be part of C2 protocol
Topics Eligibility criteria and screening form Development of coding protocol Assessing reliability of coding
Study Eligibility Criteria Flow from research question Identify specifics of: Defining features of the program/policy/intervention Eligible designs; required methods Key sample features Required outcomes Required statistical data Geographical/linguistic restrictions, if any Time frame, if any Also explicitly states what is excluded
Study Eligibility Screening Form Develop a screening form with criteria Complete form for all studies retrieved as potentially eligible Modify criteria after examining sample of studies (controversial) Double-code eligibility Maintain database on results for each study screened Example from MST review in handouts
Screening Form Effects of Multisystemic Therapy (MST) Initial Screening Form 1.0 1. Is this paper about MST (perhaps in addition to other topics)? 1. No 2. Yes 99. Can t tell 2. What kind of paper is this? 1. MST outcome evaluation 2. Review of MST outcome studies 3. Descriptive, correlational, or case study 4. Theoretical or position paper, editorial, or book review 5. Practice guidelines or treatment manual 6. Other 99. Can t tell
Effects of Multisystemic Therapy (MST): Eligibility Screening Form 1.2 1. Does this study include two or more parallel cohorts? Note: This means that at least two groups that received different treatments were assessed at the same point in time. 1. No 2. Yes 99. Can t tell IF NO THEN STOP 2. Is this a randomized experiment? 1. No 2. Yes 99. Can t tell IF NO THEN STOP 3. Does this study include a licensed MST program? 0. No 1. Yes 99. Can t tell IF NO THEN STOP 4. Does this study include youth ages 10-17 with social, emotional, or behavioral problems? 0. No 1. Yes 99. Can t tell IF NO THEN STOP
Screening Coding Guide for Internet-based Interventions for English Language Learners Report Characteristics 1. First author (Last, initials) 2. Journal 3. Volume 4. Pages Inclusion Criteria 5. Are participants English language learners? Yes/No If no stop 6. Does each student in the intervention group have Internet access at school and/or at home? 7. Does the study English language skills/proficiency as an outcome measure? Yes/No Yes/No If no stop If no stop 8. Are the participants in the U.S.A.? Yes/No If no stop 9. Is the study a between-group? Yes/No If no stop
Coding practice exercise 1 For the articles provided, code Levels 1 and 2 from the MST coding sheet Compare your codes with those created by someone else. Do you agree?
Development of Coding Protocol Goal of protocol Describe studies Differentiate studies Extract findings (effect sizes if possible) Coding forms and manual Both important Sample coding item from form Sample manual instructions for item
Development of Coding Protocol Types of Information to Code Setting, study context, authors, publication date and type, etc. Methods and method quality Program/intervention Participants/clients/sample Outcomes Findings, effect sizes
Types of Information to Code Setting, study context, authors, publications date and type, etc. Multiple publications; study vs report Geographical/national setting; language Publication type and publication bias issue Publication date vs study date Research, demonstration, practice studies Example from MST review in handouts
Types of Information to Code Methods: Basic research design Nature of assignment to conditions Attrition, crossovers, dropouts, other changes to assignment Nature of control condition Multiple intervention and/or control groups Design quality dimensions Initial and final comparability of groups Treatment-control contrast treatment contamination blinding
Types of Information to Code Methods: Other aspects Issues depend on specific research area Procedural, e.g., monitoring of implementation, fidelity credentials, training of data collectors Statistical, e.g., statistical controls for group differences handling of missing data
Types of Information to Code Method quality ratings (or not) More than 200 scales and checklists available, few if any appropriate for systematic reviews (Deeks et al., 2003) Overall study quality scores have questionable reliability/ validity (Jüni et al., 2001) Conflate different methodological issues and study design/ implementation features, which may have different impacts on reliability/validity Preferable to examine potential influence of key components of methodological quality individually Weighting results by study quality scores is not advised!
Cochrane risk of bias framework Focus on identifying potential sources of bias in studies: Selection bias - Systematic differences between groups at baseline Performance bias - Something other than the intervention affects groups differently Attrition bias - Participant loss affects initial group comparability Detection bias - Method of outcome assessment affects group comparisons Reporting bias - Selective reporting of outcomes
Types of Information to Code Program/Intervention General program type (mutually exclusive or overlapping?) Specific program elements (present/absent) Any treatment received by the comparison group Treatment implementation issues integrity amount, dose Goal is to differentiate across studies Examples
Types of Information to Code Participants/clients/sample Data is at aggregate level Mean age, age range Gender mix Racial/ethnic mix Risk, severity Restrictiveness; special groups (e.g., clinical) Examples
Types of Information to Code Outcome measures Construct measured Measure or operationalization used Source of information Composite or single indicator (item) Scale: dichotomous, count, discrete ordinal, continuous Reliability and validity Time of measurement (e.g., relative to treatment) Examples
Types of Information to Code Findings Compute effect sizes when possible May need to aggregate data or reconfigure findings Add back the dropouts Compute weighted means of subgroups (e.g., boys and girls) Code data on which computations based (common situations) We will look at this part of the coding in the next section
Development of Coding Protocol Iterative nature of development Structuring data Data hierarchical (findings within studies) Coding protocol needs to allow for this complexity Analysis of effect sizes needs to respect this structure Flat-file (example) Relational hierarchical file (example)
Data extraction Double data extraction Cohen s kappa Agreement on key decisions Study inclusion/exclusion, key characteristics, risk of bias, coding of results Pilot-test and refine codes!
Example of a Flat File Multiple ESs handled by having multiple variables, one for each potential ES. Note that there is only one record (row) per study
Example of a Hierarchical Structure Study Level Data File Note that a single record in the file above is related to five records in the file to the right Effect Size Level Data File