DATA COLLECTION AND ANALYSIS Quality Education for Minorities (QEM) Network HBCU-UP Fundamentals of Education Research Workshop Gerunda B. Hughes, Ph.D. August 23, 2013
Objectives of the Discussion 2 Discuss important principles of research when engaging in the data collection and analysis phases of the project? Operationalize the variables in the research question(s) Choose appropriate data collection methods Choose appropriate data collection tools Identify and collect from the proper data sources Be acutely aware of timing Avoid sampling error and bias Ensure privacy and confidentiality Store data properly Define the different types of validity and reliability and the relationship between these two important characteristics of data and the results of data analysis. Describe the types of measuring instruments used to collect data in qualitative and quantitative studies.
Variables 3 A variable is a construct that can take on two or more values. A constant takes on only one value. For Data Collection, variables must be operationalized. That is, the researcher must define a rule for how a variable is to be measured. Interest in Science may be operationalize as (1) the score on a science interest inventory or questionnaire, or (2) the number of science courses that an individual took during grades 9-12.
Quantitative and Qualitative Variables 4 Quantitative variables are ordinal, interval and ratio variables. Variates differ in magnitude. scores, heights, speed, age, weight Qualitative Variables are nominal or categorical variables. Variates differ in kind. political party affiliation; eye color; gender; race/ethnicity Quantitative variables exist on a continuum that ranges from low to high or less to more. Qualitative variables are qualities about how people or objects differ with no relation to natural order.
Measurement Scales 5 Measurement is the process of assigning numbers to characteristics of an object or person. The four measurement scales: Nominal Ordinal Interval Ratio Data collected on different measurement scales require different methods of statistical analysis.
Nominal and Ordinal Scales 6 Nominal scales define variables that are categorical. Examples: gender, employment status, marital status, type of school. Ordinal scales classify persons or objects and they also rank them in terms of the degree to which they possess a particular characteristic. Examples: class rank, order of finishing a race. These scales classify persons or objects into two or more categories. It is the lowest level of measurement. These scales permit comparisons of higher/lower, for example, but do not indicate how much higher or lower.
Interval and Ratio Scales 7 Interval scales have all the properties of nominal and ordinal scales, and also have equal intervals. Ratio scales have all the properties of the other scales and represents the highest, most precise level of measurement. Examples: Achievement, attitudes, motivation, etc. (educational measures). Examples: Height, weight, time, distance, speed (physical measures). Interval scales do not have a true zero point. A score of zero may indicate the lowest level of performance possible, but does not indicate total absence of the characteristic. Ratio scales have a true zero point. It is meaningful to talk about no distance. Ratio scales also permit comparisons by ratios (Aisha weighs twice as Linda).
Types of Scores from Instruments 8 Raw Scores The number or point value of items a person answered correctly on an assessment Norm-referenced Scoring A scoring approach in which an individual s performance on an assessment is compared to the performance of others Criterion-referenced Scoring A scoring approach in which an individual s performance on an assessment is compared to a predetermined external standard. Self-referenced Scoring A scoring approach in which an individual s repeated performances on a single assessment are compared over time.
Types of Scores from Instruments 9 Raw Scores Cedric earned a raw score of 92 on his biology test. Norm-referenced Scoring Jenelle s has a percentile rank of 92 on her algebra test. Criterion-referenced Scoring Richard earned 92% on his chemistry test. Self-referenced Scoring Sheri scored 92% higher on this week s geometry quiz than she did on last week s geometry quiz.
Independent and Dependent Variables 10 Experimental Research The independent variable (causal or manipulated variable) is the intended cause of the dependent variable (outcome, effect, or criterion variable). Non-Experimental Research The independent variable (status variable not manipulated) is the variable that logically has some effect on a dependent variable. Examples of IVs include: gender, race-ethnicity, marital status, eye color, employment status, etc.
11 Research Questions: Identifying Independent and Dependent Variables Do ninth-grade girls will have different attitudes toward science than ninth-grade boys? Is there a relationship between middle-school students grades and their self-confidence in science and math? Is personalized instruction from a teacher more effective for increasing students critical thinking skills than computer-based instruction?
Data Collection Methods 12 Quantitative Methods Tests Surveys Questionnaires Rubrics Checklists Qualitative Methods Observations Interviews Document Reviews Focus Groups Photographs/Drawings Recordings Social Media/E-mail Phone Calls/Recordings
Formats for Data Collection Tools 13 Selection & Supply Methods: Used predominately by quantitative researchers; paper and pencil or electronic. Selection Methods Multiple-choice, true-false, and matching items Supply Methods Administer a short answer/essay question tests; fill in the blank items; and performance assessments (assessment of a product or a process) Rubrics Interviews, Focus Groups, Observations: Used predominately by qualitative researchers. Data are collected by observation, conversation, or extended written communication.
Validity of Assessment Results 14 Validity is the most important characteristic of the assessment results; is concerned with the appropriateness of the interpretations made from assessment results; is best thought of in terms of degree; is specific to the interpretation being made and to the group being assessed.
Types of Validity 15 Content Validity- the degree to which the assessment results are a reflection of the intended content area. Criterion-Related Validity- determined by relating performance on one measure to performance on a second measure. Concurrent Validity (SAT scores and ACT scores) Predictive Validity (SAT scores and freshman g.p.a.; GRE scores and success in first year of graduate school)
Types of Validity 16 Construct Validity- is the most important form of validity because it asks the fundamental validity question: What is this assessment tool really measuring? Examples Mathematics tests and reading levels Reading and language tests Interest in STEM Consequential Validity - the extent to which the use of assessment results has intended or unintended effects for the user. Test scores and graduation, teacher certification, teacher effectiveness A narrowing of the curriculum and classroom teaching to focus only on what is tested
Reliability of Measuring Instruments 17 Reliability is the degree to which a test consistently measures whatever it is measuring. Reliability is expressed as a reliability coefficient which is obtained by using correlation. Error is present in all measurement. High reliability means small errors of measurement.
Types of Reliability 18 Test-retest Reliability Equivalent-forms Reliability Internal Consistency Reliability Split-Half Reliability Cronbach s Alpha Reliability Scorer/Rater Reliability
19 Validity and Reliability of Instruments A valid test is always reliable, but a reliable test is not always valid.
Validity and Reliability of Observational Data 20 Factors that influence the validity and reliability of observational data: The research question Errors of measurement Training of the observer(s) results in familiarity with the setting the culture the focus of the study the observation protocol how to record data (and not summaries or personal opinions)
Data Collection Procedures and Environments 21 Every effort should be made to ensure appropriate data collection procedures and ideal environments (e.g., test administration conditions such as proper lighting, minimum noise level, comfortable seating, etc.) Failing to administer procedures precisely or altering the administration procedures, especially on standardized tests, lowers the validity of the test. High noise levels may be a distraction to study participants during data collection.
DATA ANALYSIS
Types of Statistics Descriptive Statistics Descriptive statistics are used to organize, describe, and summarize a set of data. Inferential Statistics Inferential statistics are used to draw inferences about the conditions that exist in a population from study of a sample drawn from that population.
Types of Statistics Analyses in Quantitative Research Descriptive Statistics Measures of central tendency Mean, median, mode Measures of variability Range, variance, standard deviation, semiinterquartile range Effect Size Inferential Statistics Parametric Tests t-tests, Analysis of variance (ANOVA), Regression analysis Non-parametric Tests Chi-Square test; the sign test
Inferential Statistics Types of hypotheses Research hypothesis Null hypothesis Tests of the null hypothesis among Relationships Means Proportions
Correlational Techniques Correlation: A measure of the degree of association between two or more variables. Pearson s correlation, r; (both variables are continuous and quantitative) Mathematics achievement and mathematics anxiety Phi coefficient is the Pearson correlation for two variables that are both qualitative and dichotomous Gender and Science major or not Spearman s rho (both variables are expressed as ranks) Class rank and ranking in a science fair competition
Tests of Significance Simple Analysis of Variance: one independent variablegender, and one dependent variable - college gpa. Multi-Factor Analysis of Variance: (two or more independent variables - gender, SES, participation in summer bridge program; and one dependent variable, college freshman gpa). Multiple Regression: tells us how much of the variance in the dependent variable (e.g., on-time graduation ) is explained by the set of independent variables (e.g., high school gpa, SAT/ACT mathematics and verbal scores, freshman college gpa).
Correlational Techniques Correlation: A measure of the degree of association between two or more variables. Bi-serial correlation (one variable is continuous and quantitative and the other would be, expect it has been reduced to two categories) Multiple correlation, R, is the Pearson correlation between the variable to be predicted and the bestweighted combination of several predictors. To calculate R, we must know Pearson s r between each pair of variables.
Data Analysis in Qualitative Research Engage in a great deal of analysis before data collection is complete. Reflect on two questions Is the research questions still answerable? Are the data collection techniques catching the kind of data that is wanted and filtering out the data that is not wanted? Avoid premature actions based on early analysis and interpretation of data.
Data Analysis in Qualitative Research Qualitative data analysis is a cyclical, iterative process of reviewing data for common topics or themes. One approach is to follow three iterative steps: Reading and memo-ing Describing what is going on in the setting Classifying research data
Data Analysis in Qualitative Research Constant Comparative Analysis Phenomenological Approaches Ethnographic Methods Narrative Analysis & Discourse Analysis
Data Analysis Strategies Identifying Themes -- emerges for ideas found in the review of the literature and the data collection. Coding -- the process of marking units of text with codes or labels as a way to indicate patterns and meaning in data. Asking questions Who is centrally involved? ; What major activities or issues are relevant to the problem? Then seeking answers in the data. Concept Mapping a visual display of the major influences that have affected the study.
33 Questions?