Measurement and measures. Professor Brian Oldenburg

Transcription

1 Measurement and measures Professor Brian Oldenburg

2 Learning objectives 1. To identify similarities/differences between qualitative & quantitative measures 2. To identify steps involved in choosing and/or developing appropriate measures 3. To describe features of good questions 4. To understand basic principles of test theory and psychometrics.

3 Steps involved in research 1. Literature review and Formation of the topic 2. Hypothesis 3. Conceptual/theoretical definitions 4. Operational definition/methods 5. Gathering of data 6. Analysis of data 7. Test, revising of hypothesis 8. Conclusion, iteration if necessary

4 Developing mesures for your study What to measure? Choosing measure of the variable(s) Use existing measure(s) Adapt existing measure(s) Develop new measures (s0 How to measure?

5 What to measure? How to decide?

6 What is psychometrics? The field concerned with the theory and technique of (psychological) measurement The field is primarily concerned with the construction and validation of measurement instruments, such as questionnaires, tests and assessments. Two major research tasks of this field - construction of instruments and procedures for measurement - development/refinement of theoretical approaches

7 Definition of measurement in social sciences the assignment of numerals to objects or events according to some rule

8 Theoretical and analytic approaches Classical test theory, item response theory Rasch model for measurement Factor analysis, multidimensional scaling, Structural equation modeling, path analysis etc.

9 Classical testy theory X = T + E Observed score True score Error

10 Key concepts Reliability measuring a construct consistently across time, individuals, and situations. Validity measuring what is intended to mesure. A measure may be reliable without being valid, however, reliability is necessary, but not sufficient, for validity.

11 Measures Measures are often categorised as being either quantitative and/or qualitative. What matters is that the chosen measures fit the intended purposes of the research!

12 Qualitative and Quantitative Paradigms The qualitative paradigm concentrates on investigating subjective data, in particular, the perceptions of the people involved. The intention is to illuminate these perceptions and, thus, gain greater insight and knowledge. The quantitative paradigm concentrates on what can be measured. It involves collecting and analysing objective (often numerical) data that can be organised into statistics.

13 Qualitative and Quantitative Research Qualitative Research Quantitative Research Also known as interpretative / responsive positivist /hypothetico-deductive Type of reasoning Link with concepts Action (usually) inductive identifies concepts sometimes only describes a situation BUT in actionresearch openly intervenes (usually) deductive identified concepts and investigates relationships tests relationships between concepts Outcome illuminates the situation accepts or rejects proposed theory Approach to validity truth seen as context bound (socially constructed) truth seen as objective and universal

14 Comparison: Quantitative vs. Qualitative Research: Quantitative Methods: Formulate hypothesis Observe events/present questionnaire with fixed answers. Tabulate responses Summarize data Analyze and draw conclusions Qualitative Methods: Observe events and/or ask questions with open-ended answers. Record observations Interpret observations Return for new and refined observations Review data and draw conclusions Formulate hypothesis or theory

15 Role of Measurement Creates variables that are amenable to statistical analysis External, easily verifiable Uses discourse as data, long descriptive text using people s own words and dialogues. Seeks techniques to explore a phenomenon from under the skin of another

16 Deductive vs. Inductive Variables are conceived a priori Research question/hypothesis comes first No preconceived assumptions/ suspend your beliefs Patterns emerged from the collected data after the fact

17 Interventions vs. Naturalistic Interventions are seen as a way of discovery Manipulate the environment and measure changes Sensitive to the effects of the research(er) on the population studied Try to develop trust in the population being studied

18 Role of the Researcher Researcher s judgement essential before and after Researcher should be absent during experiment Researcher becomes the instrument Tries to suspend any predispostions and assume it is being experienced for the first time

19 Reduction vs. Holist Truth is to be found incrementally, one small tweak after another Science has always found out about phenomena little at a time Humanistic perspective urges a holist approach When we reduce people to variables, we lose the larger understandings possible

20 Determining the Research Approach Use quantitative if your research problem requires you to: Measure Variables Assess the impact of these variables on an outcome Test existing theories or broad explanations Apply results to a large number of people Use qualitative if your research problem requires you to: Learn about the views of the people you plan to study Assess a process over time Generate theories based on participant perspectives Obtain detailed information about a few people or research sites.

21 Quantitative Designs and Uses Intervention Research Explaining whether an intervention influences an outcome for one group as opposed to another group Associating or relating variables in a predictable pattern for one group of individuals Non-Intervention Research Describing trends or characteristics for the population of people Experimental Research Correlational Research Descriptive Research

22 Qualitative Designs and Uses Exploring the shared culture of a people group Exploring common experiences of individuals to develop a theory Exploring individual stories to describe the lives of people Ethnographic Research Grounded Theory Research Narrative Research

23 Research Methods Qualitative Research Quantitative Research Hybrid Ethnographic Grounded theory Narrative research Case Study Action Research Experimental Correlational Descriptive Current Differences Differences Over Time Basic Descriptions Case Study Action Research

24 Hybrids: Combined Designs and Uses Case Study Qualitative design w/ Quantitative data collected. Action Research Forming a theory in advance and collecting qualitative observations to quickly see if the theory has any validity or application in a small and preselected population.

25 Criteria for Judging Research Traditional Criteria for Judging Quantitative Research internal validity external validity objectivity reliability Alternative Criteria for Judging Qualitative Research credibility transferability confirmability dependability Lincoln, YS. & Guba, EG. (1985). Naturalistic Inquiry. Newbury Park, CA: Sage Publications.

26 Lincoln and Guba's Evaluative Criteria Lincoln and Guba posit that trustworthiness of a research study is important to evaluating its worth. Credibility - confidence in the 'truth' of the findings Transferability - showing that the findings have applicability in other contexts Dependability - showing that the findings are consistent and could be repeated Confirmability - a degree of neutrality or the extent to which the findings of a study are shaped by the respondents and not researcher bias, motivation, or interest.

27 Measurement Methods How you collect data and conduct measurement? Process indicators may be collected using a variety of collection tools, such as: Attendance records Lesson reports Sign-in sheets Activity logs

28 Types of Indicators Outcome indicators measure your outcome objectives. Health Promotion outcomes: Knowledge Attitudes Behaviors Healthcare outcomes Death Disease Disability

29 Methods Outcome indicators may be collected using a variety of collection tools, such as: Survey Participant interviews Focus groups Observations Measurements

30 Choosing the right method Example: there are many methods of measuring physical activity few are appropriate for large population groups Some are very accurate (e.g. DLW and indirect calorimetry as measures of energy expenditure), but do not provide information about behaviour and are very expensive. diaries and logs provide rich behavioural information, but are burdensome for respondents and require a great deal of coding for the researcher.

31 Increasing practicality PA Measurement options Global self-assessments s Recall s Pedometers o Accelerometers o Heart rate o PA logs s Diaries s Direct observation o O Objective S Subjective DLW Indirect o calorimetry o Increasing accuracy Acknowledgment: Fiona Bull; Stewart Trost

32 Designing good questions/questionnaire Keep questions as short as possible Use simple words Relax your grammar Assure a common understanding Start with interesting questions Avoid leading questions Avoid double negatives

33 Balance rating scales Don t make the list of choices too long Avoid difficult choices Avoid difficult recall questions Use closed-ended questions rather than open-ended Pre-test and pilot Give clear instructions