Experimental Research Outline: Strategy of experimental research (Platt) What are experiments and how to do experimental research, Different kinds of experimental designs. Experimental Research Strategy (Platt) Systematic and precise method of scientific thinking Accumulative method of inductive inference Can contribute to rapid scientific progress Steps in Experimental Research 1. Devise alternative hypotheses (Existing theory) 2. Devise crucial experiments with alternative possible outcomes, each of which exclude one or more possible hypotheses, (Experiment) 3. Conduct the experiment, get a clean result, (Outcome) 4. Back to step 1, making sub-hypotheses, or sequential hypotheses to refine the possibilities, exclusion and induction (exclusion and building the inductive (logical) tree) Inductive Inference Useful for exploring the unknown Distinct from deductive reasoning; nature provides us the answers Challenge is to pose the right questions; clever choice of hypothesis, experiment, outcome, and exclusion. Origins of the experimental approach Francis Bacon (interconnecting theory and experiment) building the conditional inductive tree, consecutive inductive inferences) Karl Popper, falsificationism, there is no such thing as proof in science, science advances through a series of disproofs or falsification. Assertions in science have to be falsifiable,.. it must be possible for all empirical scientific systems to refuted by experience. Fisher s work in the 1930 s and 40 s in the area of statistical inference. 1
Experimental Research Methods Experiment : Features What is an experiment? Manipulation of one or more variables by the experimenter to determine the effect of this manipulation on another variable. Carefully designed and executed plan for data collection and analysis to test specific hypotheses. Examples of hypotheses? Typically held at the discretion of the researcher. Ability to use various controls to isolate sources of variation. Ability to explore cause-effect relationships. Well-designed experiments can permit causal inferences to be made. Phases in Experimental Studies Experimental Design 1. Formulation of one or more hypotheses. - Usually deductions or derivations from theoretical explanations (of the behavioural phenomenon) or strong hunches/speculations. 2. Translation of the hypotheses into a set of treatment conditions and appropriate experimental design. 3. Conduct the experiment, collect the data 4. Statistical analysis of the data, interpretation of the results and writing up. Independent Variables: The variable(s) manipulated in the experiment. (also called manipulated variable, treatment variable or factor). Typically nominal (categorical) variable. Dependent Variable(s) Measure(s) that capture (performance) of the phenomenon Control or Nuisance Variables Undesired sources of variation in an experiment that affect the dependent variable measurement, Typically of three types: -organismic -environmental -experimental task. Approaches to control the nuisance variable Random assignment of subjects to treatment groups, Holding the (pre-identified) nuisance variable constant for all subjects, Statistical control using Analysis of Covariance (ANCOVA). 2
Sources of Bias Experiments - Advantages Experimenter cues Demand characteristics Evaluation apprehension Hawthorne Effect Negativistic subject Possibility of a variety of manipulative and statistical controls, Random assignment of subjects greater precision and higher confidence in specifying and testing causal relationships, Manipulation Checks possible. May help identify issues and problems previously unrecognised. Experiments - Disadvantages Lab experiments Problems associated with lab settings, Some phenomenon cannot be studied under controlled conditions, Limitations imposed by moral concerns. High degree of precision and internal validity possible; internal validity is a measure of the degree to which the study establishes that the treatment actually produced the effect causality. Potentially low external validity questions about the extent to which the results are generalisable to other populations, settings, organisations, groups, times etc. Procedures 1. Getting Ethics Committee Approval 2. Cover Story description and purpose 3. Recruiting participants - Sample selection - Reference to criterion population - Remuneration and motivation 3. Training the participants 4. Preparing the setting 5. Controlled manipulation of independent variable(s) 6. Manipulation checks 7. Precise measurement of dependent variable(s). Quasi-experimental Designs Used when it is impossible or difficult to perform true, controlled experiments, Particularly in organisational settings Essentially compromise designs Used when time,cost, and practicality are critical. 1. One shot design 2. One group pretest-posttest design 3. Static Group Design 3
Observational Methods Typically questionnaire surveys Strengths: Quantitative data Transparency and some level of objectivity Succinct and easily understood Comparability/reproducibility Can deal with complex problems/issues Issues: Surveys: Interviewer completion Respondent completion Types of surveys Response rates 1. Household Biases and errors 2. Telephone 3. Mail 4. Customer 5. Captive Group 6. Organisation 7. School Web-based of Information Technologies Types of Questions Open ended and pre-coded Measurement of Attitudes and opinions: Likert Scales Attitude Statements Semantic Differential Questionnaire Construction Validity/Reliability issues Pilot Testing Sample selection Coding the data Statistical Analysis. 4
Experimentation as a research method Controlled Laboratory Experiments or true experiments Non-experimental designs Simulations Non-experimental studies Observational Studies Case Studies Surveys Correlational Studies Types of Experimental Designs Completely randomised designs. CR-p Where p is the number of treatment levels of a single independent variable. Randomised block designs. RB-p Subjects relatively homogeneous w.r.t to the same nuisance variable are assigned to the same block. Completely Randomised Factorial Designs: CRF-pq Two or more treatments can be evaluated simultaneously. A Conceptual Model for Studying the Effectiveness of DSS DSS Characteristics eg features & user interface Characteristics of the Decision Maker Task Characteristics Effectiveness of a DSS Decision Perforamce Hypotheses pertaining to main effects of what-if analysis The first hypothesis proposes that the use of what-if analysis would result in better decision performance. As discussed in the Background section, what-if analysis helps to bring structure to a less-structured problem by incorporating the essential elements of the problem into some underlying models. Decision makers can thus evaluate different decision alternatives in a more systematic way. What-if analysis allows decision makers to explore the effects of changes of decision variables on the decision outcome which helps decision makers to gain deeper understanding of the problem. By receiving simulated outcome feedbacks from what-if analysis, decision makers can make necessary adjustments to their previous judgments about the problem, fine-tune their decisions until a best available solution is reached. On the other hand, decision makers without what-if analysis can only evaluate decision alternatives in a relatively intuitive way without any feedback of any sort. As a result, we postulate that what-if analysis should lead to a better decision. H1 : Decision makers using what-if analysis will show a better decision performance than decision makers without using what-if analysis. Decision madecision makers with what-if analysis will be have greater confidence in their decisions than decision makers without what-if analysis in making decisions. User Interface Main Effects Decision makers will perform better when using the interface with graphical input and graphical presentation (GI, GP) as compared to those using the interface with numeric input and graphical presentation (NI, GP) or numeric input and tabular presentation (NI, TP). 5
Interaction Effect Experimental Design There will be a significant interaction effect between the two main factors on production cost. User Interface Type with Input with Tabular Graphical Graphical What-if Without What-if Analysis 18 subjects 17 subjects 17 subjects Factor With What-if Analysis 18 subjects 17 subjects 19 subjects Experimental Task and tabular Production Planning task Computer-based decision aids developed to simulate the what-if analysis functionality and the different user interface types according to the six treatment groups. Results User Interface Type with Tabular Graphical Input Marginal Mean What-if Without What-if Analysis 49778.06 (sd=17204.78) 41477.42 (sd=13222.36) 36634.29 (sd=8108.97) 42975.79 Factor With What-if Analysis 34125.77 (sd=5695.54) 40572.80 (sd=14399.84) 31917.42 (sd=2457.53) 35378.38 Marginal Mean 41951.92 40981.34 34073.70 38953.63 Table 2 Mean Production Costs (with standard deviations) of the Six Treatment Groups 6
Graph of Interaction Effect on Production Cost Cumulative Average Production Cost ($) 50000.00 45000.00 40000.00 35000.00 30000.00 25000.00 20000.00 15000.00 10000.00 5000.00 0.00 NO WHAT-IF WHAT-IF Graphical Input with Tabular 7