Qualitative Methods in Empirical Studies of Software Engineering. by Carolyn B. Seaman

Size: px

Start display at page:

Download "Qualitative Methods in Empirical Studies of Software Engineering. by Carolyn B. Seaman"

Philip Waters
8 years ago
Views:

1 Qualitative Methods in Empirical Studies of Software Engineering by Carolyn B. Seaman

2 Overview topics of the paper qualitative methods data collection methods participant observation interviewing coding data analysis methods generation of theory confirmation of theory experimental design conclusions

interviewing coding data analysis methods generation

3 Topics several qualitative methods data collection analysis incorporation of qualitative methods with empirical studies of SW engineering combining qualitative methods with quantitative methods

methods with empirical studies of SW engineering

4 Qualitative Methods data presented as words and pictures origins in social sciences studying human behavior complex phenomena intersection between nontechnical and technical in SW engineering

human behavior complex phenomena intersection

5 Participant Observation data collection method research that involves social interaction between the researcher and informants in the milieu of the latter, during which data are systematically and unobtrusively collected methods: observing a SW developer continuously observing meetings of various types inspection meetings design meetings status meetings

systematically and unobtrusively collected methods: observing a SW developer

6 Participant Observation information gathered from meetings types of topics used terminology technical information group dynamics fly on the wall observer visible but not disruptive all participants aware beforehand confidential notes ensures normal behavior of subjects

the wall observer visible but not disruptive all participants aware

7 Participant Observation field notes contents for meetings place time participants discussions that took place (summaries) other events (in the meeting / impacted meeting) tone/mood observer's comments required detail level depends on objectives exploratory and open-ended study require a lot of details forms can be used in addition to field notes

meeting) tone/mood observer's comments required detail level depends on objectives

8 Participant Observation rater agreement excercises second observer used in few meetings both observers should agree with each others notes (coding) reveals discussions one observer may miss evaluation of validity and consistency of data data understandable to others recording of observations audio / video ensures accuracy of data

discussions one observer may miss evaluation of validity and consistency of data

9 Interviewing data collection method objectives collect historical data opinions and impressions about something identify used terminology used in combination with observations structured interview interviewer asks all questions very specific objectives for information type extreme: no qualitative information gained

combination with observations structured interview interviewer asks all

10 Interviewing unstructured interview questions asked also by interviewed person broadily defined topic open ended questions unforeseen types of information recorded extreme: interviewer gives only a topic semistructured interviews mixture of specific and open-ended questions unstructured may be too costly

information recorded extreme: interviewer gives only a topic semistructured

11 Interviewing general short explanation for the research issues in order to ensure subjects' participation too much may cause subjects to hold back data rambling subjects can be steered to right direction minimal answers can be avoided with proper questions or by pretending to be ignorant material data recorded in field notes and forms interview guide open-ended questions notes about direction: paths or list of topics

direction minimal answers can be avoided with proper questions or by pretending to be ignorant material

12 Interviewing note taking while interviewing can be difficult recording the interview refining conversational habits using a scribe no verbatim notes importance of issues may be expensive may be intimidating to the subject

conversational habits using a scribe no verbatim notes

13 Coding extracting values for quantitative variables from qualitative data enables quantitative or statistical analysis does not affect subjectivity or objectivity of the data may lose some information coding subjective data accurate interpretation of the words and meanings of the subjects minimizing data loss evaluating reliability of subjects or data sources selecting reference points for views

information coding subjective data accurate interpretation of the words and meanings of the

14 Generation of Theory (GoT) extract from a set of field notes a statement or proposition supported in multiple ways by the data proposition insightfully and richly describes the phenomenon for use as hypotheses

multiple ways by the data proposition insightfully

15 GoT: Constant Comparison Method coding the field notes attach codes to text pieces relevant to themes or ideas text pieces grouped into patterns examination of groupings codes discovery of themes and explanations preformed or postformed (modifiable) structured: categories, subcodes context of coded text should be preserved propositions strongly supported by the data based on field memos iterative process: propositions checked against new data

or postformed (modifiable) structured: categories, subcodes context of coded text should be preserved

16 GoT: Cross-Case Analysis data divided into cases two groups based on some attribute and examine similarities and differences pairs of cases compared to determine variations and similarities data divided based on source (interviews, observations etc.) helps to generate hypotheses e.g. list of propositions, each with a set of supporting and refuting evidence

similarities data divided based on source (interviews, observations etc.

17 Confirmation of Theory target: building up weight of evidence supports a certain proposition does not prove it ensuring validity of used methods representativeness of chosen people events researcher effects behavior of subjects losing objectivity

validity of used methods representativeness of chosen people

18 Confirmation of Theory triangulation confirming validity of conclusions different types of evidence replication classic: statistical testing of a hypothesis generated qualitatively anomalies in the data used to explain, shape, and support a proposition can be ignored in quantitative research

a hypothesis generated qualitatively anomalies in the data used to

19 Confirmation of Theory negative case analysis exhaustive search for evidence that might contradict a generated proposition revision of the proposition to cover the negative evidence rechecking the new proposition against data replication powerful but expensive tool qualitative: preserve conditions set forth in the theory being tested (data do no necessarily have to be collected or analyzed the same way)

proposition against data replication powerful but expensive tool qualitative: preserve conditions set

20 Confirmation of Theory member checking findings presented to subjects making them feel part of the process helping to understand results gaining support for final decisions useful when results affect the subjects' working methods

to understand results gaining support for final decisions

21 Experimental Design how combinations of qualitative and quantitative methods can be designed discussed through categories of empirical studies blocked subject-project study several different development projects are studied, with several different subjects reduced bias but increased cost qualitative data used to explain reasons between hypotheses and relationships that are based on statistical data

22 Experimental Design replicated project study multiple subjects working on same project isolates the effect of differences between subjects qualitative data illuminates quantitative data multiproject variation observe a subject on a project before some treatment is applied and after on a different project qualitative analysis used to form propositions quantitative analysis used to investigate them

23 Experimental Design single project study certain attributes of a single project are examined (possibly compared to organizational baseline) e.g. investigation of the types of errors made by developers with different backgrounds data collected qualitatively and coded to form quantitative variables statistical relationships found between quantitative variables are checked against the qualitative data

24 Conclusions qualitative data is richer than quantitative data use of qualitative methods increases amount of information diversity of information confidence in results

Efficient Models of Software Engineering

Efficient Models of Software Engineering IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 25, NO. 4, JULY/AUGUST 1999 1 Qualitative Methods in Empirical Studies of Software Engineering Carolyn B. Seaman, Member, IEEE AbstractÐWhile empirical studies