Using E-Observation to Conduct Qualitative Research Online- A Research Note Xin Liang (Ph.D) Assistant Professor Department of Educational Foundations and Leadership University of Akron Akron OH 44325-4201 Phone: 330-972-6921 Fax: 330650-4254 e-mail: liang@uakron.edu
Abstract Information and communication technologies have not only changed the way how teaching and learning take place, but also have the potential to transform, or at least enhance, many other facets of the research process, including the initial acquisition of data (Mann & Stewart, 2000). This paper builds on the author s recent experience of conducting e-observation to investigate classroom assessment as a pedagogical process in web-based instructional environment. Paying particular attention to the principal characteristics of e-observation, the paper identifies the possible strength and weakness of the e-observation. It aims to provide useful practical insights for researchers who might consider conducting the same type of research. Key Words e-observation, online, computer-mediated communication. Introduction As a result of the increased interest in research of online learning environments, there has been a subsequent increase in the interest and use of qualitative methods to gain a deeper understanding of online learning environments. A large number of studies have been published to examine, qualitatively, the perceptions, experiences, and instructional practice in the online environments. However, there has not yet been adequate exploration of the use of qualitative methods of online learning environment investigations (Bianco, & Carr- Chellman, 2002). In particular, the discussion of the electronic field or electronic means as one of the major data collection method was generally omitted (Bianco, & Carr- Chellman, 2002). As one of the key data collection methods, observation online or e-observation as we term it here, should be given more 2
attention in order to understand the social meanings, which are constitutive of and reflected in human behavior in online environment. For some researchers, the current available text-based computer- mediated communication (CMC) is not an appropriate method for research, which seeks to observe the real world (Mann & Stewart, 2000). However, recent studies, which focus on virtual communities, begin to challenge the basis of terms such as observation and natural context as used in traditional research (Mann & Stewart, 2000). The argument is represented by questions such as, 1) Is electronic space a natural field? 2) How is it possible to observe online classroom, as it really is just a cyber space? 3) Can e-observation be a viable, and legitimate data collection method to gain an understanding to teaching and learning in virtual field? This paper attempted to explore the answers to such questions by reflecting the author s own experience of obtaining online observation data to investigate instructors perceptions on online classroom assessment. In Spring 2003, the author received a faculty research grant to study online classroom assessment in a Midwestern university housing approximately 22,000 students. The purpose of the project was to investigate how to implement classroom assessment effectively to monitor and promote student performance in web-based classes (Webct). Webct was a prevalent instructional delivery method in this university. The operation of the research project was two-folded. The first part of the study was to conduct a grounded theory research to generate a process theory. By examining the dynamics of online classroom teaching and learning, the researcher hoped to develop a conceptual framework to define what effective classroom assessment practice was. Based on the theoretical framework grounded from the participants online teaching and learning 3
experience, the second part of the study was to construct a survey instrument for both students and instructors to identify the most effective classroom assessment techniques for online teaching and learning. At present, phase one of the project- conducting a grounded theory study to generate a conceptual framework for online classroom assessment has been finished. The research outcome was reported in a separate article (liang, 2004). In this paper, the major discussion will be focused on the online data collection method. In this study, web-based instruction referred to a course that was taught over 50 % of its content via webct. After sending out consent forms, ten instructors agreed to participate the study. These ten instructors taught 16 webct classes as five of the instructors taught more than one classes online. Webct instruction is the only on-line teaching software adopted by this institution at present. The webct classes included in this study were both at undergraduate and graduate levels in the College of Education. Among the ten instructors, four were males and six were females, holding the rank of assistant, or associate professor. Included also were 216 students in the webct classes, whose classroom participation, discussion, and assignment were observed and recorded. The three major qualitative data collection methods, observation, interview, and documentation were all adopted in the grounded theory study. E-observation as a key data collection method in this study played a vital role. We would never have understood the phenomena without personally experiencing it. Observation is essential for a researcher to obtain detailed descriptive information about what the setting is, what has occurred in the setting, who are the participants, and how people react to what has 4
occurred. Because the focus of the study was to understand how classroom assessment was practiced in webct classrooms, it became crucial to get into the field to obtain the first-hand experience, views, and actions of the instructors and learners in the online environment. The direct approach to accomplish the research goal was to watch what they do and listen to what they say in the field- webct classrooms. As a result, the e-observation data served as a baseline to understand what was going on in the cyber field, followed by e-interview and documentation. Two observers, the author and a doctoral student started the e- observation in Spring 2003, and continued for 9 months. Getting into the field, the two researchers wanted to capture how assessment as an instructional process was incorporated and operated in webct classrooms. Our e-observation foci entailed 1) general instructional procedures of the webct classrooms, 2) classroom interactions between learners and learners, between instructors and learners, 3) classroom assessment activities, 4) assessment oriented performance. The e-observation was recorded in forms of field notes, journals, and memos for further analysis. Particular attention was given to observe the dynamics of classroom interactions where there was a cyclical process of content, input, procedures and product to inform and direct teaching and learning. The two observers were present to conduct the e-observation independently for the same webct class or classes to control the reliability and accuracy. Discussions were held routinely between the two observers to exchange field notes, memos, and key observation findings. The discrepancy between the two observers was discussed and resolved by reobservation, reviewing the field notes, and by later comparison with documentation data and e-interview transcriptions. 5
After the e-observation of webct classrooms was finished, data was sorted, categorized, and compared. Ten instructors were interviewed via e-mail with both structured and open-ended questions. The purpose of online interview was to obtain the instructors reflections and experiences of their own teaching. Besides observation and interview data, other data resources, such as syllabi, class notes, individual projects, group projects, quizzes, tests and exams were documented, categorized, compared and analyzed along with the observation and interview data to develop themes and build a chain of evidence. The documentation started with the observation. As the observation moved along, data resources began to accumulate, diverge by topics, categories and themes. The different data resources were also merged and compared together as a way to triangulate the truthfulness of the research outcome. As the study focused on understanding the process of implementing formative assessment in relation to the unique features of online instructional environment, it seemed appropriate to use qualitative research method. It was decided to take a Straussian approach to grounded theory, in which a process theory was generated to explain an educational process of events, activities, actions, and interactions that occurred over time (Creswell, 2002, p. 441). It was believed that the process theory discovered during data collection fit the situation being researched and will work when put into use (Creswell, 2002). In retrospect of the study, e-observation played an important role in helping me as a researcher to understand the field webct classes and the instructors and learners. Some of the features were especially unique in contrast to the traditional f2f field observation. The aim of this paper is to discuss the potential of e-observation as a research tool, describing some of its characteristics, advantages and limitations. As e- 6
observation shares many strengths and weakness of the f2f observation when compared with other research methods, I ll concentrate on the ways in which e-observation differs from the more familiar method of observation based on the nine-month observation experience happened in our own research field- the 16 webct classrooms in a Midwestern university. Transforming the observation: linguistic behavior When compared with the f2f e-interview, Bampton & Cowton (2002) pointed out that time and pace were the two major displacements of human experience from e- interview. In relation to time, he referred to the asynchronous interaction between interviewer and interviewee, with pauses of varying lengths between bursts of episodes. In terms of space, the relationship took place at a distance through the medium of electronic, screen-based text. E-observation shared the same characteristics as e-interview except that without a concrete setting to locate, e-observation required even more displacement from the conventional f2f observation. Frankfort-Nachmias and Nachnias, (1996) identified four areas of observational interest in f2f observation, 1) nonverbal behavior (body language and particularly facial expressions), 2) spatial behavior (issues of physical closeness and distance), 3) linguistic behavior (both what is said and how), and 4) extralinguistic behavior (rate of speaking, loudness, tendency to interrupt, pronunciation). Using current available technology only one of these four options, that is linguistic behavior, seems to be clear focus for observation online, or e-observation. To observe participants linguistic behavior in artificial settings without visual or audio presence posed some challenges, as well as possibilities. These unique features of e- 7
observation departed from f2f observation have some significant ramifications for qualitative data collection method. The observation of linguistic behavior via Internet has several consequences. The availability of large amount of texts and images depicting the actions, feelings and interactions of the participant in the online community furthered the deeper aims of qualitative research. First of all, depending on writing as the primary media for communication and interaction of the social cultural events in virtual community has made the collection of large amount of information relatively easier, and more efficient. In the nine months e-observation of the interactions of ten instructors with over 200 students in an online learning community, I was able to collect and analyze over 1,000 messages, 100 presentation images, 50 student projects. The time required for taking field notes, or collecting audio-visual materials in a f2f observation to obtain data of this magnate would be almost impossible in such a short period of time. In a f2f observation, both descriptive and reflective field notes needed to be taken to capture the dynamics of an observed central phenomena. While conducting e-observation, there is no need to take descriptive field notes as text description already existed. This allowed me to completely concentrate on watching, instead of watching, while writing down what I saw. Secondly, the lived experience, insights, emotions were written by the participants themselves, instead of filtered through by the researcher, which happened in the conventional filed observations. Although researchers, and participants were not visually or auditorially present, researchers may also observe non-verbal and extra-linguistic behavior exhibited in emotions, paralinguistic and non-linguistic cues. I noticed in my e-observation that expressives, such as multiple punctuations, color, capital letter, multiple vowels, color 8
were used to represent intonations, gestures, surprises, exaggerations, and other nonlinguistic cues in webct classes. The use of other expressive symbols other than words helped a person to deliver not only the semantic meaning of the message, but the gesture, intonation, and emotions to the students. In e-observation, such electronic paralanguage (Mann &Stewart, 2000) was a commonly used communication by both instructors and students. The following paragraph was an example of how paralanguage was used to transmit both semantic meaning and non-linguistic cues. The only way I can evaluate the quality of my instruction and the clarity of the assignment is by looking at what YOU produce!!!! For this reason, I do not grade assignments that are submitted *early*. It is really to your BENIFIT for me to wait until I have at least half of the papers submitted before I start grading. And to be "fair" I grade all the papers first before returning any of them. Soooooo...about the only advantage to submitting your paper early is the "peace of mind -(Yeh! I m done!!!). Ethnomethodological approaches were broadly concerned with how people construct their own social situation. Online qualitative researchers with this approach can observe the natural conversations of various kinds of project groups, synchronous group discussions. The focus was on ordinary, mundane, naturally occurring talk to reveal the way meaning is accomplished by everyone involved. In cyberspace the physical presence of both researchers and participants is concealed. On one side, researcher s subculturally distinctive features, such as clothing, physical appearance which has been so useful to obtain negotiated access to the research site in the f2f research became redundant. On the other side, researchers must establish other insider status- mostly through writing to gain such access. Without an outsider physically present in the social situation in online community, the participants were most often unaware that their performance was being watched, even though they were informed so. Having someone 9
physically present in a classroom was different from knowing someone was watching your class online. Thus factors such as faking for good or bad, or the uneasiness of having someone sitting in the classroom to some extent were largely diminished in e- observation. Even though instructors and learners may not necessarily fake certain behavior, there were situations they would feel uncomfortable to discuss in front of a person they do not know well. For example, addressing a common behaviors, such as lack of effort in group discussion, or lack of preparation for the assigned reading materials, the instructor may wanted to wait to bring them up when everyone felt more comfortable with the instructor, and with one another. It was less likely that the instructor wanted to do so when they had a researcher sitting at the back of the f2f classroom. While in online classes, to bring such issue would not be so inconvenient. In conventional observational research there are also some practical difficulties in recording and writing about a setting at the same time as observing it, and time lags in recording observations can allow inaccuracies and sometimes distortion. This is clearly not a problem online. The researchers were able to watch more natural interactions without intervening in any way by lurking unseen in virtual field. On the flip side, to be heavily dependent upon dense deep readings of cybertext, qualitative analysis of e-observation might lack some of the analytic breadth, and authenticity that are possible when f2f conversations are observed. First of all, the context of the actual experiences of individual participants was at their own keyboards in their rooms through a distance mediated by computers. There is a possibility that some of the initial thoughts, feeling, emotions, and interactions get lost because the process of writing filters them through. The time taking to write a message expressing one self most 10
often results in a more crafted response than a simultaneous reaction. Although other media devices, such as visual and audio techniques, paralinguistic, and nonlinguistic cues are introduced, the richness of body language, facial expressions, emotional nuances are hard to be depicted in writing. Even if these devices were available, people usually do not want to spend much time taking care of the trivial when they are engaged in cyber communication. Some social aspects of human interaction in a conversation cannot be fully observed in a cyber community. As in f2f interaction, there is a dynamic dimension, such as people wait for their turns, or put up their hands to speak, even silence. In online conversation, these dynamic were mostly unobservable, and yet the social dimension is very important to give researcher clues in shaping the mood of interaction. This is big lost for e-observation as a data collection method to understand a social situation. Conclusion The ninth-month in depth observation of webct classes enriched our understanding to a new research arena. We were very excited to explore the unique aspects of e-observation departed from traditional f2f observation. Gotved (2000) compares movements in urban societies with movements in cyberspace. She states that the difference between the city and the Internet is based on the possibility of physical bodily representations in the city and the lack of this in space. Based on the difference, I compared the e-observation with the classic f2f qualitative observation method. The principal advantages and disadvantages of the e-observations are that: 11
It offers significant savings in terms of time and financial resources, particularly in relation to the flexibility of time and place to observe, and the elimination of recording, transcribing large amount of text material; It opens possibilities for observing multiple research sites and participants that would ordinarily lie beyond the geographical or social reach of the researcher; The degree of intrusion, which is usually involved in having a stranger watching everything happening in private settings was mostly eliminated, and In some circumstances, it might be more successful in accessing authentic data, as the experience, attitudes, and emotions were written directly by the participants, instead of being filtered by a second person; but The absence of physical interaction reduced the richness and dynamics to record the complexity of social behaviors in the cyber world; and The writing skills and linguistic behaviors of the participants largely predominated the quality of the observation data. Negotiating access, acceptance and trust to the research site is dependent upon the researchers writing competency. Other subcultural capitals such as a person s accent, clothing, and appearance necessary in f2f observation became irrelevant in conducting e- observation in a cyber community. In order to capture the dynamic of online classroom, and to understand the teaching and learning experience in online environment, as a researcher I had no other choices but to integrate technology into the traditional qualitative research method. For my webct classroom assessment research, online research in general, e-observation in particular provided me opportunities to access webct classrooms to learn how online courses were 12
taught and learned in the web. I can t imagine how one could have accomplished this task without computer-mediated communication, and e-observation. However, I am also aware that the absence of physical presence in cyberspace might mean some absence of truth as social reality is constructed by interactions both by physical and other means. The comparison of advantages and disadvantages of e-observation as a data collection tool convinced me that observation online is far from replacing f2f observation in the natural world. However, I had no doubt that we can go about observing the online classroom in responding to the specific research questions to accomplish a specific research task. As one of the three major data collection methods of qualitative inquiry, e- observation should be used as a complementary method to other method, such as interview, and documentations both f2f and online. Bianco & Carr-Chellman (2000) cautioned that e-observation is perhaps one of the most difficult things to handle in online learning environment inquiry, because we cannot even define if electronic space is the actual classroom. With the research site undefined for e-observation, many questions remained unanswered. What is the role of the researcher in online environment? How do we go about really observing the online learning? What sort of observational techniques should we use? Like many other data collection method, e-observation may not be the most appropriate research method, or even a suitable one in all circumstances, but as the use of computer-mediated communication becomes more widespread and more firmly embedded into social processes, we would see e-observation takes its place as a valued option in the qualitative researcher s toolbox. I hope this paper has provided some useful insights for researchers who might conduct similar research in the future. 13
Reference Bampton, R., & Cowton, J. C. (2002). The E-interview. Forum: Qualitative social research. 3(2), May 2002. Bianco, B. M., & Carr- Chellman, A. A. (2002). Exploring qualitative methodologies in online learning environments. The quarterly review of distance education. 3(3), 2002.pp. 251-260. Creswell, W. J. (2002). Educational Research- Planning, Conducting, and Evaluating Quantitative and Qualitative Research. Merrill Prentice Hall, Upper Saddle River, New Jersey. Frankfort-Nachmias, C., & Nachmias, D. (1996). Research Methods in the Social sciences. London and Sydney: St Martin s Press Gotved, Stine (2000). Cybersociologi - den samme på en anden måde. Ph.d. thesis, Sociologisk Institut, Københavns Universitet. Mann, C. & Stewart, F. (2000). Internet communication and qualitative research: A Handbook for research online. London: Sage. 14