BANDWIDTH EFFICIENT VIRTUAL CLASSROOM

Transcription

1 BANDWIDTH EFFICIENT VIRTUAL CLASSROOM by MARCO VAN DER SCHYFF A thesis submitted to the Faculty of Engineering for partial fulfilment of the requirements for the degree of MAGISTER INGENERIAE in ELECTRICAL AND ELECTRONIC ENGINEERING at the UNIVERSITY OF JOHANNESBURG STUDY LEADER: PROF. H.C. FERREIRA May 2005

2

3 ACKNOWLEDGEMENTS First and foremost I would like to thank God for all the blessings and talents He has given me. I would like to thank Prof. H.C. Ferreira for the guidance and support during the duration of the project. Lastly I would like to thank my family and friends for their ideas, support and interest. i

4 ii

5 SYNOPSIS Virtual classrooms and online-learning are growing in popularity, but there are still some factors limiting the potential. Limited bandwidth for audio and video, the resultant transmission quality and limited feedback during virtual classroom sessions are some of the problems that need to be addressed. This thesis presents information on the design and implementation of various components of a virtual classroom system for researching methods of student feedback with a focus on bandwidth conservation. A facial feature technique is implemented and used within the system to determine the viability of using facial feature extraction to provide and prioritise feedback from students to teacher while conserving bandwidth. This allows a teacher to estimate the comprehension level of the class and individual students based on student images. A server determines which student terminal transmits its images to the teacher using data obtained from the facial feature extraction process. Feedback is improved as teachers adapt to class circumstances using experience gained in traditional classrooms. participation. Feedback is also less reliant on intentional student New page-turner, page suggestion and class activity components are presented as possible methods for improving student feedback. In particular, the effect of virtual classroom system parameters on feedback delays and bandwidth usage is investigated. In general, delays are increased as bandwidth requirements decrease. The system shows promise for future use in research on facial feature extraction, student feedback and bandwidth conservation in virtual classrooms. OPSOMMING Alhoewel die gebruik van afstandsonderrig toeneem, is daar steeds probleme wat aangespreek moet word voordat die potensiaal daarvan tot sy volle reg kan kom. Van die probleme wat ondervind word is o.a. onvoldoende bandwydte vir goeie kwaliteit video en klank, asook beperkte terugvoer vanaf studente tydens virtuele klaskamer sessies. Informasie oor die ontwerp en die implementering van verskeie komponente van n virtuele klaskamer vir navorsing oor tegnieke vir studente se terugvoer, met n fokus op die besparing van bandwydte word aangebied. n Gelaatstrek identifiserings tegniek is geimplementeer om studente terugvoer te hanteer en te prioritiseer met die oog op bandwydte besparing. Met gebruik van die metode, kan die onderwyser bepaal of n student en die klas as geheel die werk verstaan, gebaseer op beeld informasie van die studente. n Bediener bepaal watter beelde na die onderwyser gestuur word, gebaseer op informasie verkry deur die gelaatstrek identifiserings proses. Terugvoer vanaf studente word verbeter deurdat die onderwyser sy ondervinding van tradisionele klasgee kan gebruik om die lesing aan te pas afhangende van wat waargeneem word op die beelde. Terugvoer is ook nie meer beperk tot doelbewuste aksie deur studente nie. Nuwe bladsy-omslaner, bladsy-voorsteller en n klas aktiwiteitsgrafiek komponente word ook aangebied as moontlike metodes om terugvoer te verbeter. Die effek op vertraging van beelde en die gebruik van bandwydte deur verandering van stelsel parameters word ook ondersoek. In die algemeen verhoog vertragingstye, soos die beskikbare bandwydte afneem. Die sisteem toon potensiaal as n hulpmiddel vir toekomstige navorsing in gelaatstrek identifisering, studente terugvoer en bandwydte besparing in virtuele klasse. iii

6 iv

7 CONTENTS 1. INTRODUCTION ORGANISATION OF THE THESIS 2 2. DISTANCE EDUCATION AND VIRTUAL CLASSROOMS ADVANTAGES AND LIMITING FACTORS Advantages Limiting factors VIRTUAL CLASSROOMS Common features Existing products 9 3. IMAGE PROCESSING HISTOGRAM Equalisation Contrast stretch THRESHOLD NEIGHBOURHOOD AVERAGING (KERNEL - OPERATION) MORPHOLOGICAL OPERATORS Dilation Erosion Opening Closing EDGE DETECTION CONNECTED COMPONENT LABELLING FACE DETECTION AND FEATURE EXTRACTION FACE DETECTION Knowledge-based methods Feature-invariant approach Template matching methods 25 v

8 4.1.4 Appearance-based methods FACIAL FEATURE EXTRACTION Optical flow Motion models and model based feature extraction Feature point tracking marker based feature extraction Difference images DATABASES FOR USE WITH FACE RESEACRH CHALLENGES AND PERFORMANCE EVALUATION CODING SCHEMES MPEG-4 Facial Definition (FDPs) and Action Parameters (FAPs) Facial Action Coding System (FACS) CLASSIFICATION AND INTERPRETATION OF EXTRACTED FEATURES RESEARCH OVERVIEW PROPOSED VIRTUAL CLASSROOM FRAMEWORK MAIN SYSTEM COMPONENTS Student terminal Server Teacher terminal COMMUNICATION BETWEEN COMPONENTS USER INTERFACE Student terminal Server Teacher terminal LOG FILES USED TO GENERATE STATISTICS Facial feature extraction log file Transmission statistics log file Packet processing and transmission log file IMPLEMENTATION Technology used 48 vi

9 5.5.2 Student terminal, teacher terminal and virtual classroom server application software Audio and video transmission and reception Packet transmission, reception and processing Implementation notes STUDENT FEEDBACK VIDEO MONITORING Video capture Feature extraction Classify student activity importance Packets used for feedback on monitored video FEEDBACK GATHERED THROUGH USER INTERFACE Raise hand button Whiteboard Page-turner PRIORITISING AND PRESENTING STUDENT VIDEO FEEDBACK IMPLEMENTATION Capturing video Processing the frame Packets and components used for feedback Instructions for implementing another facial feature extraction algorithm Implementation notes EXPERIMENTS AND RESULTS EXPERIMENTAL SETUP FACIAL FEATURE EXTRACTION Processing times and processing jitter Results of the facial feature extraction process Importance estimates STUDENT AND TEACHER INTERACTION Student images Video and image quality 88 vii

10 7.4 BANDWIDTH AND DATA RATES Frame request interval Number of students in class and on display Student image size Video encoding Estimating behaviour for larger class sizes CONCLUSION REFERENCES DISTANCE EDUCATION PROTOCOLS, STANDARDS AND CODECS IMAGE PROCESSING FACIAL FEATURE EXTRACTION JAVA PUBLISHED ARTICLES 109 APPENDIX A : RESULTS 111 APPENDIX B : SYSTEM CONFIGURATION 139 APPENDIX C : APPENDIX D : COMMUNICATION PROTOCOLS, STANDARDS AND CONCEPTS 147 IMPLEMENTATION - CLASS AND METHOD DESCRIPTIONS 175 APPENDIX E : IMPLEMENTATION SOURCE CODE 177 viii

11 LIST OF FIGURES Figure 1: Histogram example. 11 Figure 2: Histogram equalisation example. 12 Figure 3: Contrast stretch example. 13 Figure 4: Threshold example. 14 Figure 5: Morphological operator examples. 16 Figure 6: Edge detection examples. 19 Figure 7: Trackability example for feature point tracking. [41] 29 Figure 8: Facial Definition Parameters. [31] [59] 33 Figure 9: Facial Animation Parameter Units. [31] 33 Figure 10: User interface of the proposed virtual classroom system. 39 Figure 11: Main components of the proposed virtual classroom system. 40 Figure 12: Student terminal components. 41 Figure 13: Server components. 42 Figure 14: Teacher terminal components. 42 Figure 15: Transmission control module. 44 Figure 16: Student terminal user interface. 45 Figure 17: Server user interface. 46 Figure 18: Teacher user interface. 47 Figure 19: Transmission statistics dialog box. 48 Figure 20: Packet interface. 51 Figure 21: Student video monitoring process. 54 Figure 22: Feature extraction process. 55 ix

12 Figure 23: Student video classification. 60 Figure 24: Display indicating the state of the Raise hand button on the specific student s terminal: Not raised (left) and raised (right). 63 Figure 25: Example of a whiteboard session with controls enabled. 63 Figure 26: Example Student Page Status Components showing various suggestions. 65 Figure 27: Student activity graph example. 67 Figure 28: Average frame processing times with standard deviation for various frame processing intervals. 74 Figure 29: Contribution of facial feature extraction phases to total processing time. 75 Figure 30: Percentage of video frames processed for which eyes where found or not. 76 Figure 31: Measured vs. specified frame processing intervals. 77 Figure 32: Average processing jitter for each interval. 77 Figure 33: Average processing times and intervals for various computers and videos. 78 Figure 34: Processing results (eyes correctly (1-3) / incorrectly (4-6) found. 79 Figure 35: Average of detected movement for the various video sequences. 81 Figure 36: Average intensity value of action units for the specified importance value. 81 Figure 37: Percentage of the video s frames classified as of a specific importance. 82 Figure 38: Examples of frames with the associated importance. 82 Figure 39: Average time after packet creation a specific VideoFramePacket action is performed for specified Frame Request Intervals. 83 Figure 40: Distribution of the time a VideoFramePacket is sent after creation. 84 Figure 41: Percentage of VideoFramePackets received of each video. 85 Figure 42: Average delay between VideoFramePackets when using various Frame Request Intervals on the Teacher Terminal. 85 Figure 43: Average time before a frame in a VideoFramePacket is displayed. 86 Figure 44: Average delay between receiving VideoFramePackets from the same source for various class sizes. 86 x

13 Figure 45: Average delay between VideoFramePackets from the same source for different class and display sizes. 87 Figure 46: Teacher terminal with different number of student on display. 87 Figure 47: Video frames at various resolutions using different encoding schemes. 88 Figure 48: Examples of H263 video frame errors. 88 Figure 49: Student images of various resolutions. 89 Figure 50: Average control and status packet data rate for various Frame Request Intervals. 90 Figure 51: Percentage VideoFramePackets sent / dropped per Frame Request Interval. 91 Figure 52: VideoFramePacket data rate for various Frame Request Intervals. 91 Figure 53: Data sent and received by the student terminals in 350 seconds. 92 Figure 54: VideoFramePackets received by the teacher terminal in 350 seconds. 92 Figure 55: Average receive data rate for various class sizes and number of students displayed. 93 Figure 56: Average send data rate for various class sizes and number of students displayed. 94 Figure 57: Data sent and received by the student terminal. 94 Figure 58: Average data rate for various student image resolutions and computers. 95 Figure 59: Data sent for different student image resolutions. 95 Figure 60: Average data rate for various video encoding schemes. 96 Figure 61: Data sent using various video encoding schemes. 96 Figure 62: Connection settings dialog box. 142 Figure 63: Video settings dialog box. 143 Figure 64: Audio setting dialog box. 143 Figure 65: Transmission control settings file example. 144 Figure 66: Student settings file example. 145 xi

14 Figure 67: Server settings file example. 145 Figure 68: Teacher settings file example. 145 Figure 69: A frame containing protocol headers and data [13]. 148 Figure 70: Internet Protocol Header. (Version 4) 148 Figure 71: Internet Protocol Header. (Version 6) 149 Figure 72: Transmission Control Protocol (TCP) Header. 150 Figure 73: User Datagram Protocol (UDP) Header. 151 Figure 74: Real Time Protocol (RTP) Header. 152 Figure 75: Real Time Control Protocol (RTCP) Header. 153 Figure 76: RTCP: Sender Packet Sender Information. 154 Figure 77: RTCP: Sender and Receiver Packet Reception Report Block. 154 Figure 78: RTCP: Source Description Item Packet Data Chunk. 155 Figure 79: Request-line example. 158 Figure 80: Response-line example. 158 Figure 81: Header field example. 159 Figure 82: H323 Zone. 160 Figure 83: H.323 Terminal. [11] 161 Figure 84: H323 protocols in relation to the (ISO) reference model [14] 162 Figure 85: H225 Header Structure. 163 Figure 86: MPEG-4 Scene and logical structure of media objects. 166 Figure 87: MPEG-4 data receiving process. 167 Figure 88: Description Definition Language Definition Scheme declaration. [22] 171 Figure 89: MPEG-7 description example. [21] 171 Figure 90: MPEG-7 description tools. [21] 172 xii

15 LIST OF TABLES Table 1: Virtual classroom products. 9 Table 2: Kernels used for edge detection. 20 Table 3: Facial feature extraction methods. [27] 26 Table 4: Databases used for image research. [28] [45] 30 Table 5: FACS Action Units. [33] 34 Table 6: Intensity scores used in FACS. [32] 35 Table 7: Computers used during the experiments. 71 Table 8: Videos used during the experiments. 71 Table 9: Descriptions of the experiments performed on the system. 72 Table 10: Processing times for the phases of the facial feature extraction (Eyes found). 76 Table 11: Processing times for phases of the facial feature extraction. 76 Table 12: Recommended parameter values with estimated bandwidth usage. 101 Table 13: Software Settings. 139 Table 14: ISO 7-layer Network Reference Model. 147 Table 15: Source Description Information Items. 155 Table 16: Bandwidth and coding rate of audio standards. 165 Table 17: Visual descriptors and description schemes. 172 Table 18: Audio descriptors and description schemes. 173 Table 19: Main Multimedia Description Schemes. 174 xiii

16 xiv

17 1. INTRODUCTION Virtual classrooms have the potential to deliver a very high quality of education, especially through the increased use of multimedia content, but currently many virtual classroom systems have very limited student and teacher interaction - an essential element, forming a cornerstone of traditional classrooms. In traditional classrooms the interaction is initiated automatically, as the teacher observes the class and detects a problem that requires student and teacher interaction. Most current virtual classroom systems provide very little student feedback during lectures. The feedback that does occur also depends on the willingness of the student to provide feedback, using mechanisms such as opinion polls or questionnaires. The visual clues deduced from facial expressions used in traditional classrooms to estimate a student s level of attention and comprehension is lost. A few recent systems do monitor the activity of the students and this is then displayed next to the name of the student, but this ignores the wealth of experience gained by teachers during traditional classroom sessions in estimating a student s understanding of the subject material. Some systems do provide live video of the students to address this issue, but these systems are very bandwidth intensive and can usually only accommodate a small number of students due to the increased bandwidth requirements for more students. The objective of the thesis is to present information on the design and implementation of various components of a virtual classroom system for researching methods of student feedback with a focus on bandwidth conservation. Emphasis is placed on facial feature extraction as a method to provide and prioritise feedback from various students. To this end a simple facial feature technique is implemented and used within the system to determine the viability of using facial feature extraction as a way of conserving bandwidth in a virtual classroom system. The thesis must also serve as a reference for others who wish to begin research in facial feature extraction. New facial feature extraction methods must be easily integrated into the system for those who would like to test how their algorithms will perform in the virtual classroom system. To achieve the objectives, the proposed system aims to improve the feedback from students to teachers by monitoring a student s activity (both intentional and unintentional). The feedback is sent to the server where the information is then summarised and used to determine a student s priority. Relevant information on each student and on the class as a whole is then sent to the teacher terminal at predefined intervals. Important video frames of 1

18 INTRODUCTION the students with the highest priorities are also sent to the teacher terminal. In this way the teacher should see the most important parts of the video from a student without increasing the strain on the network to the same extent necessary when sending live video from each student. Research into the interpretation of facial features is usually limited to basic emotions (sad, happy, angry etc.) and is normally done on posed faces under extremely controlled conditions with regards to lighting and background. Many of these systems also require extensive training of learning structures to tailor the performance to work efficiently on the specific person who will use the system. Facial feature extraction research for communication purposes usually estimate facial expressions from captured video. The estimates are then sent across the network where an animation is used to synthesize the actions of the original video. The main benefit of this method is a reduction in data sent across the network. This is especially useful to send video over the network at very low bandwidths. (see e.g. [34],[36],[37],[40], [57],[60]) It is also interesting to note that the authors of [61] suggested that relative long periods of no facial expression changes can be assumed for online environments. The amount of research in facial feature extraction that focuses on education is very rare. In [55] the authors present a tutorial system under development which uses facial feature extraction and analysis for tutoring a student, while a spatio-temporal approach to estimating a person s level of interest is presented in [54]. There does not appear to be any other research project which uses facial feature extraction techniques from multiple computers to control transmission and provide feedback between student and teacher terminals in a virtual classroom. 1.1 ORGANISATION OF THE THESIS The thesis starts with a discussion on distance education and virtual classrooms in chapter 2. The chapter also contains information on common features found in virtual classroom products as well as a list of companies providing virtual classroom software. Chapter 3 provides background information on a few image processing techniques used for the proposed virtual classroom system. These include morphological operators, edge detection and connected component labelling. Chapter 4 contains various face detection and feature extraction techniques. A list of databases used for face research, information on different facial coding schemes, feature extraction classification and interpretation, as well as a facial extraction research overview is also provided. 2

19 INTRODUCTION In Chapters 5 and 6 the proposed virtual classroom is discussed. Chapter 5 concentrates on the structure, configuration and use of the virtual classroom. The main components of the system as well as communication between these components and their implementation are also featured in the chapter. Chapter 6 looks at those components used with student feedback. This includes components for video monitoring, prioritising and presentation. The use of the user interface to provide feedback is also discussed. Details on the techniques used during the implementation conclude the chapter. Chapter 7 defines the experiments conducted on the system. The results of these experiments are then grouped into three categories: Facial feature extraction, student and teacher interaction as well as bandwidth and data rates. Conclusions, recommendations and suggestions for future research on student feedback in virtual classrooms are provided in Chapter 8. APPENDIX A contains detailed results from Chapter 7 while APPENDIX B contains details on configuring system properties. In APPENDIX C some networking protocols, standards and innovative communication techniques are discussed. APPENDIX D provides a detailed list of the Java classes (with their methods) written to create the virtual classroom software, while APPENDIX E contains the source code developed for the system (>22500 lines of code). Due to the large amount of work and results there in, parts of APPENDIX A as well as the entire APPENDIX D and APPENDIX E can be found on the enclosed CD-ROM disk. 3

20

21 2. DISTANCE EDUCATION AND VIRTUAL CLASSROOMS Distance education is a formalised teaching system specifically designed to facilitate communication between a teacher and learner, not located at the same location, through the use of technology. The technology used for this includes voice, video and computer-centred techniques such as audio and videocassettes, video conferencing, computer-assisted instruction (CAI) and computer-mediated education (CME). [1] Distance education can be synchronous, asynchronous or a combination of them Asynchronous systems store course content on the system to be viewed at the learners convenience and may provide facilities for the learners to communicate and interact, such as forums. One such a system is implemented by Skillsoft ( Synchronous systems use audio, video, slideshows, blackboards etc. to mimic the traditional classroom experience in real time. Some synchronous systems may even use chattechnology where all participants connected to the system discuss content by typing questions, comments and answers on a site where it can be viewed and responded on by any person connected to the system at the time. The number of institutions providing distance education is growing at a rapid rate. There are thousands of distance education courses available for prospective students in almost any category imaginable. Many of these courses are offered by traditional institutions of higher education, while others are distance only institutions located only on the Internet. The site encourages teachers to provide online classes using their technology. Registration on this site is free and a teacher may even be paid for traffic on the site. If the teacher wishes to make his classes public, the class is reviewed and on approval it is added to the list of available online classes. 2.1 ADVANTAGES AND LIMITING FACTORS ADVANTAGES a. ACCESSIBILITY Probably the most important advantage of distance education is its potential to bring quality education to more people. Learners will also have a greater variety of potential institutions for education to select from, since their selection will no longer be limited by their geographical location. 5

22 DISTANCE EDUCATION & VIRTUAL CLASSROOMS b. COST The cost of distance education is relatively low, compared to traditional ways of education, especially in the business environment. Traditionally, when training occurs either the trainer, trainees or both travel to the training venue. The costs are very high with this arrangement, as there are costs associated with travelling to the venue, hiring / maintaining the venue and even boarding and lodging costs in some cases. Transportation costs also tend to increase due to high fuel prices, while costs relating to technology tend to decrease. c. INCREASED STANDARDS OF EDUCATION The standard of education can also be increased since a leader in a specific field can give a lecture on his topic over the Internet. It would also enable a person in Africa, for instance, to study at a university in another country specialising in a specific field. Distance education through the Internet also makes it more convenient for lecturers to use graphics, videos, sound, websites and animations than traditional educational methods. This will aid greatly in the understanding of a subject and it will also encourage the learner to consult alternative sources of information. d. FLEXIBILITY Distance education, when implemented correctly, can bring a degree of flexibility to education which is very difficult to attain with conventional methods. The learner can control the pace, time and venue where learning will take place in asynchronous systems and many synchronous systems offer the option of providing the content at a later stage. This may result in a loss of interactivity, but it is better than completely missing a class, as would have been the case when traditional education methods are used. According to [2] this was one of the features his students found very valuable, but they still preferred the interactivity if at all feasible LIMITING FACTORS a. STUDENT-TEACHER INTERACTION One of the main criticisms of distance education is the lack of student-teacher interaction in most distance education systems. Most distance education systems are asynchronous, providing their users with course information, forums etc. A few systems do offer synchronous services such as video conferencing, but only from instructor to student. These systems usually have some interaction in the form of electronic hand raising, poll results and questions being displayed on the screen of the teacher. Some applications also allow the teacher to take control of applications or see snapshots of the learners screens. 6

23 DISTANCE EDUCATION & VIRTUAL CLASSROOMS It takes real effort from the teacher to make sure the interaction is maintained, especially if there is no video or only audio. In these circumstances (without body language, group report etc.) the teacher will have to adapt his / her style to ensure distance education is successful. [2]. A good example of this (given in [3]) is when a teacher refers and points to an equation on the blackboard distant students without video won t know to which equation he is referring. Various methods for retaining a student s interest and general tips for enhancing distance education were suggested in [2] to [7] b. COMPUTER ILLITERACY AND RESISTANCE TO CHANGE Computer illiteracy is one of the main stumbling blocks for utilising the full potential of distance education. Learners and teachers will need to have a basic knowledge of computers to use the software and the effectiveness of the distance education will probably be directly related to the teacher s knowledge of computers. With a better knowledge of computers the teacher will make use of the advanced features provided by computers and the Internet, thus potentially providing better education. Most current distance education systems, which utilise the Internet, are used by institutions of higher education or business where the level of computer literacy is high. The ideal system would allow the users to specify the level of their competency and thus the way and the amount of information available to them. In this way, the resistance to change due to the unknown could also be minimised. Resistance to change will most likely also come from the teachers, as they may have to change their teaching styles to try and overcome the problems currently associated with distance education. c. RECOGNITION OF QUALIFICATIONS A very interesting comment was made in [3] about the perceptions associated with an online-degree. Will an employer believe that a degree completed online is of the same standard as a traditional degree? d. COST The initial cost of implementing / acquiring a distance education system is still high. The students will also require facilities where the distance education system can be accessed. New high bandwidth systems are also becoming available, but the costs involved with these systems currently still keep them out of the reach of the general public. 7

24 DISTANCE EDUCATION & VIRTUAL CLASSROOMS e. BANDWIDTH LIMITATIONS AND QUALITY OF AUDIO AND VIDEO One of the biggest problems with distance education is the quality of real time audio and video transmitted over the Internet. Many of the systems use only audio together with the transmission of graphics to try and limit the amount of bandwidth required. Others prefer to only use text based chat programs for student-teacher interaction. 2.2 VIRTUAL CLASSROOMS In distance education, a virtual classroom is the use of video, audio and other technology to simulate the traditional class and learning environment as closely as possible. This section provides information on existing virtual classroom systems and the common features shared by these systems COMMON FEATURES Most virtual classroom systems incorporate the following features: Registration and student management: The systems allow multiple students to register for distance education. User activity is also tracked, allowing the teacher to get a better understanding of how the features available to the system are used. A detailed record of student attendance and assessments are also kept by the system. Support for a wide range of content: Files created using Microsoft PowerPoint TM, Excel TM etc. can be displayed and used in the virtual classroom s systems. Most systems also support presentations with animations. Video and audio files in common formats (AVI, MPEG etc.) may also be used as content. Web tour, application sharing and look over shoulder : All the students in a virtual classroom can be taken on a web tour and will follow all the links that a teacher follows. Students can also track all the teachers actions (and visa versa) when using the look over shoulder feature. Application sharing allows a student to control a program currently running on the teacher s computer and visa versa as required. Real-time interaction: Interaction occurs through the use of opinion polls, quizzes, tests, assignments, multiple choice questions, hand raising, anonymous feedback and a shared whiteboard for real-time example sharing. Some systems also allow student feedback with manually selectable mood indicators. Chat or breakout sessions: The chat feature allows discussions between teachers and students where bandwidth is at a premium. Inter-student discussions are also possible. When these discussions take place while the virtual classroom session is in progress, it is known as a breakout session. 8

25 DISTANCE EDUCATION & VIRTUAL CLASSROOMS Countdown timers and session clocks: Used to monitor breaks and provide teachers with methods to ensure they are still on schedule during the classroom session. Video and / or audio conferencing: The audio and / or video is sent from the teacher terminal to all students. A connection from student to teacher is usually only made when a student wants to ask a question. Recording of sessions: The virtual classroom session can be recorded to be reviewed at a later time. Class notes: Provision is made for the student to make notes during the class session. File transfer between teacher and student terminals also accommodate the distribution of handouts and notes EXISTING PRODUCTS Table 1 contains a list of companies with their virtual classroom software products. Website addresses where more information on the products can be found are also provided. Most of these companies provide the virtual classroom as part of software designed for all aspects of distance education. Table 1: Virtual classroom products. Company Product Website Blackboard Inc. Blackboard Application Suite - Virtual Classroom / Collaboration Tool Centra Software Centra Symposium Horizon Wimba Live Classroom IBM Lotus IBM Lotus Virtual Classroom Interwise Inc. ECP Connect Placeware Placeware Virtual Classroom (uses Microsoft Office Live Meeting) main.placeware.com WebEx Training Centre WebTrain WebTrain Online Education 9

26

27 3. IMAGE PROCESSING The image processing methods discussed in this chapter focus on image enhancement and segmentation to aid understanding of material in later chapters. For a complete overview on image processing techniques and their implementation please consult [24], [25] and / or [65]. 3.1 HISTOGRAM Bin count Bin (a) Image. Figure 1: Histogram example. (b) Histogram of image in (a). A histogram of an image provides the pixel amplitude distribution of the image. The number of bins used for each of the colour bands determines the amplitude range of the bin since the total amplitude range (dynamic range) is divided into the number of bins available. All pixels falling within the amplitude range of a specific bin are counted. The graph showing the bin count in relation to each bin is known as the histogram of the band or image. Maximum accuracy is obtained if the number of bins equal the number of possible amplitude values. In three-dimensional histograms each band is represented on a separate axis. The range of each bin in each band determines the cubic volume in which a colour must appear to be counted as part of the bin. A cumulative histogram displays the sum of the current bin count and all bins formed by counting smaller amplitude values. A lot of image information is contained within the histogram. Histograms are usually used for image manipulation, enhancement and segmentation. If an object is a different colour to its surroundings, the histogram can be used to isolate the colour values of the specific object of interest. Segmentation can now be performed using thresholds and the identified colour region. Image manipulation and enhancement rearranges the histogram of an image to place emphasis on certain aspects of an image or to obtain more detail. A few histogram based image manipulation techniques are discussed below. 11

28 IMAGE PROCESSING EQUALISATION Bin count Bin count Bin (a) Original image with histogram. Figure 2: Histogram equalisation example. Bin (b) Histogram equalised image of (a) with histogram. Histogram equalisation shifts the amplitude values of pixels to attempt to ensure that the number of pixels with each possible pixel value is equal. This can be accomplished by calculating a new pixel value for each pixel. The new pixel value k is calculated using J j= 0 N k = I T j M ( ) from [24], where J is the original intensity value of the pixel, N j is the number of pixels with an intensity value of j, I M is the maximum intensity value used and T is the number of pixels in the image. The cumulative histogram of the equalised image is a straight line. After equalisation minor variations are more easily seen due to increased detail in areas with a high brightness gradient. Pixels with the same amplitude values in the original image may now have different values. Different amplitude values in the original image may also be assigned the same amplitude values after equalisation. The resultant loss of information may also give a more uniform appearance to certain image areas. In Figure 2 we applied histogram equalisation to an image we took using a digital camera. From the histogram it can be clearly seen that the pixel value distribution is more uniform after equalisation. The Java Advanced Imaging (JAI) package was used to create the histogram equalised image. For an example of the code used please consult [65]. 12

29 IMAGE PROCESSING CONTRAST STRETCH Bin count Bin count Bin (a) Original image with histogram. Figure 3: Contrast stretch example. Bin (b) Contrast stretched image of (a) with histogram. Contrast stretching is applied to images to increase the difference between the maximum and minimum amplitude values by multiplying each pixel by a constant value and then adding an offset. Following [65] the constant and offset values, c and x, for RGB colour space is given by 255 c =, ( ) m s 255 m x =, and ( ) s m where m and s are the minimum and maximum amplitude values for the region to be contrast stretched. The pixel value after applying contrast stretching is given by k = z c + x ( ) where z is the original pixel value. Contrast stretching is useful to increase the visibility of structure in an image. By increasing the visibility of structure, the efficiency and accuracy of image segmentation and processing can also be increased. In Figure 3 we applied the contrast stretching method on an image taken using a digital camera. From the histogram it can be clearly seen that the image contains more values at the extreme ends of the histogram after contrast stretching. 13

30 IMAGE PROCESSING 3.2 THRESHOLD Bin (a) Original image. (b) Histogram of original image. Bin count Bin count Bin (c) Threshold applied to image (a). (d) Histogram of threshold image in (c). Figure 4: Threshold example. In image processing thresholds are mainly used for image segmentation. A threshold is the pixel value used to segment an image into different regions based on the value of each pixel. A pixel value below the threshold is said to belong to the one region (background), while pixels with values above the threshold belong to the other region (foreground). Pixel values corresponding to dips on an image histogram (see section 3.1) are often used as thresholds for segmentation and may or may not yield good separation results depending on the image. Other image processing methods are usually applied before the threshold operation to improve segmentation results. Thresholds may either be chosen manually by a skilled operator or be generated automatically using the image histogram and other image information. Automatically chosen thresholds are known as adaptive thresholds. In Figure 4 we applied a threshold at a pixel value of 150 to the greyscale image. As a result of the operation, all pixels in the original image with a value above 150 are given a value of 255 (white) in the threshold image. All other pixels of the threshold image have the same value as in the original image. Two main approaches used for adaptive thresholds discussed in [25] are the Chow and Kaneko approach and local thresholding. The Chow and Kaneko method uses the interpolation of thresholds calculated from the histograms of overlapping sub images to determine a threshold for each image. In the local thresholding approach the pixel s threshold is determined using a statistical approach to examine the values of neighbouring pixels. 14

31 IMAGE PROCESSING Multiple thresholds may be applied to the results of various processing techniques on the same image or to different images of the same object. If different images of the same object is used, one may be a normal image while the other may be taken using infra-red etc. Combining the results / different images using logic operators may lead to improved segmentation as more characteristics of the original image is used. 3.3 NEIGHBOURHOOD AVERAGING (KERNEL - OPERATION) Neighbourhood averaging, also known as a kernel-operation, forms the base for many image processing techniques. A kernel is a matrix containing weights. The centre of the matrix is aligned with the image pixel for which the value must be determined. The value of the pixel is determined by the sum of the product of each kernel weight with the correspondingly aligned pixel divided by the sum of all the kernel weights. The process is applied to all the pixels of the image. At the boundaries of the image, when aligning the centre of the kernel to a boundary pixel, all the weights of the kernel can not be aligned to a corresponding pixel. For this reason a few strategies can be adopted. One option is to leave all the pixels where the kernel can not be aligned, unchanged. Another option sees the image with its corners connected (wrap around) to form a ball. The first row of pixels is connected to the last row and the leftmost column is connected to the rightmost column. In this way the weights of the pixel can always be aligned to image pixels. As an example consider the 3x3 kernel (3.3.1) applied to pixel 2 in row 2 of the picture represented by (3.3.2) 1 The new pixel value is calculated as: (1x1 + 2x1 + 1x2 + 3x1 + 1x2 + 3x1 + 1x2 + 2x1 + 1x1 = 18) / ( =15) = 1.2. The result of applying the kernel to all the pixels utilising the wrap around border strategy is: (3.3.3)

32 IMAGE PROCESSING 3.4 MORPHOLOGICAL OPERATORS Operations based on dilation, erosion and combinations of these are collectively known as morphological operations. Morphological operations are mostly performed on binary images, but they may also be applied to greyscale images (a) Original image. (b) Kernel. (c) Dilation. (d) Erosion. (e) Closing. (f) Opening. Figure 5: Morphological operator examples. Foreground pixels are assumed to be white while background images are assumed to be black. Dilation, erosion opening and closing morphological operators are discussed in more detail in the following sections. Information on other morphological operations e.g. the hit and miss transform, thinning, thickening and the skeletonisation / medial axis transform can be found in [25]. 16

33 IMAGE PROCESSING DILATION Dilation is the operation of adding foreground pixels to an image region using a structuring element (kernel), causing the region to grow and holes within the regions to shrink. The kernel is superimposed on each pixel of the image with the key element of the kernel aligned with the pixel under consideration. If any one of the foreground kernel elements aligns with a foreground pixel, the pixel under consideration is set to a foreground pixel. If none of the foreground elements align, the pixel value is left unchanged. The centre of the kernel is usually used as the key element. Unsymmetrical kernel elements or kernels with a non-centre key element may also be used to add directionality to the dilation operation. The dilation operation may change the shape of the region it is performed on. For example: A circle will tend to converge towards a square with each dilation operation. A square will retain its shape after dilation operations. Greyscale dilation using a disk shaped kernel expands the light regions of an image and removes / shrinks dark regions. This usually results in an increase in image brightness. The effect is most pronounced at regions with rapid intensity changes. Dilation has many uses and is usually used in conjunction with logic operators. Pepper-noise (dark spot) removal, edge-detection and region filling are just some of the specialised uses of dilation. The duel of dilation is erosion. Applying dilation to the original binary image is equivalent to applying erosion on the inverse of the original image EROSION Erosion is the operation of removing foreground pixels from an image region using a structuring element (kernel), causing the region to shrink and holes within the regions to grow. Erosion is the duel of dilation and it uses a similar procedure to perform the operation. The kernel is superimposed on each pixel of the image with the key element of the kernel aligned with the pixel under consideration. If every foreground kernel element aligns with a foreground pixel, the pixel under consideration is left unchanged; otherwise it is changed to a background pixel. Directionality is determined by the symmetry of the kernel and the position of the key element. The shape of a region may change after erosion operations. For example, applying a erosion operation to a circle repetitively, results in a diamond shaped region. The diamond shape is retained on subsequent erosion operations. Greyscale erosion using a disk shaped kernel expands the dark regions of an image and removes / shrinks light regions. The effect is most pronounced at regions with rapid intensity changes. 17

34 IMAGE PROCESSING Erosion has many uses and is usually used in conjunction with logic operators. Salt-noise (bright spot) removal, edge-detection and region separation are just some of the specialised uses of erosion OPENING The opening operation uses the same kernel to perform an erosion operation followed by a dilation operation. The opening operation keeps foreground regions similar to the kernel used or which can contain the kernel, while removing other foreground pixels. A more pronounced effect may be obtained by repeating the erosion operation multiple times followed by the dilation operation for the same number of times. Greyscale opening will reduce the brightness of regions smaller than the structuring element, while leaving larger regions unchanged. This also removes small texture fluctuations. Opening may be useful for segmentation as it only keeps regions similar to the structuring element CLOSING The closing operation uses the same kernel to perform a dilation operation followed by an erosion operation. The closing operation keeps background regions similar to the kernel used or which can contain the kernel, while changing other background pixels to foreground pixels. Greyscale closing will reduce the darkness of regions smaller than the structuring element while leaving larger regions unchanged. Once closing has been performed further closings using the same kernel will have no effect, this is known as idempotence. 3.5 EDGE DETECTION An edge can be defined as a rapid change in pixel intensity and usually corresponds to boundaries of objects in the image. In theory the edges of an image can be extracted by implementing a high pass filter in the frequency or spatial domain. The spatial domain is usually used because of faster computation and better results. Edge detection in the spatial domain is achieved by convolving an image with a kernel. Edges can be determined from the 1 st or 2 nd derivatives of the image. The maximum of the 1 st derivate or the zero-crossing point of the 2 nd derivative corresponds to potential object edges. Commonly used methods for determining the 1 st derivative of the image are gradient edge detection and Prewitt compass edge detection. When using the Prewitt compass edge detection method each pixel in the final edge detected image is determined by evaluating the results when the image is convoluted with a set of kernels. Each kernel is sensitive to a different edge direction. The maximum convolution 18

35 IMAGE PROCESSING result for each pixel is used in the final edge detected image. Prewitt, Sobel, Kirsch and Robinson s kernels may be used. (See [24], [25], [65] and Table 2) The gradient magnitude edge detection method uses two kernels, one for detecting edges in the x direction and another for the y direction. The image is convolved with the two kernels and the pixel values of the resulting images (represented by Gx(pixel) and Gy(pixel)) is used in 2 2 = Gx( pixel) Gy( ) (3.5.1) G ( pixel) + pixel to determine the value of each pixel in the gradient magnitude image. A threshold is applied to this image to obtain the edge detected image. Common edge detection kernels used with the gradient edge detection method are the Sobel, Roberts Cross, Prewitt and Frei & Chen edge enhancement kernels. Information on other methods of edge detection e.g. the Canny operator, Marr edge detector, Laplacian etc. as well as more information on the kernels used can be found in [24], [25] and / or [65]. Figure 6 shows a few edge detection examples. The example images were created using the gradient magnitude method and the Frei & Chen kernels. A threshold value of 100 was used to create the binary images from the gradient magnitude image. (a) Original images. Figure 6: Edge detection examples. (b) Edge detected images. 19

36 IMAGE PROCESSING A few kernels used with edge detection are shown in Table 2. The table was created using the information on the kernels in [65]. Table 2: Kernels used for edge detection. Name Horizontal Vertical Comments Sobel Creates an omni directional outline of objects in the image. Highlights constant brightness regions. Roberts Cross Creates a fairly-coarse directional outline of objects in the image Highlights changing brightness regions. Constant brightness regions become black. Prewitt Creates a directional outline of objects in the image by extracting the north, northeast, east, southeast, south, southwest, west, or northwest edges. Highlights changing brightness regions. Constant brightness regions become black. Frei & Chen The configuration of relative pixel values is taken into account, independent of the brightness magnitude, with higher sensitivity than other edge detection kernels. 3.6 CONNECTED COMPONENT LABELLING Connected component analysis and region labelling uses the concept of pixel connectivity. Pixel connectivity defines the relationship between pixels. The neighbourhood of a pixel are those pixels around the current pixel. A neighbourhood can consist of 4 or 8 pixels adjacent to each other with values selected from the same value set. The set of coordinates that form a 4- or 8-neighbourhood for a pixel with coordinates ( x, y) are given by {( x 1, y),( x, y + 1),( x + 1, y),( x, 1) } N 4( x, y) = y and (3.6.1) 20