Learning Motivation Evaluation for a Computer-based Instructional Tutorial Using ARCS Model of Motivational Design

Transcription

1 Learning Motivation Evaluation for a Computer-based Instructional Tutorial Using ARCS Model of Motivational Design David Wenhao Huang 1, Heidi Diefes-Dux 2, P.K. Imbrie 3, Brian Daku 4 and James G. Kallimani 5 Abstract - This pilot study demonstrates the feasibility of utilizing instructional theories, specifically on learning motivation, to evaluate a computer-based tutorial for the purpose of proposing effective instructional interventions. Keller's ARCS Model of Motivational Design provides the conceptual framework to address motivational issues while developing instruction. The ARCS Model has four dimensions: Attention, Relevance, Confidence, and Satisfaction. Keller's Instructional Material Motivation Survey measures student motivation along the ARCS dimensions and was modified for this study to evaluate students' motivation while using a computer-based tutorial. This study was conducted in an engineering problem-solving and computer tools course for first-year students. The studied computer-based instructional tutorial, M-Tutor, is designed for learning MATLAB syntactical structures. A pre-post evaluation research design was employed. A coding system was developed to categorize qualitative responses into corresponding instructional components. Qualitative and quantitative data were triangulated for instructional intervention development. Index Terms - ARCS Model, computer-based instruction, motivation, programming instruction INTRODUCTION MATLAB in Freshman Engineering Courses MATLAB is a computational software package that integrates mathematical computing, visualization, and a powerful computer programming language to provide a flexible environment for technical computing [1]. MATLAB is used professionally in a number of fields, including engineering, and is taught across the country in university engineering, science, and mathematics courses [2-5]. The challenge to using MATLAB in a first-year engineering problem solving course is to bring the computer literacy of an entire class up to a workable level, such that students can effectively use the computer tool to solve contextual problems. Explicitly teaching the computer tool can lead to considerable trouble with bored and disinterested students if lecture time is spent on low-level definitions, syntax, and point-and-click operations. Yet students need to develop computer tool skills before instructors delve into problem solving situations that use the computer tool. In addition, students can only gain an understanding of the capabilities and limitations of syntax and operations by constructively exploring a computer tool. However, instructors find that text-based materials alone do not actively lead the student in the discovery of the functionality of syntax and operations. Computer-based instructional tools for technical content (e.g. programming syntax) are emerging but require evaluation for effective implementation. Computer-Based Tutorial for MATLAB Syntax M-Tutor was developed at the University of Saskatchewan for the purpose of providing a "tutorial for independent study by learners who had little or no exposure to MATLAB" [6,7]. The goals of the tutorial were to: engage students in active learning, allow students to proceed at their own pace, permit learning off-campus, and make more effective use of faculty teaching time and student study time. M-Tutor is divided into eight sections: Getting Started, MATLAB Variables, Scalar Math, Vector Math, Vectors and Basic Plotting, Relational and Logical Math, Writing Basic MATLAB Programs, and Matrix Math. One of the unique features of this tutorial is its selfchecking interface that uses MATLAB to evaluate the student's solution to assignments, permitting content reinforcement problems to be posed. The structure of the tutorial is such that as a student moves through the content that introduces new MATLAB syntax, there are one or two interactive assignments on each page that allows the student to practice using the material. Hints are provided with the problems to guide the student, and a solution is provided once the student submits his/her work for evaluation. At the end of each section, there is a Summary Quiz and an Exercise Quiz. Summary Quiz questions consist of true-false, short open response, and multiple choice items. The Exercise Quiz questions, like the interactive assignments, provide the student 1 David Wenhao Huang, Educational Technology, EDCI, Purdue University, 3134, 100 N. University Street, W. Lafayette, IN [email protected] 2 Heidi A. Diefes-Dux, Department of Freshman Engineering, Purdue University, 400 Centennial Mall Dr., W. Lafayette, IN [email protected] 3 P.K. Imbrie, Department of Freshman Engineering, Purdue University, 400 Centennial Mall Dr., W. Lafayette, IN [email protected] 4 Brian Daku, Department of Electrical Engineering, University of Saskatchewan, 57 Campus Dr., Saskatoon, SK, S7N 5A9, Canada [email protected] 5 James Kallimani, School or Aeronautics and Astronautics, Purdue University, 31-8 Hilltop Drive, West Lafayette, IN 47906, [email protected] T1E-30

2 with an interactive environment in which to solve short problems using MATLAB syntax. A solution is provided for each quiz item once the student has submitted his/her work for evaluation. The success rate of the student is summarized in tables the student can view. M-Tutor was adopted for use in a first-year engineering problem solving and computer tools course in Spring 2001 for a number of reasons. Instructors were looking to provide a large number of students with varied computer backgrounds and learning styles an effective and personalized learning experience that would enable them to customize their MATLAB instruction. M-Tutor provided the opportunity for students to select the best time, place, and pace for their instruction, and the imbedded self-assessments provides students immediate feedback on their understanding of the material. Ultimately, it was felt that M-Tutor would provide a means for students to familiarize themselves with MATLAB syntax; this would free course lecture time to concentrate on introducing fundamental engineering concepts and problemsolving. Affective aspects of M-Tutor and its implementation have been studied. A formative evaluation of M-Tutor was done in 1997 and 1998 with second year (fourth semester) students in Electrical Engineering at the University of Saskatchewan [8,6,7]. The initial evaluation focused on students' selfreported overall impressions of an interactive computer-based approach to learning and detailed impressions of M-Tutor features. While students were in favour of the computer-based approach, they were less likely to agree that they would recall the content as a result of the approach. As for the M-Tutor features, the majority of students strongly agreed or agreed that navigation within the tutorial was straightforward (84%), information presented was well organized (97%), clear and easy to follow (97%), and the Summary Quiz and Exercise Quiz items reinforced MATLAB concepts (91%, 88%, respectively). In Spring 2002, a similar evaluation was performed with first-year students at Purdue University using M-Tutor as supplementary instruction in an engineering problem solving and computer tools course. These students rated the M-Tutor features similarly to their Canadian counterparts; the majority of students strongly agreed or agreed that they were able to navigate M-Tutor without assistance (91%) and the Summary Quiz and Exercise Quiz items reinforced MATLAB concepts (88%, 91%, respectively). Overall, the formative evaluations revealed that while M-Tutor provides a navigable framework that is organized and easy to follow, many students were not choosing to use the tool. In fact, 68% of the first-year students surveyed indicated that they used M-Tutor less than one hour per week. Based on these findings, the research team was interested in rigorously applying learning motivation theory to the evaluation of M-Tutor and its implementation. It should be noted that while the research team was working with the author of M-Tutor on a network delivery mechanism for the product, the product itself was not likely to undergo significant immediate re-development as a result of this study. Rather, the expected outcomes were to (1) adapt or develop instruments for measuring students learning motivation while using computer-based instruction, (2) identify instructional interventions to maximize the effectiveness of M-Tutor in students' learning MATLAB syntax, (3) inform the author of motivational issues for future development. ARCS Model for Motivational Design Studies have illustrated that motivational issues are influential on instructional outcomes since they are the fundamental factors that drive student s academic performance [9-13]. Therefore, a diagnostic evaluation of student motivation while using the computer-based tutorial was necessary to collect specific information to help instructors clarify underlying motivational problems. ARCS Model for Motivational Design [14-15] was adopted due to its applicability and practicability in designing, developing, and evaluating instructional materials. Keller suggested that learning motivation is affected by four perceptual components: Attention, Relevance, Confidence, and Satisfaction. Each component plays a critical role in motivating students throughout the learning process. ARCS Model is widely applied to the production of instructional material due to its connectivity between learning motivation theories and instructional design and development processes [16]. The development of ARCS Model of Motivational Design originated from various learning and instructional theories [17]. Keller s primary assumption as to how ARCS Model works is based on the interaction between instructional materials and learners. Attention refers to the learner's response to perceived instructional stimuli provided by the instructional materials. It is important to design instruction such that effective stimuli are present at the beginning of and maintained throughout the learning process at a level to arouse learner s attention and curiosity [18]. Relevance helps learners associate their prior learning experience with the given instructional materials; it also enables learners understanding of the applicability of learned knowledge or skills in their future tasks. Confidence stresses the value of building learners positive expectation towards the learning task. Meaningful experiences support learners confidence development during the learning process. Satisfaction comes when learners are allowed to practice using newly acquired knowledge or skills and receive feedback in a manner that leads to positive attitudes towards the learning task. This also enables learners to receive reinforcement to maintain desirable learning behaviors [15]. The ARCS Model was developed as a conceptual descriptive model for diagnosing problems associated with learning motivation based on theories from performance versus motivation [19] and learning motivation [17]. It is intended for design and development of instructional materials. The procedure includes pretest (pre-measurement on learner s motivational level), motivational intervention development and implementation based on pretest results, and a posttest (post-measurement on learners motivational level). T1E-31

3 Users can apply this model as a cycle for continuous improvement when developing instructional materials. This study is not focused on the use of the ARCS Model to guide the design of computer-based instruction. Rather the focus is on the feasibility of using the ARCS Model to evaluate student motivation induced by an existing product and to guide its future implementation. This paper will present the base-line data gathered during the pilot application of the ARCS Model. IMPLEMENTATION & DATA COLLECTION Setting In Summer 2002, M-Tutor was provided as supplementary instructional material in ENGR 106: Engineering Problem Solving and Computer Tools. ENGR 106 is a required 2-credit hour course for all first-year engineering students at Purdue University. Students successfully completing this course can: Develop a logical problem solving process which includes sequential structures, conditional structures, and repetition structures for fundamental engineering problems, Translate a written problem statement into a mathematical model, and Solve fundamental engineering problems using computer tools. The syllabus is a coordinated mix of introduction to engineering fundamentals, including graphical representation, statistics, and economics, and introduction to computer tools used to solve engineering problems, specifically MATLAB, Excel, and UNIX. The course consists of two 50-minute lectures per 2-hour computer laboratory. During the summer offering, the standard 16 week syllabus is compressed into 8 weeks with 4 lectures and 2 labs occurring each week. Lectures focus on fundamental engineering concepts and problem solving. Computer lab tasks provide students with some structured exploration of the use of new computer tool syntax/procedures and simple fundamental engineering problems. Students apply theory learned in lecture and syntax/procedures learned in lab to the solution of homework problems and projects with engineering context. In the first computer laboratory of the semester, students learned how to access M-Tutor and were instructed to complete the first five pages to familiarize themselves with the product. A schedule for using M-Tutor pages, examples, and quizzes corresponding to topics being covered in lecture and lab was then recommended to the students (Table 1). M-Tutor itself was accessible to the students 24-7 through a Windows Terminal Server [20]. Due Dates TABLE I SCHEDULE OF M-TUTOR USE AND DATA COLLECTION M-Tutor Topic (# pages, # examples, # summary quiz items, # exercise quiz items) June 14 Variables, Scalars (34,28,23,6) June 18 Vectors, Plotting, Script Files (41,45,26,26) June 21 Relational & Logical Operators; For Loops (29,26,22,22) July 2 Vector and Matrix Math (25,23,30,30) July 12 Conditional Statements (12,9,11,11) July 23 While Loops (8,6,6,6) July 31 Motivational Instrument Data Collection First IMMS Second IMMS An existing motivational measuring instrument was adopted for this study. Instructional Materials Motivational Survey (IMMS) [21] was developed based on the ARCS Model. IMMS consists of 36 statements that are rated using Likerttype scales (1 = Not True; 5 = Very True). Each item is mapped to an individual ARCS component and provides a measure of the respondent s perception of that particular component. Cronbach s alpha for this instrument is Studies indicate that ARCS Model is applicable in the computer-based or web-based instructional environment [22-25] although it was originally designed for developing motivating instructional materials in traditional instructional settings (face-to-face, classroom- based). For this pilot study, IMMS was modified for assessing the motivational effectiveness of M-Tutor. Each survey item was revisited and, as needed, re-focused on the research question, which was to diagnose students motivational level in using the tutorial as a learning tool with the expectation that students will effectively learn MATLAB syntax and effectively use MATLAB as a tool for solving engineering problems. For this study, in addition to the 36 rated statements, four pairs of yes/no and open-ended questions were developed for qualitative data collection. Each question was designed to map to one of the ARCS components. For example, the following question is designed for collecting qualitative data on the Attention component of ARCS Model: Generally speaking, do you think M-tutor catches your attention when you first saw it? If yes, please describe how it catches your attention in terms of its design, content, built-in activities, layout, feedback, etc. If not, please tell us how we can modify M- Tutor." The qualitative data enables the research team to obtain insights as to why students give certain ratings to certain survey items. For example, if students indicted that the tutorial interface is not user-friendly, that might lead to a low rating on the ARCS components. The qualitative data also indicates general issues that were praised or criticized by students that can be used to guide the development of instructional interventions. T1E-32

4 Participants The modified IMMS was administered online. The first survey was assigned as part of a regular homework assignment. The second survey was completed in the last computer lab period. Seventeen students were enrolled in ENGR 106 in Summer However, only 16 students completed both the first and second surveys and only these students' data were used in this pilot study. During the summer offering, student enrollment was a mix of true freshmen, students enrolled for the first time who were not freshmen, and students repeating the course (Table II). TABLE II M-TUTOR USE LEVEL BY STUDENT CLASSIFICATION True First Time M-Tutor Use Level Freshman Enrollment High Medium Low Zero Repeat Enrollment Each student was classified as a high, medium, low, or zero level user of M-Tutor. This was done to give a perspective on the time students invest in using M-Tutor to learn MATLAB syntax. As students use M-Tutor, an individual log of pages and exercises accessed and time spent on pages and exercises is created. Two approaches were used to classify the students. First students were classified according to overall time spent using M-Tutor: high users (used product over 14 hours), medium users (7 to 14 hours), low users (below 7 hours), and zero users (did not use M- Tutor beyond the initial lab introduction). Students were also classified according to the percentage of pages, examples, and quizzes that each student accessed. Over the course of the semester, 505 items were recommended (Table I). An item was considered to be accessed if a student spent more than 8 seconds on the item. The categories were high users (accessed over 70% of the recommended items), medium users (35 to 70% of items), low users (below 35% of items), and zero users (less than 5% of items). The lower of the two classifications established a student s final use level classification (Table II). DATA ANALYSIS Data analysis employed both quantitative and qualitative approaches. The purpose of the quantitative data analysis was to measure how M-Tutor effects students learning motivation across the ARCS components. Quantitative data are composed of ratings from IMMS. It is important to remember that responses to a Likert-type scale item generate categorical data that cannot be averaged to provide a mean response for an individual survey item. Therefore, the frequency with which students responded "Mostly True" and "Very True" on individual survey items were computed. For all items mapped to a particular ARCS component, the frequencies with which students responded "Mostly True" and "Very True" were averaged to provide a single quantitative measure of that ARCS component. To interpret the findings from collected qualitative data, a coding process was employed. The coding system (Table III) was developed for analyzing qualitative data based on design principles of multimedia courseware [26-28]. The instructional coding system categorizes qualitative responses from each mapped ARCS Model open-ended question into various instructional components associated with the computer-based tutorial and its implementation. Involved instructional components were coded as interface design, content, learning support, and implementation. Each qualitative response was multiple-coded to prevent subjective interpretation of given words or statements. The research direction was visited repeatedly during the development of the coding system to insure the validity of items [29]. Examples of coded qualitative responses are shown in Table III. Instructional Component Code Interface Design Learning support Implementation TABLE III EXAMPLES OF CODED RESPONSES Keyword Examples Text, graphs, navigational system, interactivity menu Relevance, easiness and difficulty of information Textbook, assistance, Help session, feedback, exercises Course structure, technical infrastructure Actual Response (to mapped ARCS Model Open-Ended Question) It appeared to be userfriendly (Attention) It looks like a professional tutorial (Attention) It's the first step into MATLAB (Attention) The actual M-tutor info that I learned helped me a lot. (Relevance) When I get correct on the screen, I feel that s kinda reward to myself. (Confidence) provides many practice problems (Confidence). It was a good supplement. (Satisfaction) Sometimes monotonous, although m-tutor was a great way to prepare for MATLAB. (Satisfaction) By triangulating the quantitative and qualitative data, valuable information can be gained. First, the research team can identify which instructional component(s) are most influential on students motivational levels. Second, the research team can map the instructional components of the tutorial (interface, content, learning support, and implementation) to the ARCS Model components (Attention, Relevance, Confidence, and Satisfaction). Both outcomes connect motivational theories with the design of instructional materials. RESULTS Pre and post quantitative measures of the ARCS components are shown in Figure 1. There was a relatively low percentage of students responding Mostly True and Very True to the Attention and Satisfaction components. More students, at least in the pre-test, responded Mostly True and Very True to the Relevance and Confidence components. The T1E-33

5 percentage of students responding Mostly True and Very True dropped across all ARCS components on the post-test. As a point of reference, M-Tutor use analysis revealed that the students used M-Tutor primarily at the beginning of the semester with a small increase in use at the end of the semester. M-Tutor use was low during the middle part of the semester. % Responding Mostly True and Very True 60% 50% 40% 30% 20% 10% 0% First (Pre) Second (Post) Attention Relevance Confidence Satisfaction ARCS Component FIGURE 1 PERCENT OF SUMMER 2002 ENGR 106 STUDENTS RESPONDING MOSTLY TRUE OR VERY TRUE TO IMMS ITEMS MAPPED TO ARCS COMPONENTS (N = 16). Table IV shows a selection of qualitative responses to the open-ended questions focused on the Attention component of the ARCS Model. Recall that each open-ended question was designed to map to an ARCS component. The responses to the questions were coded in to instructional components and each response could be coded multiple times. A quantitative summary of the coded responses to the open-ended questions is shown in Table V. This analysis was performed to aide the researchers in clarifying the connections between the instructional components and the ARCS model. For example, the interface design of the tutorial has greater impact on Attention than the content of the tutorial. Whereas, the tutorial content has a relatively high impact on the Relevance and Confidence components. Further, implementation has considerable impact on students Satisfaction level. TABLE IV EXAMPLES OF CODED QUALITATIVE RESPONSES FOR ATTENTION Qualitative Responses for Attention Code It is wonderfully set up Interface Subjects are divided by sections, so it was easyfollowing content. Overall, it is cool way to learn MATLAB. M-tutor is boring... if it wasn't important for those first few days to understand what was going on, then I wouldn't have done it at all it doesn't need to be eye catching it's a source of information nothing more It looks like a professional tutorial. The 'movie' that runs the first time you open it was very impressive. Interface, Learning Support, Interface, Interface, Implementation TABLE V SUMMARY OF CODED RESPONSES TO QUALITATIVE QUESTIONS BASED ON ARCS COMPONENTS Instructional Component Code Attention (N=15 a ) Relevance (N=15) Confidence (N=14) Satisfaction (N=14) Interface Design Learning Support Implementation a N is the total number of coded responses; each response could be coded multiple times DISCUSSION Interpretation of Initial Findings The quantitative measures of the ARCS components are not surprising given the qualitative data and an understanding of the ENGR 106 course. The initial Relevance component rating is relatively high. Students initially know very little about MATLAB and realize that learning MATLAB syntax is required for success in ENGR 106. Attention was rated relatively low; this may correspond to qualitative responses that cite the interface design as being boring. Satisfaction may also have been low due to students relatively low access rate to exercises. Students who used M-Tutor tended to access a greater percentage of the recommended content pages than the exercises. As Satisfaction comes from students practicing their skills and receiving feedback on their work, the low Satisfaction rating is expected. For two reasons, the drop in motivation across ARCS components from pre to post IMMS is also not surprising. First, from an implementation standpoint, fewer M-Tutor items were recommended as the semester progressed because the M-Tutor content becomes less applicable to the course. This might explain the drop in the Relevance rating. Second, students were generally low users of M-Tutor with most of their usage occurring at the beginning of the semester. The low use of M-Tutor may follow the initial low Attention and Satisfaction ratings. Low use may also be attributed to M- tutor being recommended as supplementary instruction to lecture, lab, and a MATLAB textbook. In order to recommend appropriate instructional interventions to improve student motivation, an understanding of the connections between the instructional components and the ARCS model is needed. The qualitative data analysis results (Table V) imply that interface design is critical for stimulating students Attention. While the content (difficulty and organization of information) plays an important role in establishing Relevance and Confidence levels. How the tutorial is implemented into the overall course structure affects students perceived Satisfaction level. Applicability of ARCS Model ARCS Model offers an approach to diagnosing students motivational issues while using computer-based instruction. By comparing pre and post survey results, researchers can identify core motivational issues and understand how they change over time. In addition, researchers can investigate the T1E-34

6 interactions among all four ARCS components to optimize the instruction for desired motivational gain. Students quantitative ratings of each ARCS component is not sufficient for developing corresponding instructional interventions. The meaningful linkage between students ratings and individual ARCS components needs to be established with qualitative data collection and coding as seen in Table III. Each identified instructional component has a unique impact on instructional outcomes. The qualitative coding process helps researchers uncover the meaning behind each rating leading to the development of effective instructional intervention(s). Interpretation of ARCS results can be difficult due to the complexity involved in studying learning motivation. Many variables need to be taken into account when investigating any motivational issues associated with learning. Conversely, any motivational issue is never caused by only one problem. Thus, researchers need to take an eclectic approach to diagnosing motivational issues. Adopting ISD Process for Evaluating Computer-Based Instruction and Implementation Instructional System Design (ISD) process [30-32] offers engineering educators various options for efficiently evaluating off-the-shelve computer-based instructional programs. With the systematic and holistic approach [33-34] educators can simultaneously address learners needs, strengths and weakness of instructional setting, effectiveness of instructional strategies, appropriateness of assessment, and measurement of learning gains. Models with specific instructional emphasis, such as the ARCS Model s focus on learning motivation, are available for any instructional evaluation task. Though the ISD process is not proven completely effective, it definitely is a useful tool for engineering educators to better understand the compatibility between students needs and computer-based instruction and it implementation. SUMMARY & NEXT STEPS The authors have presented the ARCS Model of Motivational Design and shown how it can be used to evaluate students learning motivation when using an existing computer-based tutorial. Triangulation of quantitative and qualitative data provided initial insight on the impact of instructional components on students learning motivation as represented by four motivational components. The Summer 2002 implementation of M-Tutor and the modified IMMS instrument was a pilot and provided preliminary data before full implementation in ENGR 106. In Fall 2002, a number of changes were made to the way M-Tutor is implemented in ENGR 106; the aim being to encourage a higher level of use of M-Tutor. The most significant change entailed repositioning M-Tutor as the primary means of learning MATLAB syntax rather than as supplementary instruction. This is a non-trivial issue when delivering M-Tutor through a Windows Terminal Server to over 1400 first-year engineering students. Other changes included linking M-Tutor assignments more tightly with homework assignments and grades and discussing the purpose of M-Tutor in lecture to help students set appropriate expectations for computer-based learning. The impact of this new implementation model on students motivation is under investigation using the ARCS Model. The modified IMMS instrument is also undergoing validation. REFERENCES [1] MathWorks, MATLAB, [2] Black, K., Matlab Tutorial Information, Department of Mathematics, University of New Hampshire, Durham, NH, May [3] Gockenbach M., S., A Practical Introduction to MATLAB, Michigan Technological University, [4] Messner W., Tilbury D., Control Tutorials for MATLAB and Simulink : A Web-Based Approach, Prentice Hall, [5] Sigmon K., MATLAB Tutorial, University of Florida, [6] Daku B., L., F., and Jeffrey K., D., "An Interactive Computer Based Tutorial for MATLAB", Proceedings: Frontiers in Education 2000, Kansas City, MO, November [7] Daku B., L., F., and Jeffrey K., D., "Development of an interactive CD ROM-based tutorial for teaching MATLAB," IEEE Transactions on Education, Vol. 44, No. 2, [8] Daku B., L., F., M-Tutor: An Introduction to MATLAB; M Tutor Student Evaluation Results 1997, University of Saskatchewan, [9] Weiner B., An Attributional Theory of Achievement Motivation and Emotion, Psychology Review, Vol. 92, 1985, [10] Ames C., Classroom: Goals, structures, and student motivation, Journal of Educational Psychology, 1992, pp. 84, [11] Anderman, E., M., and Maehr M., L., Motivation and schooling in the middle grades, Review of Educational Research, 1994, pp. 64, [12] Bandura, A., Self-efficacy: The exercise of control, New York: Freeman, [13] Durik A., M., and Harackiewicz J., M., Journal of Experimental Social Psychology, 2003, pp [14] Keller J., M., Strategies for Stimulating the Motivation to Learn, Performance and Instruction Journal, Vol. 26, No. 8, pp. 1-7, [15] Keller J.,M., The Systematic Process of Motivational Design, Performance and Instruction Journal, Vol. 26, No 9-10, pp. 1-8, [16] Huang W., and Johnson J., Motivational level of a computer-based simulation: A formative evaluation of the US Army Recruiting Simulation (USAREC), AECT 2003 International Conference, Dallas, TX, [17] Driscoll M., P., Introduction to theories of learning and instruction, Psychology of learning for instruction 2nd ed., Boston: Allyn and Bacon, pp.3-28, [18] Keller J., M., Motivational Design of Instruction, Instructional- Design Theories and Models, Hillsdale, NJ: Erlbaum, [19] Steers R., M., and Porter L., M., Motivation and Work Behavior, New York: McGraw-Hill, T1E-35

7 [20] Daku B., L., F., and Diefes-Dux H., A., "An Effective System for Implementing Self-Learning, On-Line Instruction for Engineering Students", Engineering Education and Research : A Chronicle of Worldwide Innovations, ineer (International Network for Engineering Education and Research), Arlington, VA, [21] Keller J., M., Instructional Material Motivational Survey, Florida State University, Unpublished documents, [22] Keller J., M., and Suzuki K., Use of ARCS Motivation Model in Courseware Design, Instructional Designs for Microcomputer Courseware, Hillsdale, NJ: Lawrence Erlbaum Associates, pp , [23] Keller J., M., Motivation in Cyber Learning Environments, International Journal of Educational Technology, Vol. 1, No. 1, [24] Keller J., M., and Song S., H., The ARCS Model for Developing Motivationally-Adaptive Computer Assisted Instruction, Proceeding of Selected Research and Development Papers Presented at the National Convention of the Association for Educational Communications and Technology (AECT), pp. 19, [25] Knowlton A., Shellnut B., and Savage T., Applying the ARCS Model to the Design and Development of Computer-Based Modules for Manufacturing Engineering Courses, Educational Technology Research & Development, Vol. 47, No. 2, [26] Park I., and Hannifin M., J., Empirically-Based Guidelines for the Design of Interactive Multimedia, Educational Technology Research & Development, Vol. 41, No. 3, 1993, pp [27] Szabo M., and Kanuka H., Effectis of Violating Screen Design Principles of Balance, Utility, and Focus on Recall Learning, Study Time, and Completion Rates, Journal of Educational Multimedia and Hypermedia, Vol. 8, No. 1, 1998, pp [28] Evans C., and Edwards M., Navigational interface design for multimedia courseware, Journal of Educational Multimedia and Hypermedia, Vol. 8, No. 2, 1999, pp [29] Coscarelli W. and Shrock S., Criterion-referenced test development. 2nd ed, International Society for Performance Improvement, [30] Gentry C., G., Introduction to instructional development: Process and technique. Belmont, CA: Wadsworth Publishing Company, [31] Gustafson K., L., and Branch, R., M., Survey of instructional development models, third edition, Syracuse, NY: ERIC Clearinghouse on Information and Technology, [32] Hannum W., and Hansen, C., Instructional systems development in large organizations, Englewood, Cliffs, NJ: Educational Technology Publications, [33] Dick W., and Cary L., The Systematic Design of Instruction, Third Edition: Harper Collins, [34] Smith P., L., and Ragan T., J., Instructional design, New York: Merrill/Macmillan College Publishing, T1E-36