Determining the Productivity of Instructional Systems Dr. Philip McGee, Program Coordinator of the Masters in Human Resource Development, Clemson University, South Carolina ABSTRACT The Productivity Model described in this article is applied in such a way as to determine the productivity of instructional systems by describing a simple to use strategy for assessing and evaluating instructional systems. Data generated by the model enables instructional designers to determine not only the effectiveness and efficiency of any instructional system, but also its overall productivity. Data generated by the Productivity Model may be used not only to assess an instructional system, but may also be used as an evaluation tool enabling instructional designers to pin-point where and when to take corrective actions. INTRODUCTION If you enter the keywords instructional design + productivity into any Internet search engine you will generate a number of links to online papers and sites extolling the benefits and advantages of using instructional system principles and techniques to design and develop training programs and materials that produce predictable learning outcomes. The same holds true for searching online libraries and databases of journal articles. However, when one examines the literature related to evaluating training systems, more often than not, one is referred to the work of Donald Kirkpatrick (1998) and Jack Phillips (1997). Unfortunately, neither Kirkpatrick s Four Levels of Evaluation nor Phillips work determining the ROI of training systems always provide the type of information needed by instructional designers to evaluate and improve the instructional systems for which they are responsible. This is not to say the topic of instructional design and productivity has not been addressed in the past. Ivor Davies wrote on the issue of designing instructional systems that are both effective and efficient in his book Instructional Technique (1981). Davies based his writings on the definitions developed by Peter Drucker (1974) who wrote that Efficiency is concerned with doing things right. While, Effectiveness is doing right things. Now, thirty years later, William Rothwell and H. C. Kazanas in their book Mastering the Instructional Design Process (2003) state that These terms have no universally accepted definitions. They continue their discussion of effectiveness and efficiency by quoting Drucker, as did Davies, and then press on to discuss that instructional design is guided by a model of human performance. It is at this point that I would like to offer a new evaluation model using the concepts of effectiveness and efficiency, and to explore how this model can be used as a strategy for improving the productivity of instructional systems. A MODEL FOR DETERMINING THE PRODUCTIVITY OF INSTRUCTIONAL SYSTEMS The Oxford English Dictionary (1989) states that productivity is the rate of output per unit of input or output/input = productivity. As with calculating the mean for a set of scores, valuable insights can be lost when data is reduced to a single measure. It is with this concern in mind that I would like to describe the Productivity Model and how it can be applied by instructional designers and trainers. Let us begin with the assumption that any system can be described as being composed of three basic elements as illustrated below by Figure 1. The Journal of Human Resource and Adult Learning * November 2006 1
Resources + Activity = Results Figure 1. Elements common to all systems First, all systems produce results. Granted, the results produced by any given system may at times be difficult to predict. Secondly, to achieve results, something must happen; something must be done. In other words, activity must take place. Third, in order to have activity, resources must be used. These three elements of a system hold true whether the system is mechanical, electrical, biological, financial, social or educational. The next step to understanding this model is to examine the dynamic relationships that exist between the elements of a system. We will start with the relationship between results and activity. Effectiveness It can be said that something is effective only when an activity produces a predetermined result. For example, when an instructional designer develops a program of instruction, the designer should first strive to create an instructional system that produces not only the desired results, but also a system that produces the desired results with a high degree of predictability. It is only when a system does what it is supposed to, with a high degree of predictability, that we able to say that it is effective. Effectiveness Dimension Qualitative Measures Asks: How Well? Resources + Activity = Results Figure 2. The Effectiveness Dimension Effectiveness is concerned with "how well" something works i.e., produces predetermined desired results. This concern for "how well" is the basis for the concept we call quality. As designers of systems, we must strive first for effectiveness. For without it, there is little reason to proceed with the design of any system. Efficiency Efficiency is the dynamic relationship that exists between resources and an activity. Efficiency is concerned with quantity and asks the question, "How much?" However, this is a dangerous question to ask. Dangerous in that if we cut resources too much, we run the risk of producing poor results. Resources + Activity = Results Efficiency Dimension Quantitative Measures Asks: How much? Figure 3. The Efficiency Dimension 2 The Journal of Human Resource and Adult Learning * November 2006
Effectiveness This is not to say that we should not be concerned with resources and their associated costs. We should, because within every system there is an optimum balance between resources and activity, and activity and results. This optimum balance is known as productivity. Effectiveness Dimension Qualitative Dimension Asks: How well? Resources + Activity = Results Productivity Efficiency Dimension Quantitative Measures Asks: How much? Figure 4. Productivity defined as a measure of effectiveness and efficiency Productivity We can see this balance between effectiveness and efficiency in the illustration below in Figure 5. System (A), which is neither effective nor efficient, cannot be said to be productive. System (B), while very effective, cannot be said to be productive, because it is not efficient. System (C), while efficient, is not effective. Therefore it cannot be considered productive. Only system (D), which is both effective and efficient, can be said to be productive. In application, it is possible to set performance standards for any system in terms of effectiveness and efficiency, and to determine corrective courses of action when evaluative data is generated and graphed. High B D Slope of Productivity Low A C Low Efficiency High Figure 5. The Slope of Productivity Applying the Model to Improving the Productivity of Training Systems If we are to measure and improve the productivity of a training system, we must focus our attention on two distinct measures. The first is the quality of the instruction, i.e., what should be taught? And the second is the quantity The Journal of Human Resource and Adult Learning * November 2006 3
Average Competency Level Achieved of resources needed to deliver the instruction, i.e., by what means should the curriculum be taught? This interrelationship is illustrated below in Figure 6. Curriculum Development Asks: What should be taught? Resources + Activity = Results Productivity Instructional Tactics Asks: By what means should the curriculum be taught? Figure 6. Curriculum development versus Instructional Tactics Let me expand on this concept. Instructional quality is the major concern of curriculum development. Curriculum development answers the question, What should be taught? It is during the development of the instructional curriculum that a performance standard for the curriculum is established and program content is identified. Instructional quality is often measured by how well participants are able to predictably achieve the objectives of the program. This data is most often gained through traditional testing procedures. For example, a particular curriculum may produce results, wherein an average participant achieves a 95% level of competency on the material presented. 100 % Instructional System 95% 90% 85% $15 $12 $10 $7 0 5 0 5 Delivery Cost per Participant Figure 7. Average competency level achieved versus delivery cost per participant Instructional quantity, on the other hand, falls into the domain of instructional tactics, which is the process of determining and selecting the most efficient method and media for delivering a program of instruction (curriculum). The goal of instructional tactics is to answer the question, By what means should the curriculum be taught? Again, a performance standard must be established by which to measure this dimension. Common standards are money, time, instructional staff, equipment required, i.e., instructional resources. 4 The Journal of Human Resource and Adult Learning * November 2006
Average Competency Level Achieved In order to determine the productivity of an instructional system, we must consider both the results produced by the curriculum and the instructional resources required to deliver the curriculum. Keep in mind that productivity is a ratio or composite measure of both the effectiveness and efficiency of a system. For example, in Figure 7 above we have an instructional system in which 95% of the participants achieve competency using a media/method to deliver the training which has a per participant cost of $125. By simply plotting the data using an X-Y graph and noting where the data points are located in relation to the Slope of Productivity, it is possible to determine the effectiveness, efficiency, and productivity for any given system. Taking Corrective Action The power of this evaluation strategy is that it enables trainers and instructional developers to identify where they should take corrective action. Using Figure 8, it can be seen that to improve the instructional system, attention should be focused upon decreasing the cost of delivering the training. 100% 95% Instructional System 90% 85% $150 $125 $100 $75 Delivery Cost per Participant Figure 8. Determining corrective action using the Productivity Model In other situations, given different data, trainers and designers may want to improve their curriculums. This brings us to the following: Factors that Influence the Effectiveness of an Instructional System (Curriculum Development) - Needs Assessments - Assessment of Learners - Analysis of Work Settings - Job, Task, or Content Analysis - Statements of Performance Objectives - Performance Measurements (test items) - Sequence Performance Objectives Factors that Influence the Efficiency of an Instructional System (Instructional Tactics) - Instructional Techniques - Designs for Instructional Materials The Journal of Human Resource and Adult Learning * November 2006 5
Average Competency Level Achieved - Media - Delivery mode / medium - Instructional Resources, i.e., money, time, instructional staff, or required equipment. Comparing Two or More Instructional Systems Trainers and instructional developers are often asked to compare and contrast two or more instructional systems and to make a recommendation. Using the Productivity Model, this becomes a simple task as illustrated below. Let us say that we have two instructional systems that deliver the same information and skills. 85% of the trainees, who use Instructional System A, achieve competency for an average delivery cost of $100 per participant. However, 95% of the trainees who use Instructional System B achieve competency using media and methods which have a per participant cost of $125. To determine which of the two instructional systems is the most productive we need only to graph out the two instructional systems as illustrated in Figure 9. 100% System A: 85%, $100 System B: 95%, $125 95% System B 90% 85% System A $150 $125 $100 $75 Delivery Cost per Participant Figure 9. Comparing two or more instructional systems using the Productivity Model Because Instructional System B falls higher and closer to the line we call the "Slope of Productivity," we can determine it is the most productive. It should be noted that using this strategy, it is possible to compare any number instructional system to determine which is the most productive. A Grounded Model The concepts and relationships detailed in this article are based upon established principles from a variety of disciplines. The basic elements of a system: resources, activity and results, are described throughout the literature on systems theory and are sometimes referred to simply as an input/output model. Accountants and financial people, who often speak of return on investment, have known the relationship between resources and results: (ROI). ROI is where returns are results and investments are costs, and in this way, are able to determine the health of a business enterprise (activity). Physicists and engineers recognized long ago that in order to describe the performance of various phenomena and systems, they had to be described in terms of dynamic relationships between two variables. For example, miles per gallon or feet per second. 6 The Journal of Human Resource and Adult Learning * November 2006
Thomas Gilbert, a founding father of the human performance field, developed the First Leisurely Theorem, which says that worth is equal to value divided by costs. In other words, activity adds value to resources (cost) and results in something of greater worth. Peter Drucker pointed out in the 1970s that effectiveness was doing right things, while efficiency was doing things right. In the field of education, these concepts were expanded upon by Ivor Davies and applied to decisionmaking concerning instructional methods. CONCLUDING THOUGHTS Because the Productivity Model is grounded in general systems theory, it can be used for a wide range of design and evaluation activities. The author has not only used the Productivity Model to analyze various training systems, but has used to the Productivity Model to analyze production processes, as a starting point for group discussions that focused on improving processes ranging from sales, to product development, to distribution and logistics. In the classroom, the Productivity Model has proved useful as a teaching tool by providing students with a framework on which to arrange problems solving concepts that in turn help to facilitate the development of their critical thinking skills. REFERENCES Davies, Ivor K. (1981). Instructional Technique. New York: McGraw-Hill Book Company. Drucker, Peter F. (1974). Management Tasks, Responsibilities, Practices. New York: Harper & Row Publishers. Gilbert, Thomas F. (1978). Human Competence: Engineering Worthy Performance. New York: McGraw-Hill Book Company. Kirkpatrick, Donald (1998). Evaluating Training Programs (2nd ed.). San Francisco: Berrett-Koehler Publishers, Inc. Simpson, J.A. and Weiner, E.S.C. (1989) Oxford English Dictionary (2 nd ed., Vols. 1-20). New York: Oxford University Press. Phillips, Jack J. (1997). Handbook of Training Evaluation and Measurement Methods (3rd ed.). Houston: Gulf Publishing Company. Rothwell, William J. and Kazanas, H. C. (2003). Mastering the Instructional Design Process (3rd ed.). San Francisco: Pfeiffer. The Journal of Human Resource and Adult Learning * November 2006 7