1 21st Annual Conference on Distance Teaching and Learning click here -> Using Online Surveys to Evaluate Distance Education Programs Elaine Strachota, PhD Distance Learning Coordinator and Faculty Milwaukee Area Technical College Steve Schmidt, PhD Training Coordinator Harley-Davidson Motor Company Simone Conceição, PhD Assistant Professor University of Wisconsin-Milwaukee Steps for Designing Surveys for Distance Education The process of gathering feedback from educational program participants, whether in a corporate or formal education setting, is a critical step in educational program development and implementation. One of the best ways to gather opinions about a particular topic or issue is through the use of survey research (Wallen & Fraenkel, 2001). Just as hard-copy surveys are useful in traditional classroom settings, online surveys can be useful in gathering information from participants in online educational situations or from employees who participate in online or computer-based training in the workplace. Although the use of online surveys can be an efficient way to gather data for both research and evaluation, it is computer programmers who have been at the forefront of online survey development. Dillman and Bowker (2001) posit that the use of web surveys seems to have caught the survey methodology community somewhat by surprise. They further explain that Rather than being at the forefront of this latest innovation in the conduct of social surveys, survey methodologists are playing catch-up as they learn to master these new survey development tools" (p. 1). The development of an online survey instrument, however, means that factors in addition to those associated with traditional survey development should be considered. Dillman, Tortora, and Bowker (1998) recommend that questions must be presented in a way that they can be understood and answered accurately by all recipients. And, the likelihood of responding should not be affected by peoples lack of computer skills (p. 3). To address these issues, Dillman et al. (1998) advise online survey developers to consider the following: Survey design should be respondent-friendly, and take into account the inability of some respondents to receive and respond to web questionnaires with advanced programming features. Survey design should consider both the logic of how computers operate and the logic of how people expect questionnaires to operate. As many survey populations do not have access to the Internet, a survey design that allows for survey use in other modes of data collection (i.e., paper questionnaires, interviews, or telephone surveys) is important. (p. 3) Matrix Comparing Online Survey Software Programs A matrix comparing online survey software programs is provided in Table 1. We selected only five online survey programs. These programs were selected based on the level of free or discounted options for creating and conducting online surveys. Our matrix compares seven common features in online surveys: menu types, cost, minimum requirements, survey deployment, design, security, and report capabilities. 1
2 21st Annual Conference on Distance Teaching and Learning click here -> Table 1. Online Survey Software Programs Matrix FEATURE Zoomerang SurveyMonkey SurveyShare WebSurveyor QuestionPro Menu Types 15 question 17 question types 22 question types 16 question types 14 question types Cost types Free templates; unlimited free subscription limited to 30 questions and 100 responses Professional level - $599/yearunlimited surveys, questions, responses, performs higherlevel analysis, reporting Same product available for $350/year for educational and non-profit organizations Basic subscription is free; professional subscription $200/year (1000 responses/month) 2 surveys 100 max responses Free 1 survey 200 max responses $49 for 3 months 2 surveys 400 max responses (to both surveys) $99 for one year 5 surveys 1000 max responses (to all surveys total) $199 for one year 15 surveys 2000 max responses (to all surveys total) $299 for one year 40 surveys 6000 max responses (to all surveys) $599 for one year Note: Student pricing is available Free templates; one month free access; 100 responses Educational institutions can apply for unlimited free use through an Academic Grant program Conduct a 1 time survey with unlimited questions, unlimited responses $495/3 months ideal for grad student Standard level- $1495/year unlimited surveys, questions, responses Premium and server levels also available Free templates; one month free access; 100 responses; professional subscription $588/year Minimum Requirements Survey Deployment Design Browser; Internet URL address is created - can be deployed in either email or website link Templates or customized survey creation, copy and edit past created survey, can use multiple pages, enable skip or branch logic, can make questions mandatory, can randomize answers, survey reminders, can use company logo, personalize invitations, available in 40 languages. Link to an email message; link for a web page; link to an email list Web-based; add logo; randomize answer choices ; customize the path a respondent; custom themes; easy copying/pasting surveys URL address is created - can be deployed in either email or website link Surveys can be designed from scratch or using one of Surveyshare s templates. URL address is created - can be deployed in either email or website link Templates or customized survey creation, question and response libraries, comprehensive skip logic, validation and formatting, Multi-lingual surveys - available in 20 languages Email / Website Link; Email/List Management; Intelligent Popup Surveys; Newsletter/Email Campaign Integration Page breaks; headers and footers thank you page; buttons; graphics; survey progress bar; multi-lingual; branching quota ; management extraction questions; text piping; extraction with text piping and looping; randomization of questions; randomization of answers; automatic redirect 2
3 21st Annual Conference on Distance Teaching and Learning click here -> FEATURE Zoomerang SurveyMonkey SurveyShare WebSurveyor QuestionPro Security Report Capabilities Website Address password to access results Stores all data in secured, online databases Automatic graphs of survey responses, customized charts and graphs available, Download results to local computer for analysis password to access results; control viewing of results and how they can be used Download results to local computer; transfer results to Excel; filter results password to access results, Security and privacy policies to protect your surveys, respondents and results, Password protected survey access for shared terminals Cross tabulation options and customized charts and graphs available. Subscription members (only) can download survey password to access results, Security and privacy policies to protect your surveys, respondents and results, Password protected survey access for kiosks or shared terminals Automatic graphs of survey responses, Provides comments and narrative to reports, Export raw data to Excel, SAS, SPSS, Lotus, and Access. password to access results Integrated Results Portal; Automatic Chart/Graphs; Integrated Cross- Tabulation Tool; Segmentation and Grouping Analysis; Time based Grouping Analysis; Integrated TURF (Total Unduplicated Reach and Frequency) Analysis; Integrated Conjoint Analysis; Excel data download Zoomerang.com SurveyMonkey.com SurveyShare.com Websurveyor.com QuestionPro.com Advantages of Online Surveys for Evaluating Distance Education Programs There are advantages to using online surveys to gather data. When compared with traditional mail or telephone surveys, Schonlau, Fricker, and Eliott (2001) note that Sampling error issues with Internet surveys are generally the same as those with conventional surveys (p. 29). Schonlau et al. (2001) state that Internet surveys may be preferable to mail or telephone surveys when the survey contains questions of a particularly sensitive nature. The use of Internet-based surveys may allow the program developer to delve deeper into evaluation-related topics, and may result in more honest feedback than would traditional survey methodologies. Advantages of online survey distribution include increased time efficiency, decreased data entry error, increased item response rate and decreased cost. Due to available technology administration of online surveys has resulted in increased time efficiency in both distribution and receiving of survey results. The majority of colleges, universities, and businesses today that deliver distance learning courses and programs have invested in a Learning Management System (LMS). An LMS is designed to allow for the posting of an announcement with a direct web link to an online survey instrument. The LMS offers efficiency for distance learning students who are enrolled in a course because they can access the survey instrument and complete it at a convenient time. An LMS also contains a database of e-mail addresses for all students. When an LMS is not available the process becomes far more time-consuming as e-mail addresses must be obtained before a mass mailing can be conducted. Both of these methods however still remain far more time efficient than telephone or mail surveys. 3
4 21st Annual Conference on Distance Teaching and Learning click here -> Decreased data entry error is an important factor when conducting survey research. Traditional telephone and mail surveys require manual data entry. Electronic survey distribution does not require manual entry as data are collected through a given software program and then downloaded into a statistical program such as SPSS for detailed analysis. This feature saves time, is more accurate, and is especially useful when collecting data with a large sample size. Increased item response rate can also be controlled through electronic surveys as most of the software programs can be set so that all items must be completed before submission. Traditional mail surveys often are returned with data being absent in given fields. Further, qualitative item response rate can be increased due to legibility of electronic responses versus occasional illegible mail responses. Schaefer and Dillman (1998) found that electronic surveys resulted in a quicker return rate, a slightly lower item non-response rate, and more complete answers to open-ended questions. Another advantage of electronic survey distribution is decreased cost compared to telephone and mail distribution. A major concern of survey research is the potential for high non-response error. In order to minimize non-response error, multiple contacts are made to have participants complete a given survey. Schaefer and Dillman (1998) support the use of multiple contacts as a method for survey dissemination as it increases the response rate. Informing participants that a survey will be forthcoming through an announcement, presenting the actual survey link, a thank-you or reminder e-mail with the survey link, as well as a final e-mail reminder with the survey link can all be entered in an LMS and/or e-mailed at no cost. Contacting participants multiple times through mail or telephone distribution, however, is far more costly and time consuming. Electronic survey distribution, therefore, is a far more effective method for data collection and evaluation of distance learning programs. Limitations of Online Surveys for Evaluating Distance Education Programs The online survey must be designed so that it has user-friendly navigation. Certain computer skills such as selecting radio buttons, use of drop-down menus, and scrolling are commonly required skills. Correctly reading the direction and paying attention to detail is also important. Although, distance learners should have the necessary computer skills that an online environment requires, it is important that the design and layout of the survey be user-friendly. Dillman and Bowker (2001) identify the need to provide specific directions for each required computer action as well as any other necessary instruction to avoid frustration and termination of the survey before completion. A potential limitation of e-mail survey research includes the difficulty in sorting out responders versus non-responders especially with a large sample size. Occasional annoyance may be perceived by receiving multiple e-mails requesting a response. E-mail, like mail delivery, may be perceived as junk mail and deleted without responding. Non-response error tends to be higher among distance learners than among on-campus learners. Surveying on-campus learners can be conducted at the conclusion of an educational program. Although distance learners can be surveyed at the end of an educational program, compliance with survey completion may be lower. In order to achieve higher compliance among distance learners it is suggested that some incentive be established as part of the process. It is important to recognize that conducting online surveys versus telephone or mail surveys is a good fit for distance learners. Although conducting online surveys is far more efficient and cost effective it is important to recognize the audience being surveyed. Although we live in a technological age, we are still faced with the issue of computer access. While access has increased such that 204.3 million Americans have Internet use in their home, 25 percent of Americans still do not have Internet access (Kim, 2004). Timmerman (2002) suggests, Before assuming web-based surveys are the most advantageous method for surveying a target population, it should be determined if those advantages exist within that population (p. 5). Although conducting online surveys is far more efficient and cost effective, knowledge and 4
5 21st Annual Conference on Distance Teaching and Learning click here -> application of online survey software as well as demonstrating the necessary technology skills is required for this methodology. The efficiency and cost effectiveness of conducting online survey research however far outweighs the time and efforts required to research and learn a specific software program application. References Dillman, D. A., & Bowker, D. K. (2001). The Web questionnaire challenge to survey methodologists. Retrieved from http://survey.sesrc.wsu.edu/dillman/zuma_paper_dillman_bowker.pdf Dillman, D. A., Tortora, R. D., & Bowker, D. (1998). Principles for conducting Web surveys. Social and Economic Sciences Research Center Technical Report 98-50. Retrieved from http://survey.sesrc.wsu.edu/dillman/papers/websurveyppr.pdf Kim, G. (2004). Three out of four Americans have access to the Internet, according to Nielsen/Netratings. Retrieved May 22, 2005, from: http://direct.www.nielsen-netratings.com/pr/pr_040318.pdf Schaefer, D., & Dillman, D. (1998). Development of a standard e-mail methodology: Results of an experiment. Retrieved May 22, 2005, from: http://survey.sesrc.wsu.edu/dillman/papers/e- Mailppr.pdf Schonlau, M., Fricker Jr., R. D., & Elliott, M. N. (2001). Conducting research surveys via e-mail and the Web. Rand Publications. Timerman, A. (2002). Introduction to the Application of Web-Based Surveys. ERIC Document 474097 Wallen, N. E., & Fraenkel, J. R. (2001). Educational research: A guide to the process. Mahwah, NJ: Lawrence Erlbaum. Biographical Sketches Elaine Strachota is a faculty member and Distance Learning Coordinator at Milwaukee Area Technical College. Address: Milwaukee Area Technical College Distance Learning Department 7 00 W. State St. Milwaukee, WI 53233 E-mail: strachoe@matc.edu Phone: 414-297-7160 Fax: 262-377-4392 Steve Schmidt is a Training Coordinator in the Sales Department of Harley-Davidson Motor Company. Address: Harley-Davidson Motor Company 3700 W. Juneau Avenue Milwaukee, WI 53208 E-mail: steve.schmidt@harley-davidson.com Phone: 414-343-4660 Fax: 414-343-4161 Simone Conceição is an Assistant Professor at the University of Wisconsin-Milwaukee. Address: University of Wisconsin-Milwaukee School of Education P.O. Box 413, Milwaukee, WI 53201 E-mail: simonec@uwm.edu Phone: 414-229-4615 5
6 21st Annual Conference on Distance Teaching and Learning click here -> Fax: 414-229-5300 6