Performance-based Assessment in an Online Course: Comparing Different Types of Information Literacy Instruction

Transcription

1 Yvonne Mery, Jill Newby, Ke Peng 283 Performance-based Assessment in an Online Course: Comparing Different Types of Information Literacy Instruction Yvonne Mery, Jill Newby, Ke Peng abstract: This study investigates whether the type of instruction (a single face-to-face librarianled instruction, instructor-led instruction, or an online IL course the Online Research Lab) has an impact on student information literacy gains in a Freshman English Composition program. A performance-based assessment was carried out by analyzing bibliographies in a required controversy paper. Descriptive, correlation, and regression analysis showed that the type of instruction did impact the quality of the bibliographies. Students in the online IL course had higher quality bibliographies than those students who received a one-session face-to-face instruction. Introduction Given current economic conditions, it is becoming more critical to provide evidence of value through quantitative and qualitative data. Academic librarians are increasingly interested in demonstrating the value of their services and resources to primary stakeholders. Given current economic conditions, it is becoming more critical to provide evidence of value through quantitative and qualitative data. For information literacy (IL) instruction, the gold standard for evidence of value is being able to show that information literacy fluency has an impact on students academic success. Often information literacy assessment is portal: Libraries and the Academy, Vol. 12, No. 3 (2012), pp Copyright 2012 by The Johns Hopkins University Press, Baltimore, MD

2 284 Performance-based Assessment in an Online Course done via a quick survey or multiple choice quiz at the conclusion of a session. However, the information literacy skills we teach go beyond what can be assessed with fixed-choice instruments, and as librarians we are more concerned with what our students can do and what they can produce, as opposed to what they can recall from a one-shot library session. The ACRL guidelines for information literacy best practices advise that more than one kind of outcome measures be used and to focus on student performance, knowledge acquisition, and attitude appraisal. 1 University of Arizona instruction librarians wanted to study the impact that an online IL course had on students work and compare online instruction with more traditional forms of face-to-face instruction. In the past, UA librarians had been very successful in training English 102 graduate teaching assistants to incorporate information skills into their syllabi, thus freeing up several librarians from teaching multiple English 102 IL sessions. 2 In 2009, the University of Arizona instructional librarians moved away from face-to-face instruction. Part of this movement included the creation of an online IL course, the Online Research Lab (ORL), designed to supplement the English 102 course. 3 In order to provide evidence that this online teaching approach could be as effective as the face-to-face instruction, learning outcomes assessment was first piloted in 2009 and fully implemented in spring Librarians created and adopted test items from the original SAILS bank of questions that were analyzed, revised, and finally proven to be valid and reliable. 4 In addition, pre- and post-tests were administered to a sampling of students from three different types of instructional methods. Analysis of results showed that students in the online course performed better than the other two methods of IL instruction delivery in most cases. 5 For the authors of this study, the third part of the learning outcomes assessment was to examine the application of the information literacy skills taught in the course. This was done through a review of student bibliographies gathered from a research paper required in the English Composition course. An analysis of students citations allows librarians to assess student learning in a more authentic manner than a multiple-choice test, since a bibliography is a performance-based type of assessment. Megan Oakleaf offers a number of benefits of performance-based assessment including an increase in students intrinsic motivation, the ability to assess critical thinking skills, and a closer alignment An analysis of students citations allows librarians to assess student learning in a more authentic manner than a multiplechoice test, since a bibliography is a performance-based type of assessment. with instructional goals. 6 Karen Hovde writes that an investigation of student bibliographies allows for a flexible, non-invasive, and timeefficient assessment of student behavior and work. 7 A citation study of student research behavior also has the added benefit of not requiring students to produce any additional work or take a test. 8 This sort of study typically looks at the number and kind of sources that students are using and may also include a review of citation formatting and correlation with other student performance measures, most often grades. Thus, the purpose of this research study was to determine if the type of instruction students received, face-to-face and online, made a difference in the quality of their bibliographies.

3 Yvonne Mery, Jill Newby, Ke Peng 285 Literature Review Citation analysis as a tool to measure the effectiveness of information literacy instruction is well documented in the literature. One of the best known citation analysis studies is that of Philip Davis and Suzanne Cohen, first conducted in 2001 and then replicated in In order to determine the effect that the Web had on student research papers, Davis and Cohen collected and analyzed two sets of student term papers from different years and found that the number of citations students used had not changed over the three years. Additionally, they found that students were using the same amount of citations, fewer books and scholarly sources, and more Web pages. In the end, Davis and Cohen conclude that a more prescriptive approach to student paper requirements, where students must use a certain number of scholarly sources, is needed. When Davis replicated the study, with added emphasis in the form of verbal guidelines from the course instructor to include scholarly sources, he found little change in the types of resources students used. However, when the instructor required the use of scholarly sources, Davis found that students significantly increased their use of journals and scholarly sources. Andrew Robinson and Karen Schlegl tested and confirmed Davis findings that instruction alone has little effect on student use of resources. 10 In this study, the researchers analyzed the bibliographies of three groups of students from a second-year politics course: a control group of students who did not receive any instruction; a second group who received library instruction and encouragement from their instructor to use scholarly sources; and a third group who received the same instruction, plus written requirements to use a minimum number of scholarly sources. In their analysis, Robinson and Schlegl found that students who received instruction, plus a penalty for not including scholarly sources, had higher quality citations than the other two groups of students. Specifically, they found that with the added penalty students used more scholarly sources and fewer Web resources. Sarah Clark and Susan Chinburg used citation analysis to research and compare the impact of an embedded librarian program with traditional face-to-face instruction for a group of students enrolled in an upper-division MIS course. 11 Some students received librarian support in the form of a one-shot information literacy session, while others received support from a librarian embedded in the course management system who provided students with an online tutorial, question and answer sessions, and links to important resources. The researchers found that there was no significant difference between the two groups of students in terms of the types of resources they used in their papers. In 2006, Rui Wang conducted a study that assessed the long-term impact of a library credit course by analyzing students citations. 12 In this study, Wang tracked students who had previously taken an elective credit course and compared them to students who had not taken the library course. Wang collected student papers and analyzed the citations from both groups of students. He found that students who had taken the course cited more scholarly sources, wrote more complete citations, and received higher grades on both their papers and in the course. We have used some of Wang s criteria in our study. Although citation analysis studies can tell us what information resources students are able to locate, several studies reveal their limitations. Most of the citation analysis studies mentioned previously do not address an assessment of higher order informa-

4 286 Performance-based Assessment in an Online Course tion literacy skills. That is, they tell us what specific sources students found, but not how those students used their sources. However, a number of researchers have begun to address this limitation. In their research study, David Scharf et al. examine the research papers of graduating seniors for evidence of several higher order information literacy skills. 13 The authors found that although students had received satisfactory scores on the writing model of assessment, these same students did not receive similar scores on the information literacy assessment for the same research papers. As the authors point out, the more complex the skill, the lower the score; students scored highest on the variable of formatting a citation and lowest on the variable of using information effectively in their papers. More recently, Stephanie Rosenblatt also addresses the important limitations of citation analysis studies. 14 Rosenblatt used citation analysis to investigate the effectiveness of a library session on two different sets of students: one group of upper-division undergraduate students who received library instruction and another group that did not receive direct instruction. Rosenblatt also found that although students could locate quality sources, they were not necessarily using them critically. Half of the students were not able to use the sources in a manner that showed synthesis of the material. Rosenblatt s conclusions are important in the citation analysis literature as most studies look at citations in isolation of student use, which only allows for a superficial analysis of students information literacy skills. In order to assess the true information literacy skills of students, more in-depth studies such as the Rosenblatt study need to be carried out. Methodology The authors analyzed the bibliographies from a paper assigned to English 102 students. This assignment is required of all English 102 students regardless of the type of IL instruction they receive. The typical English 102 student tends to be a first-year student with little to no experience using the university library for research. The assignment asked students to analyze the different viewpoints of a current, controversial topic of their choice. Students were instructed to use information resources to explore their topic and cite those sources in a bibliography. Although the English 102 curriculum is standardized and all students are required to write this paper, it should be noted that there is a slight variation in the requirements among the different instructors in terms of types and number of sources that students are required to use. A random selection of 156 student papers was collected from class sections with three different types of IL instruction. Only the bibliographies of the papers were reviewed. The three groups of student papers collected were as follows: Group 1: Student papers from sections that received information literacy instruction from their English instructors. These instructors had been previously trained by librarians on the use of library resources and how to use different databases with students. Instructors are also aware of and use the English Composition Subject Guide with their students. Instructors vary on how they approach the topic and which information literacy topics they cover in their classes. (48 students) Group 2: Student papers from sections that received information literacy instruction from a librarian during a face-to-face, fifty-minute session. These sessions were all given by

5 Yvonne Mery, Jill Newby, Ke Peng 287 the same librarian and covered Boolean searching, an introduction to a general database, and narrowing topics. (48 students) Group 3: Student papers from students enrolled in the ORL. The ORL is a credit online course offered to students enrolled in English 102. This is a ten week course that covers the basics of information literacy from formulating a research question, to using databases, to documenting sources. Mery et al. offers a detailed description of the course and its development. 15 (60 students) The authors developed a set of criteria based on the Wang study to evaluate the quality of the bibliographies. 16 Student bibliographies were analyzed for number of citations, variety of resources, type of resource, citation completion, and currency. We included number of citations and variety of resources as criteria because the use of more resources generates a more well-rounded argument with for number of citations, variety of more diverse perspectives. Using a resources, type of resource, citation variety of sources is also indicative completion, and currency. of students ability to locate and access different types of resources. The criterion complete citations was included since this is a measurement of students knowledge of citation creation and use. The completeness of citations were judged according to Gloriana St. Clair and Rose Mary Magrill s definition of completeness as including an author, title, publisher, and date for a book, and author, title, journal title, date, and page numbers for journals. 17 However, unlike Wang, who considered a Web source citation as incomplete if it did not contain a URL, we counted Web sources as complete without the inclusion of a URL because the current edition of the MLA Handbook does not call for the use of URLs. 18 Since the students needed to write on a current, controversial topic, we also included a criterion for currency of sources. Much of the focus in the ORL is about getting students away from using popular sources and Web pages and using more sophisticated scholarly sources. Thus, we included criteria for the use of reliable sources and the number of Web pages used. Reliable sources included scholarly journals, books, and government documents. Based on these criteria, the following research questions were developed: Student bibliographies were analyzed 1. Did the method of instruction affect the number of citations in student bibliographies? 2. Did the method of instruction affect the completeness of citations in student bibliographies? 3. Did the method of instruction affect the currency of citations in student bibliographies? 4. Did the method of instruction affect the variety of sources in student bibliographies? 5. Did the method of instruction affect the percentage of reliable sources in student bibliographies? 6. Did the method of instruction affect the percentage of Web sources in student bibliographies?

6 288 Performance-based Assessment in an Online Course The authors developed a scoring sheet to analyze the student bibliographies. Eight instruction librarians reviewed student bibliographies after receiving a training session on using the scoring sheet with sample bibliographies to norm the evaluations. After the bibliographies had been evaluated and scored, the lead author and another instruction librarian reviewed those papers for which there was a disagreement between raters. Appendix 1 shows the scoring sheet that was used by each librarian. Results Descriptive statistics. As Table 1 shows, Group 3 (M = 7.08, SD =2.87) outperformed their peers in Group 1 (M= 5.15, SD=2.98) and Group 2 (M= 4.94, SD= 3.30) in the average number of citations they used for their papers. That is to say, each member in Group 3 included an average of 7+ citations for their annotated bibliography, while their counterparts only had about 5 citations. Additionally, students in Group 3 also provided more complete citations (M = 6.65, SD =2.81) than those in Group 1 (M = 4.67, SD =2.94) and Group 2 (M = 4.70, SD =3.27). The same pattern was also found in the average use of current sources, variety of sources, and reliable sources (See Appendix B-D for more details). In short, simply based on the mean scores, students enrolled in the online course were able to provide higher quality annotated bibliographies as evaluated by the IL competency standards. In order to understand the statistical significance of the numbers, correlation and regression analysis was also conducted. Correlations among variables. Table 2 presents the inter-correlations among the variables in the study. The results suggest that the type of instruction (or treatment groups) was positively correlated with the number of citations, r(154) =.264, p =.001, the number of complete citations, r(154) =.273, p =.001, the currency of sources, r(154) =.208, p =.009, and the variety of sources, r(154) =.214, p =.007. However, we failed to locate any strong correlation between the type of instruction and the percentage of reliable sources used in students bibliographies. Regression analysis. Regression analysis was conducted to test if the type of instruction (or treatment groups) significantly predicted students final performance in their bibliographies. The dependent variables, or criteria variables, of the regression analysis in the present study refer to the six criteria of a quality bibliography, that is, the number of citations, the number of complete citations, the currency of sources, the variety of sources, the percentage of reliable sources, and the percentage of Web pages used. The independent variable, or predictor variable, was the type of instruction represented as the treatment groups. Table 3 provides an overview of the results from the regression analyses on the six criterion variables. Unfortunately, the type of instruction was not able to account for or predict students use of reliable sources or Web pages. For the other four variables, the regression was a poor fit, describing less than seven percent of the variance in the quality of students annotated bibliographies. However, the overall relationship was statistically significant between the predictor variable, namely, the type of instruction, and the other four variables. For instance, the results of the regression for the number of citations indicated that the predictor, namely the type of instruction or treatment groups, explained only a small portion (6.9 percent) of the variance, R 2 =. 063, F =11.497, p adj 1,154

7 Yvonne Mery, Jill Newby, Ke Peng 289 Table 1 Descriptive Statistics of Criterion Variables by Types of Instruction. Number of Complete Currency of Variety Reliable Web. Citations Citations the Source of Sources Sources Pages Group 1 IL Instruction via English Instructor (N=48) Mean % 30.44% Std. Deviation % 29.83% Group 2 IL Instruction via Librarian (N=48) Mean % 23.46% Std. Deviation % 26.11% Group 3 IL Instruction via ORL (N=60) Mean % 28.42% Std. Deviation % 23.07%

8 290 Performance-based Assessment in an Online Course Table 2 Correlations Among Citation Criteria and Instruction Type (N=156) Type of Instruction 2. Number of Citations.264 ** 3. Number of Completed Citations.273 **.959 ** 4. Currency of the Source.208 **.712 **.739 ** 5. Variety Of Sources.214 **.829 **.797 **.537 ** 6. Web Page **.296 ** * 7. Reliable Sources **.300 ** ** ** **. Correlation is significant at the 0.01 level (2-tailed). *. Correlation is significant at the 0.05 level (2-tailed). =.001, but the type of instruction significantly predicted the number of citations used in the bibliography, ß =.264, t(156) = 3.391, p =.001. That is, the type of IL instruction students received was a significant predictor of the number of citations they employed in their bibliographies. It can be reasonably argued that when students received the IL instruction from the ORL course, they used more citations in their papers. The results of the regression for the percentage of complete citations indicated that the type of instruction only accounted for a small portion (7.5 percent) of the variance, R 2 =. 069, F =12.327, but again the predictor significantly predicted the percentage of adj 1,154 complete citations used in the bibliography, ß =.273, t(156) = 3.511, p =.001. The results of the regression for the currency of the source indicated that the type of instruction was only able to explain a small portion (4.3 percent) of the variance, R 2 =. 037, F = adj 1,154 Similarly, the type of instruction significantly predicted the currency of the source used in the bibliography, ß =.208, t(156) = 2.636, p =.009. The same pattern of results was found for the variety of sources, that is, the type of instruction tended to reliably predict the variety of sources students used in their annotated bibliographies, but it was not able to account for the majority of the variances in the population sample. That is, the type of IL instruction students received was a significant predictor of the number of citations they employed in their bibliographies.

9 Yvonne Mery, Jill Newby, Ke Peng 291 Table 3 Regression Analysis of Types of Instruction and Citation Criteria Criterion Variables R 2 R 2 Adj.. ANOVA Results Coefficients Number of Citations F 1,154 =11.497, p =.001 ß =.264, t(156) = 3.391, p =.001 Percentage of Complete Citations F 1,154 =12.327, p =.001 ß =.273, t(156) = 3.511, p =.001 Currency of the Source F 1,154 = 6.947, p =.009 ß =.208, t(156) = 2.636, p =.009 Variety of Sources F 1,154 =7.407, p =.007 ß =.264, t(156) = 2.722, p =.007 Percentage of Reliable Sources F 1,154 =.048, p =.001 ß =.018, t(156) = -.218, p =.828 Percentage of Web Pages F 1,154 =.101, p =.858 ß =.048, t(156) = -.218, p =.828

10 292 Performance-based Assessment in an Online Course Discussion To achieve a greater predictability between types of instruction and the criterion variables, we conducted a regression analysis. Results showed that there was a predictive relationship between the type of instruction and the following criteria: number of citations, number of complete citations, currency, and the variety of sources. However, there was not a significant correlation or a predictive relationship with the type of instruction and the reliability of sources, as determined Results showed that there was a predictive relationship between the type of instruction and the following criteria: number of citations, number of complete citations, currency, and the variety of sources. by the percentage of reliable sources and the number of Web pages. We were somewhat disappointed by these results, as much of the ORL curriculum focuses on the use of reliable sources. However, the lack of reliable sources usage is partially due to the topics students chose and the instructors requirements. Many students chose current topics that do not lend themselves to scholarly literature. As mentioned earlier, instructors varied in their assignment requirements. Students may have chosen not to use more scholarly and reliable sources, not because they were not able to locate or identify them, but because their instructors did not require them to do so. While we provide a model for criteria to evaluate the quality of the bibliographies, the criterion variables will differ depending on the paper requirements. For example, in our study, the bibliography was based on a current controversial topic. In this case, currency is an important standard. However, for a paper in a History course, currency may not necessarily be a measure. Instead, the use of primary sources could be a highly relevant criterion. We chose not to investigate the relationship between the quality of students bibliographies and their assignment or course grade because of the additional variables that determine a student s final grade. Course and essay grades may be an important variable to include in future studies to correlate with the criterion variables, in order to investigate how different types of IL instruction impact students in areas other than IL skills. Conclusion As previously mentioned, authors of other citation analysis studies have pointed out that these studies do not reveal specific information literacy competencies that students should have, such as the ability to synthesize information. Our study here is no different. Although we are able to assess the impact of the ORL on student learning outcomes, without a reading of their papers we do not know if they can effectively use the information source they found. A citation analysis measures student performance in a more authentic manner than a multiple choice test can. However, given the amount of time and resources involved in a citation analysis study (gathering papers, creating criteria, norming evaluators, rating

11 Yvonne Mery, Jill Newby, Ke Peng 293 papers, inputting data, statistical analysis), we propose that using a valid and reliable multiple-choice test suffices. A multiple choice test can aid in the diagnosis of specific areas where students continue to struggle after instruction. For example, a previous analysis of students post-test scores showed that students continued to score low in the areas of using finding tools and documenting sources. 19 Evaluating the annotated bibliographies did not reveal such specific information. While either type of assessment could have been used to come to the same conclusion, it is important to note that both types of methods required the use of statistical analysis beyond descriptive statistics. Statistical analysis provides for a deeper understanding of the correlation among variables, such as different teaching methods In order for the library community to and specific dependent variable, in this case, the criteria for evaluating produce and evaluate valid and reliable the citations. The research study could not have been completed quantitative research studies, we need to have a greater understanding of sophisticated statistical analysis and be competent in using statistical analysis tools. without the expertise of one of the authors, a graduate student, who has knowledge and training in research design and statistical analysis. This is a skill gap for most academic librarians. In order for the library community to produce and evaluate valid and reliable quantitative research studies, we need to have a greater understanding of sophisticated statistical analysis and be competent in using statistical analysis tools. As mentioned in the methodology section above, there was some variation in the instruction students received across and within treatment groups. For example, some English instructors may have emphasized adherence to a specific citation style, while others may have focused the information literacy instruction on using a specific database. It would have been useful to observe the instructors and use student papers from instructors who had little variation in what they taught. Additionally, similar to Davis conclusions, having stricter guidelines for the types of resources students must use in their papers will force students to focus on using more reliable results as opposed to resources that are easier to come by. 20 It would have been useful to meet with the instructors as part of the creation of the research design to get their buy-in to establish the number and type of citations required. With statistical analysis of the citation study, we have been able to corroborate the findings from the pre- and post-test assessment. In our previous study, English 102 students enrolled in the ORL course performed better on their post-tests than the other two groups. 21 This leads us to the conclusion that the information literacy instruction students received from the online course will ultimately lead to higher quality bibliographies and better research skills than the learning gained by students in a single face-to-face library instruction session. Yvonne Mery is Assistant Librarian, Instructional Services Team, University of Arizona Libraries, meryy@u.library.arizona.edu; Jill Newby is Associate Librarian, Instructional Services Team, University of Arizona Libraries, newbyj@u.library.arizona.edu; Ke Peng is Assistant Professor, Western Kentucky University, ke.peng@wku.edu

12 294 Performance-based Assessment in an Online Course Appendix 1 Citation Analysis Scoring Sheet Paper ID: Rater: Number of Citations: Citation Type Complete Citation? Currency Citation #1 Citation #2 Citation #3 Citation #4 Citation #5 Citation #6 Citation #7 Citation #8 Citation #9 Citation #10 Citation #11** ** Additional citations can be placed on the backside Type A. Scholarly Journal (Article) B. Book C. Government Document D. CQ Researcher E. Reference Resources F. Magazine G. Newspaper H. Wikipedia I. Web page (that does not fit into any other category) J. Personal communication (not a blog) K. Video/Film L. Primary source of study M. Other explain Complete Citation Yes = Book: author, title, publisher, date Journal: author, title, journal, title, date Other Sources: Complete if with info. provided, you can locate the source No= you cannot locate the source with the info. Provided. URLS are not asked for by MLA. Page numbers not needed. Currency Enter the date of the resource. If no date available, write n.d.

13 Yvonne Mery, Jill Newby, Ke Peng 295 Appendix 2 Descriptive Statistics (Treatment Group 1: ENG102) N Min Max Mean Std. Deviation Number of Citations Complete Citations Year Year Year Before No Year Information Variety Of Sources Scholarly Journal % 80.00% 14.99% 22.95% Book % 60.00% 11.77% 16.71% Government Document % 25.00% 1.47% 5.16% CQ Researcher % 66.67% 3.68% 10.93% Reference Resources % 28.57% 1.91% 6.10% Magazine % 75.00% 9.15% 16.87% Newspaper % 57.14% 7.76% 13.50% Wikipedia % 0.00% 0.00% 0.00% Web page (that does not fit into any other category here) % % 30.44% 29.83% Personal Communication % 28.57% 0.60% 4.12% Video/ Film % 20.00% 1.13% 4.48% Primary Source of Study % 25.00% 0.75% 3.92% Other explain % 50.00% 1.77% 8.09% Reliable Sources % % 28.23% 30.31% Valid N (listwise) 47

14 296 Performance-based Assessment in an Online Course Appendix 3 Descriptive Statistics (Treatment Group 2: ENG102 with One-shot Librarian Session) N Min Max Mean Std. Deviation Number of Citations Complete Citations Year Year Year Before No Year Information Variety Of Sources Scholarly Journal % 83.33% 14.06% 22.17% Book % 42.86% 11.49% 13.87% Government Document % 33.33% 2.92% 7.83% CQ Researcher % % 11.01% 25.25% Reference Resources % 16.67% 1.03% 3.54% Magazine % 33.33% 5.44% 9.72% Newspaper % 66.67% 6.90% 15.86% Wikipedia % 50.00% 1.04% 7.22% Web page (that does not fit into any other category here) % % 23.46% 26.11% Personal Communication % 16.67% 0.65% 3.14% Video/ Film % 25.00% 0.52% 3.61% Primary Source of Study % 0.00% 0.00% 0.00% Other explain % 20.00% 1.18% 4.64% Reliable Sources % 83.33% 28.47% 28.09% Valid N (listwise) 46

15 Yvonne Mery, Jill Newby, Ke Peng 297 Appendix 4 Descriptive Statistics (Treatment Group 3: Online Research Lab) N Min Max Mean Std. Deviation Number of Citations Complete Citations Year Year Year Before No Year Information Variety Of Sources Scholarly Journal % 80.00% 15.38% 20.14% Book % 50.00% 6.93% 13.34% Government Document % 33.33% 4.85% 8.28% CQ Researcher % 16.67%.94% 3.58% Reference Resources % 16.67% 1.26% 3.88% Magazine % 50.00% 7.12% 12.72% Newspaper % % 22.11% 25.18% Wikipedia % 10.00%.17% 1.29% Web page (that does not fit into any other category here) % 81.82% 28.42% 23.07% Personal Communication % 71.43% 3.59% 10.62% Video/ Film % 28.57% 2.28% 6.23% Primary Source of Study % 0.00% 0.00% 0.00% Other explain % 25.00% 1.62% 5.16% Reliable Sources % 80.00% 27.16% 21.86% Valid N (listwise) 59

16 298 Performance-based Assessment in an Online Course Notes 1. Characteristics of Programs of Information Literacy that Illustrate Best Practices: A Guideline, American Library Association, (2012), characteristics (accessed April 2, 2012). 2. Leslie Sult and Vicki Mills, A Blended Method for Integrating Information Literacy Instruction into English Composition Classes, Reference Services Review 34, 3 (2006): Yvonne Mery et al., Developing an Online Credit IL Course for a Freshman Writing Program in a University Setting, in Best Practices for Credit-Bearing Information Literacy Courses, ed. Christopher V. Hollister (Chicago: Association of College and Research Libraries, 2010) Yvonne Mery, Jill Newby, and Ke Peng, Assessing the Reliability and Validity of Locally Developed Information Literacy Test Items, Reference Services Review 39,1 (2011): Yvonne Mery, Jill Newby, and Ke Peng, Why One-shot Information Literacy Sessions are Not the Future of Instruction: A Case for Online Credit Courses, College and Research Libraries, preprint, (accessed October 7, 2011). 6. Megan Oakleaf, Dangers and Opportunities: A Conceptual Map of Information Literacy Assessment Approaches, portal: Libraries and the Academy 8, 3 (2008): Karen Hovde, Check the Citation: Library Instruction and Student Paper Bibliographies, Research Strategies 17, 1 (2000): Sarah Clark and Susan Chinburg, Research Performance in Undergraduates Receiving Face to Face versus Online Library Instruction: A Citation Analysis, Journal of Library Administration 50, 5/6 (2010): Philip M. Davis and Suzanne A. Cohen, The Effect of the Web on Undergraduate Citation Behavior, , Journal of the American Society for Information Science and Technology 52, 4 (2001): ; Philip M. Davis, Effect of the Web on Undergraduate Citation Behavior: Guiding Student Scholarship in a Networked Age, portal: Libraries and the Academy 3, 1 (2003): Andrew M. Robinson and Karen Schlegl, Student Bibliographies Improve When Professors Provide Enforceable Guidelines for Citations, portal: Libraries and the Academy 4, 2 (2004): Clark and Chinburg, Research Performance, Rui Wang, The Lasting Impact of a Library Credit Course, portal: Libraries and the Academy 6, 1 (2006): David Scharf et al., Direct Assessment of Information Literacy using Writing Portfolios, Journal of Academic Librarianship 33, 4 (2007): Stephanie Rosenblatt, They Can Find It, but They Don t Know What To Do With It: Describing the Use Of Scholarly Literature by Undergraduate Students, Journal of Information Literacy 4, 2 (2010): Mery et. al., Developing an Online Credit IL Course, Wang, Lasting Impact, Gloriana St. Clair and Rose Mary Magrill, Incomplete Citations in Undergraduate Term Papers from Four Campuses, RQ 30, 1 (1990): Modern Language Association, MLA Handbook for Writers of Research Papers, 7 th ed. (New York: Modern Language Association of America, 2009). 19. Mery, Newby, and Peng, Why One-shot Information. 20. Davis, Effect of the Web, Mery, Newby, and Peng, Why One-shot Information.