Pop-Up Quiz: Using Course Management Software for Library Instruction Assessment

Transcription

1 Pop-Up Quiz: Using Course Management Software for Library Instruction Assessment Library Assessment Conference University of Washington August 4, 2008 Diana Hartle, University of Georgia Deb Raftus, University of Washington Amy Watts, University of Georgia

2 Abstract From the annual library instruction statistics collected for ARL, UGA librarians could see that they were reaching an impressive number of students and classes with library instruction. These demographic statistics gave a clear picture of the reach of UGA s library instruction program, but they weren t able to show how effective librarians were in teaching information literacy skills in the classroom. While some evaluation methods were used by individual librarians, there was no Libraries-wide strategy for assessing library instruction. In an effort to measure the effectiveness of the instruction program, Reference department librarians created an online instruction quiz. This instrument is designed to accurately reflect information literacy skills acquired during library instruction sessions by evaluating student learning outcomes. Learning outcomes are defined by ACRL standards for information literacy and local concerns. The instrument needed to be uniform, easy to administer, and provide manageable data collection capabilities for analysis of results. Therefore, UGA Libraries developed the quiz for students in introductory level classes across the curriculum. The quiz is distributed via WebCT, UGA s campus-wide course management system.

3 What s this quiz all about, anyway? Who or what is being assessed? Information literacy and library skills of students in introductory level classes; librarians who teach those sessions. Why is the assessment happening? Directive from library administration to report more than just attendance numbers. Request for findings about learning outcomes in information literacy. What to do with the findings? Demonstrate effectiveness and importance of library instruction; improve teaching in these sessions.

4 Why Use WebCT (Course Management Software)? Student familiarity with interface Availability Technical and administrative support Data collection possibilities Course integration

5 How does your quiz grow? Initiate collaborative development of quiz Determine learning outcomes to be measured Choose type of questions for quiz Select other data to be collected

6 Collaboration Collaboration is not just working with your local colleagues; it can also refer to collaborating with the collective knowledge, past and present of your colleagues worldwide. Do a literature search in library literature. Web search for other libraries initiatives Trawl the listservs if you ve already searched the archives and nothing s there, ask! Consult colleagues at your own library and other libraries.

7 Whaddya know? Learning outcomes were defined by: ACRL standards for Information Literacy Standard 2 The information literate student accesses needed information effectively and efficiently. Outcomes include: Identifies keywords, synonyms, and related terms for the information needed Local concerns

8 How s the skillz? Research strategies Moving from a question to a search strategy Narrowing/broadening topic Identifying keywords, synonyms Truncation and wildcards Boolean operators Choosing which database/resource to search Revising search based on results

9 Ya got more skillz? Locating full text of a journal article using a: Direct full-text link in a database Link to another database (FindIt@UGA) Electronic Journal Locator Library OPAC for print Using citation information and OPAC record to identify and locate specific item needed

10 What are the ingredients in your quiz? Present questions as multiple choice - easier to grade. Deploy graphics that students should recognize from using library tools. Give students the option to provide evaluative comments about the instruction session. Keep it short enough that students won t hate you.

11 What s your sign, babe? This is your chance to collect demographic information about your quiz participants. Do you want to know: Their year of study The number of library instruction sessions they ve attended previously (if any) Their field of study Gender Native language

12 How do you play nice? Consult with instructor of record for class regarding administration of quiz Explain to students why you re giving the quiz...respect their time and effort. Mention to students that quiz is not being graded. Explain that quiz is to grade the librarian, not them. Thank the students for their participation and feedback.

13 What now? Once your students have taken the quiz, it s not the end, just the beginning. Use results to: Revise questions for future iterations of quiz. Modify content/method of instruction sessions to improve comprehension of desired learning outcomes. Prove the worth of your library instruction program. Provide a synopsis to your department head. Use as an outreach tool to new faculty.

14 How can we make it better? Using discrimination factor, determine if your question is measuring learning outcome or student test-taking ability. Toughen up any softball questions. Remove any questions that aren t providing meaningful statistics. Update any outdated terminology and/or graphics.

15 How should we teach now? Modify content of sessions: To cover areas in which students scored poorly. To address areas students identified as still confusing. To de-emphasize content that students expressed was repetitive or already known.

16 How ya like us now? Numbers from quiz results can be used to supplement standard ARL statistics. Evaluative comments, particularly those praising the library and its staff, can be submitted to higher ups to justify instruction program and other service efforts. A synopsis of student success with learning outcomes may be useful to instruction program coordinators in determining future efforts.

17 Do you want to buy what we re selling? Demonstrate to faculty student need for library instruction, particularly with comments regarding what they learned in a session. Identify services and resources that need better marketing.

18 How exactly did you do that? Quiz questions Results of graded questions Summary of evaluative comments Other findings

19 Heeeere s the quiz!

20

21

22

23 How did they do? Snapshot for Fall ,029 students took the quiz. Overall score was 81.5% correct. Spring students took the quiz 83.6% was the average score

24 How did they do? Results: Graded Questions, Fall 2006 Question Mean Discrimination #1 94.3% 0.48 #2 81.6% 0.56 #3 92.6% 0.52 #4 91.5% 0.48 #5 43.3% #6 71.7% 0.48 #7 95.7% 0.51

25 How did they do? Results: Graded questions, Spring 2007 Question Mean Discrimination #1 92.8% 0.67 #2 84.3% 0.61 #3 92.1% 0.68 #4 91.0% 0.71 #5 54.2% 0.54 #6 76.6% 0.58 #7 94.4% 0.65

26 What did they learn? 25.00% 20.00% 15.00% 10.00% 5.00% 0.00% Truncation / Wildcards Boolean operators Forming / inputting / refining searches GIL GALILEO (including finding password) Identifying / Choosing / Using Specific Databases Unclear / vague response

27 What did they say they learned? I learned so much it was crazy. I didn't know the best way to use GALILEO. Now I know how to use it quickly and efficiently in order to get the results I want. I found out how to use Galileo, which I will now use a lot because of its credibility. I also did not know the availability of help by the librarians. That it is more important how you search than what you search. Being more specific with searches greatly affects how much time you have to put into a search.

28 What did they say they learned? I learned so much it was crazy. I didn't know the best way to use GALILEO. Now I know how to use it quickly and efficiently in order to get the results I want. I found out how to use Galileo, which I will now use a lot because of its credibility. I also did not know the availability of help by the librarians. That it is more important how you search than what you search. Being more specific with searches greatly affects how much time you have to put into a search.

29 80.00% What are they Still confused about? 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% Research strategies Locating full text Confused, other Need more practice Unclear/vague response Praise for librarian Not confused

30 What s still confusing (in their words)? Why we weren't told about this earlier. Ask me when I'm doing it on my own. I feel like I have it now, but that doesn't mean I can do it when I don't have someone leading me through it. I am overwhelmed with the system and am not sure how to find the article that I need. Not so much as confused, but a little overwhelmed. I like the fact that the instructor used an example to lead us through the search. I just think that it will take personal practice to really understand how to use the library. I think it would be helpful to have us work on something outside of class immediately to see if we have any questions and make sure we can do it on our own. I am not really confused about the search engines but overwhelmed because of the amount of information given to me in a little amount of time. Why I had to sit through an hour lecture... when only 30 minutes was necessary.

31 Any comments that made you laugh (or cry)? What I learned: That we actually have librarians! Well, first off, I finally learned where the library was. Why everyone thinks Galileo is so great. I never have needed to use it. All the articles I need are available any time online through the UGA Libraries homepage. I reviewed library skills I learned in 9th grade. What s still confusing: Nothing; this guy made it perfectly clear. He is amazing. He should get a raise and take over President Adams job. Why would there be a physical subscription to a journal but not an electronic when generally speaking the electronic is or should be cheaper... Nothing! It was brilliant...brilliant!

32 Do students value library instruction? Do you think this library instruction session will help you with your research? Definitely! Yes Maybe No No answer

33 Questions or comments? Please contact Amy Watts and Diana Hartle at the University of Georgia Libraries: