Name: Toby Leigh Matoush Campus Phone:

Transcription

1 San Jose State University CSU Information Literacy Grant Progress Report CSU Information Literacy Grant Progress Report Developing and Assessing a Freshman Library Program at San Jose State University: the Metropolitan University Scholar Experience Name: Toby Leigh Matoush Campus Phone: Department: Reference Department Dr. Martin Luther King Jr., Library Department Chair: Jo Bell Whitlatch Library Dean: Ruth Kifer Campus Address: San Jose State University One Washington Square Dr. Martin Luther King Jr. Library San Jose, CA tmatoush@sjsu.edu I. Objectives & Abstract This information literacy assessment project seeks to complete five objectives: 1) Complete a two-year longitudinal (three-year if funds are available) study comparing the research and technology skills of freshman students in the Metropolitan University Scholar Experience (MUSE) Freshman Experience Program at San Jose State University (SJSU) with a group of students not participating in the MUSE program utilizing ETS ICT data, 2) Use the ETS ICT test as a pre-test Fall 2006 to determine needed areas of work in information and technology literacy for students in the MUSE program, 3) Utilizing the 2006 pre-test data, develop information literacy learning objectives for the MUSE program based on the ACRL Information Competency Requirements for Higher Education, 4) Develop information literacy outcomes and an information literacy competency rubric to be used in the MUSE library program which builds on the MUSE university scholar competency rubric already developed and currently being utilized, and 5) Conduct a MUSE faculty workshop with faculty and librarians which incorporates information literacy objectives and outcomes into the MUSE syllabi. II. Activities Completed I. Activities completed to-date include the following: 1) Utilized the ETS iskills to test approximately 600 students between Fall 06 and Spring 07. Of the 162 freshman tested, only 59 were MUSE students. We were not able to meet the goal of testing 100 MUSE students. However we did meet the goal of testing 100 non-muse students. These freshmen were taken primarily from English 1B composition classes. English 1B is a required English class for SJSU students and is generally taken

2 San Jose State University CSU Information Literacy Grant Progress Report at the freshman level but not in the first semester of school. MUSE is taken in the first semester of school at SJSU and does not include transfer students. We will increase efforts to test MUSE freshman Fall ) Utilized the ETS iskills test as a post-test (rather than as a pre-test) after the library session for most test periods. Was not able to use the test as both a pre-test and post-test due to time constraints and the fact that faculty did not wish to give away 3 class periods for an iskills pre-test, library session, and iskills post-test. Will attempt to do pre and post testing in Fall 2007 with a small group of faculty (5-10) but am not confident that faculty will be willing to release 3 class sessions. 3) Based on the data gleaned from the iskills test analysis, determined needed areas of work in information literacy for students in the MUSE program. Planning to focus on those areas for Fall instruction. 4) Developed information literacy objectives for the MUSE program based on the freshman iskills data. (See Appendix 1: MUSE Library Learning Objectives). These objectives will be analyzed and revised as needed in Spring 2008 based on the data gleaned from Fall 2007 MUSE testing. 4) A preliminary analysis of the data has been completed. (See Part IV and Part V) III. Adjustments to the original timeline: March-April 2007: Develop information literacy competency rubric for the MUSE program-will complete by March 2008 September 2007: Host workshop with MUSE faculty and SJSU librarians which will accomplish the goal of incorporating specific information literacy objectives into the MUSE syllabi utilizing the MUSE information literacy objectives crafted by librarians. MUSE faculty will also have an opportunity to craft discipline-specific library information literacy objectives for their classes. Your Students Compared With Reference Group This workshop will be set for January 2008 during the annual MUSE faculty 2-day orientation. The campus MUSE Director is setting aside an afternoon for this activity. IV. ETS iskills Report Analysis Methods: There were 3 groups of faculty targeted for testing: 1) Faculty who had participated previously in the Spring 06 testing, 2) MUSE Faculty, and 3) English 1B faculty. Convenience testing has worked in the past for SJSU testing and was used again for testing. For the freshman classes, a total of 4 MUSE faculty and 2 English 1B faculty (5 classes) agreed to participate in testing. Population:

3 San Jose State University CSU Information Literacy Grant Progress Report All students, including freshman were given the Advanced version of the iskills Test. The valid number (usable test results) of students tested were Freshman (151), Senior (150), and Junior (187) and slightly lower for Sophomore (91). The total number of students tested was 605 with a valid number of 579. Test Analysis: Skill Area Report: SJSU Institutional Skill Area Report freshman sophomore junior senior Define Access Evaluate Manage Integrate Create Communicate The biggest surprise for the Skill Area Report was the high scores of SJSU Freshman in all skill areas compared to the scores of SJSU seniors except for the skill area of Define. Freshman scored higher than Sophomores, Juniors, and Seniors for Access, Evaluate, Integrate, and Create and scored the same as Sophomores for Manage, and the same for Sophomores and Juniors for Communicate. This data has important implications for the SJSU Library Information Literacy Program. In addition, SJSU Freshman scored equal to the ETS ICT Literacy Reference Group for all areas except Manage and Create. In the areas of Manage and Create they scored above the ETS ICT Literacy Reference Group. (See SJSU Freshman Institutional Skill Area Report, Page 5) SJSU Freshman Aggregate Task Performance Report: Lowest Scoring Information Literacy Tasks (Lowest 5) This chart represents the information literacy tasks that students scored most poorly on. I have included only the information literacy areas and related areas (Create Data Display is somewhat information resource based) from the ICT literacy areas since the library program does not include instruction of technology skills. This score report was less helpful than expected for information literacy instruction improvement since the majority of the skill areas rated tasks that were more technology-based (for example, managing folders under the Manage Skill Area) than information literacy based.

4 San Jose State University CSU Information Literacy Grant Progress Report Information Literacy Areas DEFINE SKILL AREA Task Feedback % of SJSU students who provided highest scoring response Choose a research topic according to specific criteria and explain your choice You correctly reported the criteria fulfilled by the research topic selected. EVALUATE SKILL AREA Task Feedback % of SJSU students who provided highest scoring response Judge the usefulness of Web pages and article abstracts (Step 4) You selected the two best abstracts from the article database EVALUATE SKILL AREA Task Feedback % of SJSU students who provided highest scoring response Judge the usefulness of Web pages and article abstracts (Step 1) You visited only promising Web pages CREATE SKILL AREA Task Feedback % of SJSU students who provided highest scoring response Create a data display (Creating a Graph: Movie Trends) (Step 3) You created the data display very efficiently CREATE SKILL AREA Task Feedback % of SJSU students who provided highest scoring response % of reference group who provided highest scoring response 9% 9% % of reference group who provided highest scoring response 15% 16% % of reference group who provided highest scoring response 17% 17% % of reference group who provided highest scoring response 15% 19% % of reference group who provided highest scoring response Create a data display (Creating a Graph: Movie Trends) (Step 1) You organized the layout of the data display logically and effectively 19% 22%

5 San Jose State University CSU Information Literacy Grant Progress Report SJSU Freshman Institutional Skill Area Report SJSU Freshman scored equal to the ETS ICT Literacy Reference Group for all areas except Manage, Create, and Communicate. In the areas of Manage, Create, and Communicate they scored slightly above the ETS ICT Literacy Reference Group. This finding shows that SJSU freshman do at least as well as the reference group; future instruction could focus on developing the skill areas of Define, Access, Evaluate, and Integrate so SJSU freshman score higher than average. SYour Students Compared With Reference Group Your Students Compared With Reference Group Your Students Compared With Reference Group

6 San Jose State University CSU Information Literacy Grant Progress Report SJSU Senior Institutional Skill Area Report SJSU Seniors scored lower than the ETS ICT Literacy Reference Group in all areas except for the skill area of Define. For this area, they scored equal to the ETS ICT Literacy Reference Group. This finding has important implications for library instruction at the 100W level. 100W is an upper-division required writing course in most majors which has a built-in library instruction component. Students generally take 100W at the junior or senior level.

7 San Jose State University CSU Information Literacy Grant Progress Report MUSE Freshman and English 1B Freshman Institutional Skill Area Report As part of the grant, I was interested in whether freshman in MUSE classes scored higher than freshman not taking MUSE classes. I used English 1B classes as my non- MUSE group of freshman. English 1B is a required English class for SJSU students and is generally taken at the freshman level but not in the first semester of school. English 1B requires completion of the information literacy tutorial InfoPOWER and a library session. MUSE is taken in the first semester of school at SJSU and does not include transfer students. The scores for all skill areas for freshman were fairly uniform between English 1B freshman and MUSE freshman with only slight differences. English 1B freshman scored slightly higher in the skill areas of Access and Communicate while the MUSE freshman scored only slightly higher in the skill area of Evaluate. In Fall 07, I will attempt to test several English 1A (a required English class taken before English 1B) classes which do not have a required library session in order to compare their score to MUSE students. Freshman Institutional Skill Area Report English 1b MUSE Define Access Evaluate Manage Integrate Create Communicate MUSE ICT Score One of the additional data questions added to the demographic section of the SJSU iskills test was the question Have you ever had a MUSE class? The students who had taken a MUSE class scored only slightly higher than students who had not taken a MUSE class; the difference was not statistically significant. 85%- 90% of MUSE sessions have a library instruction session.

8 San Jose State University CSU Information Literacy Grant Progress Report MUSE ICT Score ICT Score MUSE Class yes no Have you ever taken a MUSE class? Library Session ICT Score Another additional SJSU demographic question was Have you ever attended a library session at SJSU? Students who had had a library session at SJSU scored slightly higher than those who had not. However, once again the difference was not statistically significant. This chart shows that attending a library session did not make a difference in ICT scores. More analysis of this question will be done in the future. Library Session ICT Score ICT Score no yes Have you ever attended a library session at SJSU? V. ETS iskills SPSS Data Analysis The test was administered to approximately 600 students between October 2006 and April The following is a preliminary analysis of the data. Data analysis was performed in SPSS with the assistance of a colleague.

9 San Jose State University CSU Information Literacy Grant Progress Report Overall Performance Test scores ranged between 455 and 625. The average score for students taking the test during this time period is Performance by Class Status Surprisingly, freshman scored higher overall with a score of 554 than juniors and seniorsboth 548. The range of scores for freshman was also greater ( ) than for juniors or seniors. This could be because some freshmen are entering college with more highly developed technology skills. Clearly, more work needs to be done analyzing why freshman scored higher than juniors or seniors. Not surprisingly, grads scored much higher with 580 than any other class. Grade N Mean Std. Deviation 95% Confidence Interval for Mean Minim um Maximum Upper Lower Bound Bound 10th grade grade Freshman Sophomore Junior Senior Grad Other Total Performance by Gender Males scored higher overall than female students. There was also a greater spread of scores for males. GENDER N Mean Std. Deviation Std. Error Mean REPORTED Female SCORE Male Performance by Race Whites scored highest with a mean of 564 with Asians scored second highest with a mean of 545. Mexicans, Puerto Ricans, and Other Hispanic were all added to one group. REPORTED SCORE N Mean Std. Deviation Std. Error Minimum Maximum Native

10 San Jose State University CSU Information Literacy Grant Progress Report American Asian Black Mexican Puerto Rican Other Hispanic White Other Total Performance by Native/Transfer Native students scored higher with a mean of 552 than transfer students with a mean of 547. REPORTED SCORE N Mean Std. Deviation Std. Error 95% Confidence Interval for Mean Minimum Maximum Lower Bound Upper Bound not enroll Native Transfer Total Performance by Major There was not enough numbers in the majors groups except for business to get statistically valid means. However, the major chart is included anyway. The iskills test still lacks an efficient means to gather major data. Only some majors choices are available. A number of justice studies took the iskills test in Spring 2007 but since there is no major for Justice Studies in the iskills major choice, these students are identified as Other. SJSU suggested that a write-in major be added to the test but that hasn t solved the problem of unclear or unidentifiable majors. REPORTED SCORE N Mean Std. Deviation Std. Error Minimum Maximum Agriculture or Natural Resources Architecture or Environmental Design Arts: Visual or Performing Biological Sciences Business

11 San Jose State University CSU Information Literacy Grant Progress Report Communications Computer or Information Science Education Engineering English Language and Literature Foreign or Classical Languages Health Sciences Library Sciences Mathematics Military Science Philosophy, Religion, or Theology Physical Sciences Public Affairs and Services Social Sciences and History Technical and Vocational Other Undecided Total Performance by Language Not surprisingly, students with English only as their language scored higher than both students with English and another language and students with another language only. REPORTED SCORE N Mean Std. Deviation Std. Error Minimum Maximum English only English and another another language only Total

12 San Jose State University CSU Information Literacy Grant Progress Report VI. Grant Assessment After the freshman program is revised based on the iskills data, assessment of the grant project will be conducted in the following ways. 1) The skills of MUSE freshman will be evaluated using the iskills test and 2006 freshman compared to 2007 freshman. 2) Students will complete a formative evaluation form at the end of the MUSE class session in Fall 2007 based on the MUSE library learning objectives. 3) The information and technology literacy of sophomores who have taken MUSE classes and sophomores who have not taken MUSE classes will be evaluated using the iskills test.

13 San Jose State University CSU Information Literacy Grant Progress Report Appendix One MUSE Library Learning Objectives 1) Define and refine a research topic using keywords 2) Recognize the difference between scholarly and popular sources 3) Search library catalog using keyword or subject search as appropriate 4) Search an appropriate database by keyword or subject search 5) Create effective searches using AND, OR or NOT 6) Locate full-text of an article from a citation 7) Recognize parts of a citation 8) Evaluate information sources for relevance, authority, and currency 9) Identify the role of the library on campus and the services it provides 10) Locate and use the SJSU Research Topic Web Pages