Studies of Student Learning Classroom-Based Pre-/Post Testing Audrey Pereira, Assistant Professor Business Administration Third Annual Winter Assessment Day January 19, 2010
Session Objective Review pre-/post testing process used to determine computer literacy of Business Administration students, including analysis of results
Classroom-Based Pre-/Post Computer Literacy Testing Assess Computer Literacy Pre and Post BSAD 1700 Introduction to Computer Information Systems Measure Student Computer Literacy from the time students begin class until completion
Computer Literacy Pre-/Post Testing 8 Step Process Define Computer Literacy Determine Components to be included on Pre-/Post Tests Map components to Course Objectives Revisions needed? Yes Determine how literacy & learning will be assessed, tracked, scored, & analyzed Develop assessment and tracking tools Implementation (testing, instruction, learning) Analyze results Modification & process fine tuning
Step One: define computer literacy Majority of college and post-secondary students are already familiar with computer, the Internet & WWW Don t need rudimentary instructions on doubleclicking, resizing windows, etc.
Step One: define computer literacy Need to be taught to be confident, in-control users of IT Navigate independently in ever-changing worlds of information and technology Solve problems on own Apply power of IT tools for personal, education, & career goals
Step One: define computer literacy Computer Literacy can be broken into 3 types of knowledge (1) Skills: proficiency with contemporary computer applications Concepts: fundamental knowledge underpinning IT in order to build new understanding as IT evolves Capabilities: higher-level thinking processes, such as problem solving, reasoning, troubleshooting (1) Synder, L. (2008) Fluency with information technology: Skills, Concepts, & Capabilities (3ed). Boston: Addison Wesley
Step One: define computer literacy General understanding of information technology (Concepts & Capabilities) Microsoft Office 2007 skills (Skills)
Step Two: determine specific concepts, capabilities, & skills Determine specific general information technology concepts and capabilities required for computer literacy & to be included in pre-/post testing Determine specific Microsoft Office 2007 skills required for computer literacy & to be included in pre-/post testing
Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) Map to course objectives (learning outcomes) Verify support goal Adjust if required
Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) General IT Concepts & Capabilities Course Objectives To understand basic computer information technology hardware and software terminology To comprehend the fundamental patterns on which all information technology is built To be familiar with Web site organization, how search engines work, and how to evaluate the accuracy of Web content To become familiar with the components and implications of graphical user interfaces within information technology To explore basic computer software and hardware debugging strategies and workarounds. To have an understanding of basic privacy and digital security concepts and consider their implications To recognize and discuss ethical issues and the impact of computers on society
Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) Map specific general IT concepts & capabilities course objectives Course Objective To understand basic computer information technology hardware and software terminology To comprehend the fundamental patterns on which all information technology is built To be familiar with Web site organization, how search engines work, and how to evaluate the accuracy of Web content Specific concepts to be included in Pre- /Post testing Why important to know the correct terminology Define basic terminology including: Software vs Hardware Different monitor displays RGB Processor Bits and Bytes Link between patterns, symbols, and information PandA encodings using physical phenomenon Binary systems ASCII Meta data Digital vs. analog To be familiar with Web site organization, how search engines work, and how to evaluate the accuracy of Web content
Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) Specific general IT concepts & capabilities mapped to course objectives Course Objective To have an understanding of basic privacy and digital security concepts and consider their implications To recognize and discuss ethical issues and the impact of computers on society To become familiar with the components and implications of graphical user interfaces within information technology Specific concepts to be included in Pre- /Post testing Privacy practices in United States versus the European Union Identity theft Privacy principles and uses, including no uses, approval, objection, and no limits Netiquette Computer viruses, worms, and other malware Permitted vs. not permitted uses of software Software copyrights Analogies Strategy to learn a new software packages Advantages of consistent GUIs To explore basic computer software and hardware debugging strategies and workarounds Basic strategy for debugging Work around
Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) Microsoft Office 2007 skills course objectives To demonstrate fundamental software application skills in MS Word, Excel, PowerPoint and Access in support of basic business applications
Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) Specific Microsoft Office 2007 skills mapped to course objectives Course Objective To demonstrate fundamental software application skills in MS Word, Excel, PowerPoint and Access in support of basic business applications Microsoft Word: Basic Concepts Layout & Graphic Features Specific concepts to be included in Pre- /Post testing Use Word to create word processing document Format text fragments and paragraphs bold, italic, underline, font face, font size, and color Paragraph alignment options: left, center, right, and justify Use search and replace Use spell check Set page margins Set ruler measurement and display ruler Set paragraph line spacing, margins, first line, hanging indentation Understand the concept of tabs and tab alignment use the ruler to set tabs Insert images into documents
Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) Specific Microsoft Office 2007 skills mapped to course objectives Course Objective To demonstrate fundamental software application skills in MS Word, Excel, PowerPoint and Access in support of basic business applications Microsoft Excel: Basic Concepts Charting Concepts Database Concepts Specific concepts to be included in Pre- /Post testing Create and edit a spreadsheet document Describe advantages of using formulas rather than entering values only Use copy, cut, and paste to coy and move data including formulas Use built-in functions View formulas Format data in cells, including font, bold, underline, italic, and shading Etc
Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed Determine how learning will be assessed Pre-instruction Post-instruction Determine how results will be tracked, scored and analyzed Determine what constitutes success
Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed Assessment of Microsoft Office 2007 skills Pre-Assessment Questions selected & incorporated within self assessment survey Students rated their skill levels at beginning of course on Likert 4-point opinion scale (none 0, low 1, medium 3, high 4)
Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed
Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed Assessment of Microsoft Office 2007 skills Post-assessment Specific skills mapped to lab homework & in-class lab assignments Grading of lab homework & in-class assignment observation scored on 4 point scale to correlate with self-assessment survey s Tracked in master spreadsheet and compared with selfsurvey Success if post-assessment 3 point or higher
Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed Assessment of Basic IT concepts & capabilities Pre-Assessment Specific questions for each objective/concept & capability selected & incorporated within a preassessment test given at beginning of semester
Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed
Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed Assessment of Basic IT concepts & capabilities Post-assessment Pre-assessment test questions embedded within first and second course exams Student answers to questions entered in master tracking spreadsheet and compared to original responses Success if 75% or higher by category within objective (and overall)
Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed Objectives & Concepts Spreadsheet Template Summarizing Step 4 (prior to implementation)
Step Five: develop assessment & tracking tools prior to implementation Develop pre-assessment survey pre-assessment test tracking spreadsheet
Step Six: implementation Pre-Assessment administer pre-assessment survey & log results administer pre-assessment test & log results Instruction/Learning/Post-Assessment Throughout semester teach IT concepts, capabilities, and skills As taught, assess learning via embedded test questions and lab assignment, and log results
Step Seven: Analysis Analyze results Run statistics and graphs on pre-/post results Individual question within each sub-category Sub-category Category (objective) Microsoft 2007 Skills Summary General IT Concepts & Capabilities Summary Overall Computer Literacy Determine what worked & what didn t
Step Eight: Modification & Fine Tuning Pre-/Post Testing Process Modify teaching methods, homework assignments, test questions, pre-/post-assessment methods, etc. per results of analysis
Step Eight: Modification & Fine Tuning Pre-/Post Testing Process Fine tune process (streamline & make more efficient)
Tracking & Analysis Spreadsheet Microsoft Office Literacy Tab Basic IT Literacy Tab
Results Computer Literacy Total 100 80 60 40 20 0 70 Computer Literacy (BSAD 1700) Fall 2009 97 Microsoft Office 2007 Skills (converted to 100% scale) 42 79 Basic IT Concepts & Capabilities 56 Total 88 Pre Post
Analysis (results) Basic IT Concepts & Capabilities Basic IT Concepts & Capabilities 100 90 80 70 60 50 40 30 20 10 0 83 81 73 37 95 76 69 31 33 32 79 54 75 20 92 59 65 39 42 79 Pre Post
Analysis (results) Basic IT Concepts & Capabilities Be familiar with basic IT hardware & software 100 90 80 70 60 50 40 30 20 10 0 95 95 100 91 77 86 100 95 68 91 100 95 98 82 55 70 0 10 73 83 Pre Post 100 90 80 70 60 50 40 30 20 10 0 Familiar with Web site organization, how search engines work, & evaluate accuracy of Web 45 86 14 59 55 91 18 68 33 76 Pre Post 100 90 80 70 60 50 40 30 20 10 0 Have an understanding of GUI in IT 100 81 75 68 60 32 37 18 Question 1 Question 2 Question 3 Total Question 1 Question 2 Question 3 Question 4 Total Pre Post
Analysis (results) Basic IT Concepts & Capabilities Fundamental patterns which IT is built 100 90 80 70 60 50 40 30 20 10 0 91 77 86 41 68 59 68 59 55 55 73 14 52 0 73 65 52 5 5 5 31 69 Pre Post 100 90 80 70 60 50 40 30 20 10 0 Privacy & digital security 95 82 80 79 63 63 54 18 Question 1 Question 2 Question 3 Total Pre Post 100 90 80 70 60 50 40 30 20 10 0 Debugging strategies & using work arounds 95 95 95 45 32 18 Question 1 Question 2 Total Pre Post
Analysis (results) Basic IT Concepts & Capabilities 100 90 80 70 60 50 40 30 20 10 0 Spreadsheet Concepts 100 95 91 91 92 82 64 59 45 36 Question 1 Question 2 Question 3 Question 4 Total Pre Post 100 90 80 70 60 50 40 30 20 10 0 Computers impact on society & ethical considerations 96 85 75 75 45 50 23 20 9 0 Question 1 Question 2 Question 3 Question 4 Total Pre Post 100 90 80 70 60 50 40 30 20 10 0 Database Concepts 85 84 65 45 41 39 32 25 Question 1 Question 2 Question 3 Total Pre Post
Analysis (results) Microsoft Office 2007 Skills 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 3.82 4.00 4.00 3.38 Microsoft Access 2007 Skills 3.79 4.00 3.59 3.73 4.00 3.73 3.94 3.88 2.92 2.81 2.57 2.31 2.05 1.68 Pre Post
Analysis (results) Microsoft Office 2007 Skills Word Basic Concepts 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 3.95 4.00 3.95 3.91 3.95 4.00 4.00 4.00 4.00 3.91 3.82 3.95 3.95 3.82 4.00 3.23 Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Skill 6 Skill 7 Total Pre Post 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 Word Layout & Graphic Features 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 3.68 3.73 3.68 3.77 3.45 3.18 3.23 3.36 3.32 3.38 2.86 2.91 Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Skill 6 Skill 7 Skill 8 Skill 9 Skill 11 Skill 12 Total Pre Post
Analysis (results) Microsoft Office 2007 Skills 4.50 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 Excel Basic Concepts 3.91 3.91 3.86 3.91 3.64 3.64 3.64 3.79 3.79 3.41 3.14 3.32 3.05 2.92 2.64 2.64 2.50 2.68 Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Skill 6 Skill 7 Skill 8 Total Pre Post 4.50 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 Excel Charting Concepts 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 2.82 2.32 2.45 2.55 2.68 2.64 2.68 2.41 2.57 Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Skill 6 Skill 7 Skill 8 Total Pre Post 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 Excel Basic Database Concepts 3.59 3.64 3.59 3.59 3.55 3.59 2.41 2.45 2.27 2.23 2.18 2.31 Pre Post Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Total
Analysis (results) Microsoft Office 2007 Skills 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 Access Basic Database Concepts 3.71 3.71 3.71 3.71 3.81 3.76 3.71 3.73 2.14 2.05 2.14 2.18 1.95 2.05 2.05 1.82 Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Skill 6 Skill y7 Total Pre Post 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 Access Advanced Database Concepts 4.00 4.00 4.00 4.00 4.00 4.00 4.00 1.77 1.77 1.82 1.59 1.55 1.59 1.68 Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Skill 6 Total Pre Post
Analysis (results) Microsoft Office 2007 Skills 4.00 3.90 3.80 3.70 3.60 3.80 3.77 PowerPoint Presentation Concepts 4.00 4.00 4.00 3.94 3.90 3.77 3.77 3.73 3.68 3.64 Pre Post 3.50 3.40 Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Total
Modification & Process Fine Tuning Modifications (include) Concepts/Topics o Add computer network concepts to literacy definition and assessment (and instruction) Basic IT Concepts and Capabilities o Revise instruction within Familiar with Computer Terminology component (83%, but missing few key points about hardware/software definitions) o Revise instruction and reformat some questions within Understand fundamental patterns on which all information technology is built component (only 69%, but 38 point increase one question only 10% correct) o Add question about evaluating Web site accuracy to Be familiar with Web site organization, how search engines work, and how to evaluate accuracy of Web component
Modification & Process Fine Tuning Modifications (include) Microsoft Office 2007 oin-class assessment of Advanced Access skills (not homework assignment)
Modification & Process Fine Tuning Fine Tuning (include) Determine better ways to test ethical reasoning and other higher-level thinking processes Determine ways to incorporate electronic methods to streamline assessment & tracking process (e.g. Blackboard pre-tests & survey) Don t track by specific student (in aggregate) Add post-self assessment survey for Microsoft Office 2007 skills