Studies of Student Learning

Transcription

1 Studies of Student Learning Classroom-Based Pre-/Post Testing Audrey Pereira, Assistant Professor Business Administration Third Annual Winter Assessment Day January 19, 2010

2 Session Objective Review pre-/post testing process used to determine computer literacy of Business Administration students, including analysis of results

3 Classroom-Based Pre-/Post Computer Literacy Testing Assess Computer Literacy Pre and Post BSAD 1700 Introduction to Computer Information Systems Measure Student Computer Literacy from the time students begin class until completion

4 Computer Literacy Pre-/Post Testing 8 Step Process Define Computer Literacy Determine Components to be included on Pre-/Post Tests Map components to Course Objectives Revisions needed? Yes Determine how literacy & learning will be assessed, tracked, scored, & analyzed Develop assessment and tracking tools Implementation (testing, instruction, learning) Analyze results Modification & process fine tuning

5 Step One: define computer literacy Majority of college and post-secondary students are already familiar with computer, the Internet & WWW Don t need rudimentary instructions on doubleclicking, resizing windows, etc.

6 Step One: define computer literacy Need to be taught to be confident, in-control users of IT Navigate independently in ever-changing worlds of information and technology Solve problems on own Apply power of IT tools for personal, education, & career goals

7 Step One: define computer literacy Computer Literacy can be broken into 3 types of knowledge (1) Skills: proficiency with contemporary computer applications Concepts: fundamental knowledge underpinning IT in order to build new understanding as IT evolves Capabilities: higher-level thinking processes, such as problem solving, reasoning, troubleshooting (1) Synder, L. (2008) Fluency with information technology: Skills, Concepts, & Capabilities (3ed). Boston: Addison Wesley

8 Step One: define computer literacy General understanding of information technology (Concepts & Capabilities) Microsoft Office 2007 skills (Skills)

9 Step Two: determine specific concepts, capabilities, & skills Determine specific general information technology concepts and capabilities required for computer literacy & to be included in pre-/post testing Determine specific Microsoft Office 2007 skills required for computer literacy & to be included in pre-/post testing

10 Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) Map to course objectives (learning outcomes) Verify support goal Adjust if required

11 Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) General IT Concepts & Capabilities Course Objectives To understand basic computer information technology hardware and software terminology To comprehend the fundamental patterns on which all information technology is built To be familiar with Web site organization, how search engines work, and how to evaluate the accuracy of Web content To become familiar with the components and implications of graphical user interfaces within information technology To explore basic computer software and hardware debugging strategies and workarounds. To have an understanding of basic privacy and digital security concepts and consider their implications To recognize and discuss ethical issues and the impact of computers on society

12 Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) Map specific general IT concepts & capabilities course objectives Course Objective To understand basic computer information technology hardware and software terminology To comprehend the fundamental patterns on which all information technology is built To be familiar with Web site organization, how search engines work, and how to evaluate the accuracy of Web content Specific concepts to be included in Pre- /Post testing Why important to know the correct terminology Define basic terminology including: Software vs Hardware Different monitor displays RGB Processor Bits and Bytes Link between patterns, symbols, and information PandA encodings using physical phenomenon Binary systems ASCII Meta data Digital vs. analog To be familiar with Web site organization, how search engines work, and how to evaluate the accuracy of Web content

13 Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) Specific general IT concepts & capabilities mapped to course objectives Course Objective To have an understanding of basic privacy and digital security concepts and consider their implications To recognize and discuss ethical issues and the impact of computers on society To become familiar with the components and implications of graphical user interfaces within information technology Specific concepts to be included in Pre- /Post testing Privacy practices in United States versus the European Union Identity theft Privacy principles and uses, including no uses, approval, objection, and no limits Netiquette Computer viruses, worms, and other malware Permitted vs. not permitted uses of software Software copyrights Analogies Strategy to learn a new software packages Advantages of consistent GUIs To explore basic computer software and hardware debugging strategies and workarounds Basic strategy for debugging Work around

14 Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) Microsoft Office 2007 skills course objectives To demonstrate fundamental software application skills in MS Word, Excel, PowerPoint and Access in support of basic business applications

15 Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) Specific Microsoft Office 2007 skills mapped to course objectives Course Objective To demonstrate fundamental software application skills in MS Word, Excel, PowerPoint and Access in support of basic business applications Microsoft Word: Basic Concepts Layout & Graphic Features Specific concepts to be included in Pre- /Post testing Use Word to create word processing document Format text fragments and paragraphs bold, italic, underline, font face, font size, and color Paragraph alignment options: left, center, right, and justify Use search and replace Use spell check Set page margins Set ruler measurement and display ruler Set paragraph line spacing, margins, first line, hanging indentation Understand the concept of tabs and tab alignment use the ruler to set tabs Insert images into documents

16 Step Three: map literacy skills, concepts, & capabilities to course objectives (learning outcomes) Specific Microsoft Office 2007 skills mapped to course objectives Course Objective To demonstrate fundamental software application skills in MS Word, Excel, PowerPoint and Access in support of basic business applications Microsoft Excel: Basic Concepts Charting Concepts Database Concepts Specific concepts to be included in Pre- /Post testing Create and edit a spreadsheet document Describe advantages of using formulas rather than entering values only Use copy, cut, and paste to coy and move data including formulas Use built-in functions View formulas Format data in cells, including font, bold, underline, italic, and shading Etc

17 Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed Determine how learning will be assessed Pre-instruction Post-instruction Determine how results will be tracked, scored and analyzed Determine what constitutes success

18 Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed Assessment of Microsoft Office 2007 skills Pre-Assessment Questions selected & incorporated within self assessment survey Students rated their skill levels at beginning of course on Likert 4-point opinion scale (none 0, low 1, medium 3, high 4)

19 Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed

20 Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed Assessment of Microsoft Office 2007 skills Post-assessment Specific skills mapped to lab homework & in-class lab assignments Grading of lab homework & in-class assignment observation scored on 4 point scale to correlate with self-assessment survey s Tracked in master spreadsheet and compared with selfsurvey Success if post-assessment 3 point or higher

21 Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed Assessment of Basic IT concepts & capabilities Pre-Assessment Specific questions for each objective/concept & capability selected & incorporated within a preassessment test given at beginning of semester

22 Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed

23 Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed Assessment of Basic IT concepts & capabilities Post-assessment Pre-assessment test questions embedded within first and second course exams Student answers to questions entered in master tracking spreadsheet and compared to original responses Success if 75% or higher by category within objective (and overall)

24 Step Four: determine how initial literacy and learning will be assessed, tracked, scored, and analyzed Objectives & Concepts Spreadsheet Template Summarizing Step 4 (prior to implementation)

25 Step Five: develop assessment & tracking tools prior to implementation Develop pre-assessment survey pre-assessment test tracking spreadsheet

26 Step Six: implementation Pre-Assessment administer pre-assessment survey & log results administer pre-assessment test & log results Instruction/Learning/Post-Assessment Throughout semester teach IT concepts, capabilities, and skills As taught, assess learning via embedded test questions and lab assignment, and log results

27 Step Seven: Analysis Analyze results Run statistics and graphs on pre-/post results Individual question within each sub-category Sub-category Category (objective) Microsoft 2007 Skills Summary General IT Concepts & Capabilities Summary Overall Computer Literacy Determine what worked & what didn t

28 Step Eight: Modification & Fine Tuning Pre-/Post Testing Process Modify teaching methods, homework assignments, test questions, pre-/post-assessment methods, etc. per results of analysis

29 Step Eight: Modification & Fine Tuning Pre-/Post Testing Process Fine tune process (streamline & make more efficient)

30 Tracking & Analysis Spreadsheet Microsoft Office Literacy Tab Basic IT Literacy Tab

31 Results Computer Literacy Total Computer Literacy (BSAD 1700) Fall Microsoft Office 2007 Skills (converted to 100% scale) Basic IT Concepts & Capabilities 56 Total 88 Pre Post

32 Analysis (results) Basic IT Concepts & Capabilities Basic IT Concepts & Capabilities Pre Post

33 Analysis (results) Basic IT Concepts & Capabilities Be familiar with basic IT hardware & software Pre Post Familiar with Web site organization, how search engines work, & evaluate accuracy of Web Pre Post Have an understanding of GUI in IT Question 1 Question 2 Question 3 Total Question 1 Question 2 Question 3 Question 4 Total Pre Post

34 Analysis (results) Basic IT Concepts & Capabilities Fundamental patterns which IT is built Pre Post Privacy & digital security Question 1 Question 2 Question 3 Total Pre Post Debugging strategies & using work arounds Question 1 Question 2 Total Pre Post

35 Analysis (results) Basic IT Concepts & Capabilities Spreadsheet Concepts Question 1 Question 2 Question 3 Question 4 Total Pre Post Computers impact on society & ethical considerations Question 1 Question 2 Question 3 Question 4 Total Pre Post Database Concepts Question 1 Question 2 Question 3 Total Pre Post

36 Analysis (results) Microsoft Office 2007 Skills Microsoft Access 2007 Skills Pre Post

37 Analysis (results) Microsoft Office 2007 Skills Word Basic Concepts Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Skill 6 Skill 7 Total Pre Post Word Layout & Graphic Features Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Skill 6 Skill 7 Skill 8 Skill 9 Skill 11 Skill 12 Total Pre Post

38 Analysis (results) Microsoft Office 2007 Skills Excel Basic Concepts Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Skill 6 Skill 7 Skill 8 Total Pre Post Excel Charting Concepts Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Skill 6 Skill 7 Skill 8 Total Pre Post Excel Basic Database Concepts Pre Post Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Total

39 Analysis (results) Microsoft Office 2007 Skills Access Basic Database Concepts Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Skill 6 Skill y7 Total Pre Post Access Advanced Database Concepts Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Skill 6 Total Pre Post

40 Analysis (results) Microsoft Office 2007 Skills PowerPoint Presentation Concepts Pre Post Skill 1 Skill 2 Skill 3 Skill 4 Skill 5 Total

41 Modification & Process Fine Tuning Modifications (include) Concepts/Topics o Add computer network concepts to literacy definition and assessment (and instruction) Basic IT Concepts and Capabilities o Revise instruction within Familiar with Computer Terminology component (83%, but missing few key points about hardware/software definitions) o Revise instruction and reformat some questions within Understand fundamental patterns on which all information technology is built component (only 69%, but 38 point increase one question only 10% correct) o Add question about evaluating Web site accuracy to Be familiar with Web site organization, how search engines work, and how to evaluate accuracy of Web component

42 Modification & Process Fine Tuning Modifications (include) Microsoft Office 2007 oin-class assessment of Advanced Access skills (not homework assignment)

43 Modification & Process Fine Tuning Fine Tuning (include) Determine better ways to test ethical reasoning and other higher-level thinking processes Determine ways to incorporate electronic methods to streamline assessment & tracking process (e.g. Blackboard pre-tests & survey) Don t track by specific student (in aggregate) Add post-self assessment survey for Microsoft Office 2007 skills