Assessing the Effectiveness of an Online Simulation Exercise in Homeland Security Education Ouanessa Boubsil Interim Executive Director, Security Studies Laboratory University of Maryland University College
Outline Background University of Maryland University College Homeland Security and Information Assurance Education Simulation Exercise The village of San Luis Rey (SLR) Study Setting Methodology Study Tools Results SLR Survey results Course Evaluations results Areas for further exploration
The University of Maryland University College Founded in 1947 to provide education for military personnel and working adults Fully accredited institution of higher education headquartered in Adelphi, Maryland, USA Globally focused with more than 3,300 faculty members and 90,000 students worldwide 29 bachelor s degrees, 19 masters degrees, a doctoral program and more than 80 certificates, most of them offered both online and face-to-face.
Homeland Security & Information Assurance Programs Undergraduate Majors Homeland Security Information Assurance Specializations within different Master of Science degrees Information Assurance Homeland Security Management Undergraduate and Graduate Certificates in Information Assurance Homeland Security Management Doctor of Management program Core course requirement Enterprise Continuity and Information Assurance
Graduate Enrollment Graduate enrollment in Information Assurance (IA) and Homeland Security (HS) courses 1999-2007 2000 465 862 1295 1489 1800 1751 1600 1400 1200 1000 800 600 400 Enrollement 62 111 115 201 200 0 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06 FY07 FY
Undergraduate Enrollment Undergraduate enrollm ent in Inform ation Assurance (IA) and Homeland Security (HS) courses 1999-2006 1829 2000 18 0 0 1509 16 0 0 14 0 0 1197 12 0 0 1003 10 0 0 579 725 820 800 600 351 400 200 0 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06 FY
The Simulation Exercise: The Village of San Luis Rey (SLR) Multimedia scenario-based simulation, enhanced with news releases, online discussions, photo and schematic imagery, and audio elements to create a realistic experience A large database of sector reference material including maps, news articles, a library etc. Instructor controlled staged information releases
Setting An introductory graduate course in Homeland Securityincluding infrastructure protection and issues in technical areas such as interconnectivity and interoperability A scenario-based simulation exercise -The Village of San Luis Rey (SLR)- by Teleologic Learners apply their learning of homeland security and critical infrastructure protection to try to prevent a multi-prong terrorist attack on the critical infrastructure sectors.
Project Class is divided into Homeland Security Strategy Teams (HSSTs) 4 or 5 students Each HSST is assigned to the protection of one of six critical infrastructure sectors (Banking, Electric Power, Water, Healthcare, Transportation and Communication) HSSTs inventory, assess, analyze and submit a written report to the mayor of the village of SLR detailing their team s findings and recommendations on how to protect the infrastructure.
Methodology 2 modules of the same course, taught by the same instructor, using the same textbooks and other instructional materials 2 consecutive semesters (Fall 2005 and Spring 2006) Fall 2005 the control module used a group paper Spring 2006 used the SLR simulation exercise
Study Tools Student Survey specifically designed for the use of SLR The generic end-of-semester course evaluations used for all courses taught at UMUC Course statistics measuring students participation and interactions Students comments
San Luis Rey (SLR) Survey Subjects 2 online sections of ITSM 620 Concepts in Homeland Security 56 students in total Survey 30 questions 41 completed surveys (77% response rate) 21 students provided comments
Areas Surveyed Personal information Technology SLR exercise content Students learning Students satisfaction Team dynamics Exercise organization and workload requirements Students comments and suggestions
Survey Results: Strengths Login was easy Links worked well Technology functioned well Sector information was mostly helpful SLR helped in gaining knowledge
SLR helped in gaining knowledge The SLR exercise was helpful in gaining understanding of critical infrastructure concepts 60% 50% 40% Percentage 30% 20% 10% 0% 1 2 3 4 5 Opinion 1: Strongly Disagree 2: Disagree 3: Neutral 4: Agree 5: Strongly Agree
SLR was enjoyable I enjoyed the exercise Percentage 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% 1 2 3 4 5 Opinion 1: Strongly Disagree 2: Disagree 3: Neutral 4: Agree 5: Strongly Agree
SLR would attract students I would take another course that uses the SLR exercise 40% 35% 30% Percentage 25% 20% 15% 10% 5% 0% 1 2 3 4 5 Opinion 1: Strongly Disagree 2: Disagree 3: Neutral 4: Agree 5: Strongly Agree
Improvement areas Library information Maps Staged scenarios Time period too short Some team members didn t work well together
Comments Common Themes Some felt it added depth to the course Some wanted to be in more than one sector Directions were too broad Time to complete the exercise was too short Too much information
Student Course Evaluation Evaluation Item/Module on a scale of 1 to 5 0509(without simulation) Course Overall 4.05 4.20 0602(with the simulation) Interaction 3.90 4.18 Pedagogy 4.15 4.46 Course Objectives 3.94 3.99
Students Interactions Item/Module Conference postings (Number of words) 0509(without simulation) 77428 263632 0602(with the simulation) 70858 243338 Study Group postings (number of words) 20644 42343 Teacher 2465 4664
Students Comments 0509 (without the SLR) Strongest features of the course Reading the material (2/6) Discussion and Interaction (2/6) Recommendations to improve the course Less paper writing assignments Adapt course to include response to natural disasters and illnesses
Students Comments 0602 (with the SLR) Strongest features of the course Teacher developed a really cool project which helped us learn a lot Strongest feature was class discussion(2) Most invigorating course I have taken SLR was the best Enjoyed interaction with other class members although we don t always see eye to eye
Positive responses when prompted for areas for improvement Recommendations to improve the course I can t think of any at the moment - I am very pleased It is a solid built course Should be taught face to face (2) Needs a sequel as for this course it was excellent. There are no further recommendations. Although the topic for the group project was a good idea, I would not change the scenario the week the project is due.
Areas for further exploration Correlation between satisfaction and select variables from the survey such as sector assigned, grade, etc. Focus on the cognitive gains by comparing samples of the written assignments Group dynamics and effects on student performance and satisfaction ( size and homogeneity) Explore the Return On Investment (ROI) factors--impact on enrollment and retention
Thank you for your attention! oboubsil@umuc.edu