STUDENT LEARNING ASSESSMENT REPORT

STUDENT LEARNING ASSESSMENT REPORT SUBMITTED BY: DIANE MURPHY DATE: OCTOBER 16, 2013 BRIEFLY DESCRIBE WHERE AND HOW ARE DATA AND DOCUMENTS USED TO GENERATE THIS REPORT BEING STORED: BLACKBOARD COURSES FOR ACADEMIC YEAR 2012-2013 FILING CABINET IN BALLSTON ROOM 407 INSTITUTIONAL EFFECTIVENESS REPORTS AVAILABLE FROM HTTP://WWW.MARYMOUNT.EDU/OFFICES/IE List all of the program s learning outcomes: EXECUTIVE SUMMARY Learning Outcome 1. Build a broad knowledge of information technology, allowing students to function as IT professionals that can successfully analyze problems and implement IT solutions to these problems both in an internship and in a job after graduation. 2. Build specific skills in a segment of the information technology field (such as software testing) allowing students to compete successfully for internships and entry-level jobs and to work effectively in these areas. 3. Become proficient in at least one programming language, but be able to learn additional programming languages based on knowledge of programming principles, and develop complete working applications. 4. Become proficient in a variety of skills such as computer repair, computer networking, database design, and cybersecurity through a combination of theoretical knowledge and hands-on experience sufficient to obtain industry certifications. 5. Conduct a capstone project that includes research in an aspect of information technology (hardware, software, data, or cybersecurity) and apply that research to a current information technology concern for businesses or society in general. Year of Last Year of Next Planned Assessment Assessment 2009-2010 2012-2013 2009-2010 2011-2012 2007-2008 2008-2009 2007-2008 2008-2009 2012-2013 2012-2013 2011-2012 1

6. Compose and construct written documents and give presentations articulating business needs, identifying solutions, and considering decision implications, with arguments backed up by data. 7. Work successfully in a team environment both as a team leader and as a participant of a team, and communicate effectively with team members who do not have a technical background. 8. Conduct themselves as responsible professionals and global citizens who are aware of ethical issues and societal needs and who can determine the most ethically response to common ethical dilemmas in the field 9. Possess the knowledge and skills required to pursue life-long learning, including advanced degrees in areas relating to information technology and to adapt to an ever-changing, global technological and business environment. 2007-2008 2011-2012 2008-2009 2013-2014 2009-2010 2013-2014 2013-2014 Almost every major challenge in the world today is turning to information technology (IT) for a solution, from conquering disease to running a large business, from improving education and healthcare, to running a government agency. Marymount s BS in Information Technology program combines a solid foundation of knowledge across the life cycle of computer systems, from requirements definition to managing a production computer system, and includes courses covering hardware, software, databases, networks, software testing, project management, and cybersecurity. In addition to a broad base of fundamental IT courses and the liberal arts core, students have the ability to focus in one or more of seven concentration areas: Applied IT: for students who wish to apply their IT knowledge in a specific field of application such as biology or education. Computer Science: for students who wish a career designing and building software products or designing, building, and securing complex networks. Forensic Computing: for students who wish to specialize in this new and growing field, investigating computer crimes, and preparing evidence for use in a court of law. Health IT: for students who wish a career in the emerging field of electronic medical records and other uses of IT to improve health care. Information Systems: for students who will administer computer networks, provide technical support to system users, and design and manage Web sites, usually in a business context. Interactive Media: for students who wish to work in the creative arts field or specialize in the use of multi-media on the Web. Networking and Cybersecurity: for students who wish to specialize in network administration and the protection of today's computer systems. (Started in 2012-13) 2

Enrollment in the IT program continues to increase (2012-13 enrollment being 125 for Fall and 130 for Spring). This has resulted in certain core undergraduate classes being oversubscribed and we will have to consider multiple sections for some of these courses in Fall 2013, particularly the writing intensive courses (limit 20 students). The undergraduate IT program was listed in the top five most popular majors at Marymount (Quick Facts about Marymount University). The prestige of the program has been raised significantly this year by the award of a National Science Foundation grant for the Cybercorp Scholarship for Service program and its designation as a National Security Agency/Department of Homeland Security Center of Academic Excellence/ Information Assurance Education (less than 150 universities in the country have this designation). Describe how the program s outcomes support Marymount s Mission, Strategic Plan, and relevant school plan: The BS in Information Technology program is fully compliant with the mission of Marymount University and the outcomes include both the foundation in arts and sciences (needed for a well-rounded education) and career preparation (broad IT knowledge, specialized knowledge, and specific skills proficiency leading to certifications). The program recognizes the new job skills required in the profession and offers specialized courses (e.g., cybersecurity, software testing, health information systems) to increase each student's success in getting a good internship, first job, and ultimately a career in the field. These job skills are matched with current industry certifications such as A+, Net+, and Security+ from CompTIA, and the Certified Software Tester (CSTE) from the QAI Global Institute. The program also includes skills that are commonly requested by today s employers (communication skills and the ability to work in teams). Finally, it recognizes the changing nature of the IT field and hence the need for life-long learning. The Applied IT concentration also recognizes that the application of IT covers a wide range of disciplines and can be instrumental in everything from government to education and provides opportunities for students to apply IT to another field, such as biology. In the 2011-12 academic year the President articulated a new "vision" for the university. The items in the vision statement that apply specifically to the undergraduate IT program include: Academic Excellence Emphasize inquiry learning at all levels and provide students and faculty with opportunities for research collaboration. Ensure a personalized education through small classes and faculty/student collaboration. Integrate an emphasis on ethics throughout the curriculum. Encourage cross-disciplinary collaboration. Inquiry learning is a key in the IT program and all professors (full-time and part-time) are encouraged to use classroom time for hands-on application of the subject matter through individual and group assignments. Students are also encouraged to develop their skills outside the classroom through membership of clubs such as the Cybersecurity Society and through acquisition of software 3

through the department s Dreamspark license (access to Microsoft developer s software for free) or through open source arrangements). We constantly evaluate the subject matter covered in the program to ensure that we match the knowledge and skills required in the current work place. Recently we became a certified Cisco Academy and began the very popular Networking and Cybersecurity specialty. Ethics is central to several courses beginning in the freshman year. Cross-disciplinary collaboration is encouraged and both Graphic Design majors and Mathematics majors have IT courses in their programs. Community Engagement Make its facilities and expertise available to support and serve the community Instill in its students and the entire University community a commitment to giving back through good citizenship, civic engagement, and volunteer service Actively engage Marymount alumni in serving current students and creating a strong network of support for one another. IT is now essential to all organizations and IT students are encouraged to serve the community through their own individual activities, jointly through club activities, and as assigned on projects in various classes such as in IT125, Web Development, or in their capstone project (IT489, IT Capstone Project). We have a close relationship with IT alumni, particularly those that have gradated over the past six years. Several Marymount alumni act as tutors for the difficult IT courses such as IT230, Advanced JAVA Programming, and IT370, Computer Forensics. In addition, the IT alumni remain in close contact with the faculty and regularly attend events such as seminars, networking events, and participate in the honors society, Upsilon Pi Epsilon, events. Student Profile Achieve improved gender balance Enroll and serve a geographically, racially, ethnically, and religiously diverse student body One of the main concerns in the IT education field is the lack of underrepresented minorities (women, African-American, Hispanics). Most programs have fewer than 10% female participation. Marymount has close to 27% female students and has a good representation of African-Americans and Hispanics. Many of these students are transfer students from the Northern Virginia Community College system with which we have good relations at the faculty level. The program also includes a number of international students and several athletes. 4

Provide a brief description of the assessment process used including strengths, challenges and planned improvements: This is the sixth year of the program and several students have graduated in the last few years. Several alumni have gone on to graduate school (some at Marymount) and many have good jobs in government and industry. Most recently graduated students remain in contact with faculty through LinkedIn and regular visits to the school. The first learning outcome related to the need for students to build a broad knowledge and acquire skills to successfully implement IT solutions in the workplace. Three areas were examined: internships, job placement, and confidence in their ability to be successful in a job in the field. A significant challenge is the ever-changing landscape in IT and the need to add to add more material to the curriculum (e.g., mobile app development, cloud computing,). Each of the required standards was met and Marymount enjoys an excellent reputation in the work place for both internships and entry-level jobs. For example, two students were awarded cybersecurity internships at the FBI this summer, a very competitive process. Much of the internship and job placement is from referrals, many from our alumni who regularly conduct the faculty with opportunities. The second learning outcome was in the area of programming language proficiency. All jobs in the IT industry require some level of knowledge of coding (whether programming, scripting or tagging languages) and all students take two programming courses. Students can choose between Java and Web development and must show an ability to develop an entire application by the end of the advanced classes. This was the least successful of the outcomes (only one of three of the outcomes was entirely successful) and this reflects many students difficulty in the rigors of the programming process. Computer science specialty students were very successful but other students lagged. Supplemental instruction sessions will be developed to assist in the development of these important skills and alumni have volunteered to provide this support. The third learning outcome assessed was proficiency in a wide variety of skills such as computer repair, computer networking, database design, and cybersecurity through a combination of theoretical knowledge and hands-on experience sufficient to obtain industry certifications. Outcomes were generally good in this area; however more students need to be taking the industry certification exams. Certification study groups are starting to be formed by students and this will hopefully result in greater participation in the certification process, considered by some an important criterion in the hiring process. In the 2013-14 academic year, certifications will be stressed in the IT seminars. Four sources of information were used for all three outcomes: results from classroom exercises corresponding to the outcomes, particularly the capstone course; the Internship Program Semester Reports from Fall 2012, Spring 2012 and Summer 2013 prepared by David Pomeroy, SBA Director of Business Learning; and the Graduating Student Survey conducted by the Office of Institutional Effectiveness. 5

Describe how the program implemented its planned improvements from last year: The seminars (IT223, IT323, IT423) were restructured in Fall 2012 to introduce a series of individual and group activities that practiced the 21 st century skills identified by our advisory group and described in the literature as desirable traits for STEM employment (critical thinking, problem-solving, decision making, communication and personal responsibility). During these seminars, students are introduced to the concept that education is for your career and training and certifications are for your job and that you might need a degree and certifications to be competitive. Independent quality research was listed as a concern in the 2011-13 field. This still appears to be a problem for a portion of the IT students and we have assessed research across the curriculum this year. We have began to closely correlate the research activities in each IT core course and to try to scaffold students so that they are able to conduct independent research by their last semester of study, The increase in focus on research activities has uncovered a trend towards students buying papers and we have made some program changes, particularly to the IT489, IT Capstone Project, to incorporate more interim deliverables, some in-class sessions, and more intense oral defense of their project. Writing continues to be an issue and we are putting more faculty through the writing-intensive workshop, even though they may not actually teach a writing-intensive course. We are trying to standardize our approach to writing to ensure consistent work throughout the program. By Fall 2013, 5 0f 8 of the faculty in the department will have been through the writing intensive workshop. We are also making it a requirement in the first year of any new faculty. As noted last year, one of the major issues is the large number of transfer students, mainly from NVCC, and we still need to work on a transition plan for these students in the writing area. Provide a response to last year s University Assessment Committee review of the program s learning assessment report: The summary from last year s report was This was a solid report. Good self-reflection on actual assessment process. The one comment was that the Graduate Student Survey should be listed as an indirect measure not a direct measure and this designation has been correctly used in this report. 6

Outcome and Past Assessment Learning Outcome 1: Build a broad knowledge of information technology, allowing students to function as IT professionals that can successfully analyze problems and implement IT solutions to these problems both in an internship and in a job after graduation. Is this outcome being reexamined? Yes (2009-10) The outcome was achieved in the last review but it was felt that the standards should be higher for success in the internship and job placement area (expectations were raised from 70% to 90%). To do this we introduced seminars (sophomore, junior, and senior) to reinforce the non-technical job skills and to improve areas such as job hiring and networking. Assessment Activity Outcome Measures Explain how student learning will be measured and indicate whether it is direct or indirect. Direct: Each student's ability to satisfy their internship supervisor s expectations in the exit survey of the internship Performance Standard Define and explain acceptable level of student performance. Internship supervisor s rate student performance on a 5-point scale (1 being strongly disagree and 5 being strongly agree). 90% of student should receive an overall 4 or 5 and at least 50% will receive a 5. Data Collection Discuss the data collected and student population The data is taken from the Internship Program Semester Reports from Fall 2012, Spring 2013 and Summer 2013. prepared by David Pomeroy, SBA Director of Business Learning. A total of 26 students were included in the analysis. Analysis 1) Describe the analysis process. 2) Present the findings of the analysis including the numbers participating and deemed acceptable. All students were given a 4 or 5 by their supervisor in response to the question: The intern possessed a comprehensive understanding of the basics of the professional/business field. 56.5% (15) of the 26 students received a score of 5 across the three semesters. The outcomes assessment standard was met; however, there were some specific areas where less than stellar performance was noted. This includes time management (2 students), awareness of cultural differences (1 student), and spoken communication (1 student). 7

Indirect: Ability of the students to find a job Indirect: Ability to be successful in an ITrelated job. Getting a job is an important outcome for a career-oriented program, particularly in today's economy. At least 70% of graduating students should be able to get a job in their field and be successful in that job, or get a promotion in their current position, within 90 days of graduation Students must not only get a job but they must be successful in that job and see their education as a career not just a first job. At least 80% of the students should feel confident that they can succeed in a job in the field Results of the 2012-13 Graduating Student Survey prepared by the Office of Institutional Effectiveness. Results of the 2012-13 Graduating Student Survey prepared by the Office of Institutional Effectiveness. As shown in the Graduating Student Survey results 75% of the 16 students who responded were confident (4 or 5) of their ability to find a job in the field. 81.3% of students also felt that they could attain a promotion within their existing employment situation. The outcomes assessment standard was met, despite the economic realities of finding a job, even in IT. IT has plenty openings but they tend to require at least two-years of experience making it hard to get the entry-level position. 87.5% of the 16 students who responded were confident that they would success in an IT job (compare with 71% of the general Marymount graduating students). Also important was the fact that high levels of confidence were given for the 21 st century skills needed for success (93.8% to work in a team, 100% to lead a tam, 100% to conduct research to support a position, and 93.8% to deliver a coherent oral presentation. Less confidence was expressed for the written communication (81.3%). The numbers are also higher than the general Marymount graduating student population as reflected in the 2013 Graduating Student Survey Evaluation of Preparation The outcomes assessment standard was met. 8

Interpretation of Results Extent this Learning Outcome has been achieved by students: The students were confident in their ability to find a job and succeed in a job in their field and several were awarded internships in very prestigious government agencies and companies (including two students at the FBI and one student at Boeing). IT is a growing field but the technology used is constantly changing, so the ability to learn new things is an important part of the student s success. I believe we achieved that as reflected in the higher than university average scores in the 2012-13 Graduating Student Survey Evaluation of Preparation. For example, 75% of our student felt confident that they could find an IT job in contrast to the 61.3% for the university as a whole, 100% of IT students felt that they could lead a team as opposed to 77.8% of the general university graduating students. Time management still seems to be an issue for all students. In addition, the individual comments from the student in this survey substantiated the individual attention given by faculty to career advice and job placement. Program strengths and opportunities for improvement relative to assessment of outcome: The addition of the IT seminars (IT223, IT323, and IT423) in the past two academic years has helped students understand the importance of technical and non-technical skills (such as writing and presentations) in the workplace and to focus on the skills that differentiate them when searching for a good job in the workplace. Students have come to better understand that getting a job is not only about your technical knowledge and skills, they need soft skills and to be engaged in their community. We continue to develop these semesters. Based on the feedback from the Graduating Student Survey and the supervisor reports from the internships, more attention will be given to understanding the importance of time management, which continues to be a problem with our students. Discuss planned curricular or program improvements for this year based on assessment of outcome: Mobile application development is becoming an important component of the workplace and additional courses are being developed to cover this area. The IT360, Topics in Information Technology, course will focus on mobile app development in Spring 2014. In addition, existing courses will be modified to include additional topics in cloud computing and big data management. 9

Outcome and Past Assessment Learning Outcome 2: Become proficient in at least one programming language, but be able to learn additional programming languages based on a knowledge of programming principles, and develop complete working applications. Is this outcome being reexamined? Yes (2007-2008 and 2008-2009) Difficulties in Java programming were identified as a problem in the earlier assessments and required standards were NOT met.. To combat this, students who were not in the Computer Science specialty, were offered a second option to achieve the programming requirement of the program. The sequence of courses are IT125, Web Development, and IT225, Advanced Web Development. The second course offers the student the opportunity to explore multiple programming languages. The programming techniques are less sophisticated than those in the Java sequence but expose students to many of the Web-based technologies in today s workplace. Outcome Measures Explain how student learning will be measured and indicate whether it is direct or indirect. Direct: Ability of student to develop a complete a complete program in the final project in IT230, Advanced JAVA Programming. Performance Standard Define and explain acceptable level of student performance. At least 60% of the students must be able to compile a program of moderate complexity that is free of compiler errors, includes comments, and is able to run and meet the requirements, without errors. Assessment Activity Data Collection Discuss the data collected and student population Code will be submitted by each student for their final assignment in IT230, Advanced Programming in JAVA. The object and source code, together with written documentation, for each student, was submitted to an independent faculty member for review. The rubric is attached as Rubric 1. Analysis 1) Describe the analysis process. 2) Present the findings of the analysis including the numbers participating and deemed acceptable. The independent faculty member first reviewed the documentation to independently examine the capabilities of the software developed and evaluate it as shown in Rubric 1. Of the 18 students in IT230, only 15 met the standards required by the rubric 83%). Most of the deficiencies were in understanding the requirements and in the ancillary items such as comments and warning errors. Object-oriented programming (such as JAVA) remains one of the more difficult components of the IT major. 10

Direct: Ability of students to complete the assignments in 3 different programming (scripting) languages in IT225, Advanced Web Development. 75% of students should effectively meet the requirements in each of the 3 coding techniques. In IT225, students are required to produce three assignments, three of which involve different programming techniques (JavaScript, php, and C#). The assignments were reviewed by the instructor for usability, functionality, and documentation, with the Rubric shown in Rubric 2 The outcomes assessment standard was met. IT225, Advanced Web development, focuses on ensuring that students have the necessary coding skills to enter the workforce as an entry-level Web developer. The instructor first assessed how well the student understood them and implemented the coding to meet those requirements The Web page and application was run and the functionality requested was tested. The coed was reviewed for clarity and effectiveness. The assignments were evaluated using the Rubric 2. In the evaluation, 80% of the students met the requirements for 2 of the languages but only 70% met the standard for all 3 coding techniques. Direct: Ability of students to develop a complete application as part of IT489, the IT Capstone Project. 70% of students who decided to do application development for their capstone projects successfully completed the assignment as evidenced by a demonstration of the application. The data was collected from the assessment reports submitted by the 2 faculty members who evaluated the demonstration session. The outcomes assessment standard was NOT met. Of the 24 students who took IT489, IT Capstone Couse, (Fall 2012 8. Spring 2013 10, and Summer2013 6), only 6 (25%) did a project that involved considerable coding. 5 of 6 of these students (83%) demonstrated a complete application that met the requirements for their project. The last student was unable to demonstrate a fully functional application. The outcomes assessment standard was met. 11

Interpretation of Results Extent this Learning Outcome has been achieved by students: Students who were interested in being programmers or software developers (Computer Science specialty) and were very successful in the programming assignments and two standards of these students performance were met. Students in the programming with less aptitude for software development generally took the Web development sequence with less emphasis on pure coding. In the later parts of the course, three programming assignments are given and a few students had major difficulties in understanding the requirements, logic development and syntax. This standard was NOT fully met although students were able to make more progress than previously evidenced in the Java courses. Program strengths and opportunities for improvement relative to assessment of outcome: The dual programming paths (JAV A or Web Development) has been successful and has helped students with less aptitude for programming to scaffold their logic and coding skills in a more structured way. They do need more support in the later programming Discuss planned curricular or program improvements for this year based on assessment of outcome: To assist students struggling in programming concepts assignments a supplemental instruction approach is being evaluated. Two alumni have volunteered to be tutors for the students. Workshops will also be offered in a variety of programming languages to further engage students. 12

Outcome and Past Assessment Learning Outcome 3: Become proficient in a variety of skills such as computer repair, computer networking, database design, and cybersecurity through a combination of theoretical knowledge and hands-on experience sufficient to obtain industry certifications. Is this outcome being reexamined? Yes (2007-2008 and 2008-2009) The skill set for the IT field continues to increase and so being proficient in a variety of skills is an important criteria for getting a job and keeping it. Students in 2008 met this outcomes and so the standards have been increased, particularly the requirements for certifications. Assessment Activity Outcome Measures Explain how student learning will be measured and indicate whether it is direct or indirect. Direct: Ability of students to successfully complete a project, and defend it as part of an oral defense, in either networking, database development, or cybersecurity as part of IT489, the IT Capstone Project. Performance Standard Define and explain acceptable level of student performance. 70% of students who decided to do networking, databases, or cybersecurity for their capstone projects successfully completed the assignment as evidenced by their project report and their oral defense of it. Data Collection Discuss the data collected and student population The data was collected from the evaluation reports submitted by the 2 faculty members who evaluated the report evaluation and oral defense sessions. Analysis 1) Describe the analysis process. 2) Present the findings of the analysis including the numbers participating and deemed acceptable. Of the 24 students who took IT489, IT Capstone Couse, (Fall 2012 8. Spring 2013 10, and Summer2013 6), only 11 (46%) did a project that involved either a database, a network development or cybersecurity. 8 of 11 of these students (72%) successfully defended their project report. The other 3 had difficulties in articulating one or more part of their project activities or results. The outcomes assessment standard was met. 13

Direct: Students took associated industry certifications and successfully passed them. Indirect: From Graduating Student Survey, ensure that students feel confident about their ability to solve problems in your field using their knowledge and skills. 50% of students took one or more industry certification tests and passed them. 80% of students believe that they will be effective at solving problems in the workplace. In the seminars, students are surveyed on their certification interest and status and the results posted to Blackboard. Results of the Graduating Student Survey prepared by the Office of Institutional Effectiveness for the question Solve problems in your field using your knowledge and skills Only 42% of graduating students has at least one certification when they graduated. The outcomes assessment standard was NOT met. 87.5% of the 16 students who answered the survey rated their ability to solve problems in the workplace. The outcomes assessment standard was met. Interpretation of Results Extent this Learning Outcome has been achieved by students: Two of the three measures were met. Certifications, as an adjunct to their education, still needs to be emphasized as an important component of evidence of job skills in the IT industry. Program strengths and opportunities for improvement relative to assessment of outcome: The IT seminars are helping to prepare students for the workplace. Discuss planned curricular or program improvements for this year based on assessment of outcome: More emphasis will be placed on the need for certifications and some study groups will be formed. 14