The Integration of Secure Programming Education into the IDE

The Integration of Secure Programming Education into the IDE Bill Chu Heather Lipford Department of Software and Information Systems University of North Carolina at Charlotte 02/15/13 1

Motivation Software vulnerabilities are a major contributor to information security problems Many vulnerabilities can be avoided by adhering to secure programming practices, such as input validation/sanitization Example: SQL injection Discovered a long time ago Easily preventable (e.g. using prepared SQL statements) Yet it is still a problem for about 7% of enterprise web applications We are not doing enough for secure programming education 02/15/13 2

Challenges Most CS courses do not talk about secure programming Courses focus on subject matters (e.g. OS, Database, AI) Not enough time for all primary topics Difficulties of providing a secure programming course May be elective, and thus reach only a subset of computing students There may not be room in the degree program Students may not take them until they are juniors or seniors, and thus may have learned bad secure programming habits without being aware 02/15/13 3

Our Approach To provide training in, and support for, good secure programming practices as part of the tools that students use to program throughout all courses Serves as a continuous educational opportunity that adds to or reinforces the students secure programming training while they are performing their coding activities. 02/15/13 4

Integrative Learning Theory Our brains connect stored information based on use patterns. Stronger connections make it easier to recall and apply information. When the learning process facilitates these connections, students learn more effectively.

ESIDE Demo ESIDE stands for Education Security in Integrated Development Environment Based on Eclipse Java Development Tooling (JDT) https://www.youtube.com/watch?v=vjzlpccmjtm Major features Instant security warnings (e.g. input validation/output encoding/dynamic SQL statements) Code generation Interactive annotation Explanations webpages 02/15/13 6

Instructional Page

Instructional Page 2

Seven Mixed Method Studies Advanced Student Studies UNCC NBAD Assignment Deployment UNCC NBAD Semester Deployment Intermediate Student Study Elon CS2 Classroom Activity Early Student Studies Elon CS1 Classroom Activity JCSU CS1 Focus Group JCSU CS1 Interactive Walkthrough / Interview Elon CS1 Points Incentive on 10 Day Assignment

ESIDE Research Questions RQ1 - Can ESIDE positively influence a student's secure coding mindset? RQ2 - Can ESIDE motivate and incentivize students to learn about secure programming (vulnerabilities, coding, etc)? RQ3 - Can ESIDE motivate and incentivize students to implement secure programming practices?

Advanced Students Study I Students enrolled in a web-programming course 20 students with no prior secure programming training Worked on their programming assignments in the lab using Eclipse/Java for 3 hours Two students in the lab at same time, based on their sign ups Students are at different stages of project implementation Study setup Screen recording & ESIDE logs Pre- and post-tests On secure programming knowledge Two exams each with 18 true/false questions Counterbalanced Semi-structured interview Perceptions about warnings, explanations pages Actions they took with ESIDE What they learned 11

Sample Test questions (Choose ALL that apply) For the following statements, which of the declared variables will require subsequent validation? Integer time = System.getTime(); String title = request.getparameter( Title ); String message = in.readline(); (in is an instance of BufferedReader) Boolean complete = myorder.checkcomplete(orderdata); (Choose ALL that apply) Which of the following statements may be subject to SQL injection? String sql = "SELECT composer, date FROM symphony WHERE conductor =?"; prest = con.preparestatement(sql); prest.setstring(1,request.getparameter( conductor )); ResultSet rs1 = prest.executequery(); statement.executeupdate ( UPDATE users SET item= + user.getitem() + ); 02/15/13 12

User Study Results 20 participants 12 male Master s students, 6 female Master s students and 2 male undergraduate students 4 reported they know a few but very limited security concepts None of them had secure programming experience before the study Test scores Student behavior observations Student perception interviews 13

Test Results Analysis 10.3 increased in average scores after using ESIDE Average pre-test score: 53.03 Average post-test score: 63.33 The differences between pre- and post test scores are statistically significant Wilcoxon Signed Ranks Test on raw test scores t (Z = -2.931, P = 0.003) Mean Std. Deviation Min Max Pre- test percentage Post- test percentage 53.03% 12.81 33.33% 83.33% 63.33% 10.86 44.44% 77.78% 02/15/13 14

Student Behavior 461 distinctive warnings generated overall 70% clicked (321/461) 47% resolved (217/461) 02/15/13 15

Interaction Findings Interaction / Impression [p 86] it wasn't too disruptive or in the way [p183] easy to use just had to click in one spot [p190] the popups are great and super helpful Usage Study Length (days) Avg Days ESIDE Ran Interaction avg per day Interacted 1 Interacted >10 NBAD F12 10 3.57 1-7 range NBAD Sp13 95 5.56 1-34 range 7.3 6 6 <1 32 9

Advanced Findings RQ1- Can ESIDE positively influence a student's secure coding mindset? [p 9] I started to be able to predict when the icon would flag my code before I even wrote it [p10] I hadn t realized my code was insecure [p11] I found it helpful and enlightening to see points of insecurity in my code [p190] the explanations helped me understand why certain code practices are dangerous

Advanced Findings RQ2 - Can ESIDE motivate and incentivize students to learn about secure programming? [p 26] the explanations helped me learn about dangerous code practices [p189] I learned why my code was unsafe Pre (SD) Post (SD) Change NBAD F12 (n8) 60.4 (16.8) 65.4 (13.5) 5.0 ran 4 days (n4) 54.4 (19.2) 69.1 (18.3) 14.7* ran < 4 days (n4) 66.4 (13.9) 61.8 ( 7.4) -4.63 NBAD Sp13 (n18) 30.6 (25.1) 38.9 (23.5) 8.3 ran 5 days (n9) 27.8 (26.8) 43.3 (30.0) 15.5* ran < 5 days (n9) 33.3 (24.5) 34.4 (15.1) 1.1

Advanced Findings RQ3 - Can ESIDE motivate and incentivize students to implement secure programming practices? [p50] I was always concerned about getting the project done and not necessarily the quality (in regards to security) of the finished product [p11] I did not use ESIDE to correct insecurities as that was not my primary concern [p16] I have not had a chance to work with ESIDE because my main focus was getting [the] project done with full functionality before the due date Assignments Reviewed auto-gen secure code Secure code in homework NBAD F12 8 5 2 NBAD Sp13 256 4 0

Advanced Themes Functionality Mindset Timing Students do learn secure programming from ESIDE

Early / Intermediate Overview School / Class Date Level Participants Female / Male Study Type Elon CS1 F12 early 61 25 / 36 Classroom Activity Elon CS2 Sp13 intermediate 22 9 /13 Classroom Activity JCSU CS1 Sp13 early 5 1 / 4 Focus Group JCSU CS1 F13 early 4 1 / 3 Walkthrough Elon CS1 F13 early 57 17 / 40 Assignment

Study #1: Early: Elon CS1 Goal: Explore receptiveness to ESIDE support, assess support level alignment to student knowledge and abilities, obtain student perceptions of ESIDE interaction Study: Single lab setting, three sections, 61 participants (25F / 36M). Students wrote a program that accepted a username and averaged grades. Data: Pre/post surveys, assignment code, interaction logs, instructional page visits, group interaction notes, informal interviews, open ended survey questions. Quick Findings: Appreciated ESIDE, code modification willingness, course content confusion, encouraged to ignore Eclipse warnings [p56] Eclipse pop-up windows are always kind of vague.

Study #4: Early: Elon CS1 Goal: Examine the influence of a 5% points incentive has on coding behavior and knowledge gain and explore the receptiveness and appropriateness of the ESIDE support. Study: Assignment study with 57 (17F/40M) students from 3 sections. Students completed a review assignment focused on array manipulation with simple i/o data acquisition. Data: Pre/post surveys, assignment code, interaction logs, informal interviews, open ended survey questions. Quick Findings: Students un-expectantly struggled with the assignment, 34/43 completed the assignment, those with no incentive interacted at a lower rater than those with an incentive.

Study #5: Intermediate: Elon CS2 Goal: Assess how well ESIDE s materials and evaluation survey aligned with intermediate student knowledge and obtain intermediate student impressions of their ESIDE interaction. Study: Single lab setting, one section, 22 (9F/13M) participants. Students worked on an app which took a name from a user, checked it against a female list and a male list (as found in a separate file), then return the number of occurrences. Data: Post-survey, interaction logs, ethnographic notes. Quick Findings: Two learning topic difficulty e.g., course instruction vs. ESIDE instruction (Directories - Path Traversal).

Early / Intermediate Findings RQ1- Can ESIDE positively influence a student's secure coding mindset? [p 54] it allowed me to be aware of security risks [p 93] It helped me to learn more about how to make better, more secure code [p134] it shows me that though my code will run with proper user inputs, its still vulnerable [p137] showed me what exactly insecure code was [p124] I believe it gave some good information about something I did not know about [p103] as a beginning programmer it was nice for me to learn more about this for future programs

Early / Intermediate Findings RQ2 - Can ESIDE motivate and incentivize students to learn about secure programming? [p 99] taught me a little about code security [p134] I wish I had more practice using ESIDE I only had a week [p124] it gave some good information about something I did not know about [p117] the help page taught me things about secure code that I didn't know before Section 1: Points: V2 Section 2: No Points: V1 Section 3: No points: V2 (exam) n pre Pre SD post Post SD Gain 11 62.8% 9.05 73.6% 6.1 10.7%* 15 69.5% 14.42 71.4% 13.68 1.9% 9 67.7% 9.69 78.8% 4.3 11.1%* *Significant at the p <0.05 level. Note small sample size

Early / Intermediate Findings RQ3 - Can ESIDE motivate and incentivize students to implement secure programming practices? [p134] I was too focused on getting the assignment done [p143] I was more focused on getting my program to run correctly [p 90] for a class assignment I am not worried about the security of my code [p 89] this was a CSC130 homework assignment - I wasn't concerned about security CS1 Assignment Study Assignments collected Complete assignments Wrote secure code Section 1: Points 16 14 11 Section 2: No Points 14 11 2 Section 3: No points (exam) 12 09 0

Early / Intermediate Themes Timing Preparedness Functional Mindset Motivation / Incentives Institutional Variation

Conclusion ESIDE s approach can improve student s awareness of security programming ESIDE may need to be customized for students at different stages of learning Incentives from instructors is important 30

Current/future work Redesign ESIDE taking into account of our research results Focus initially on advanced students Web application Mobile application More research on strategies of using ESIDE for early and intermediate students is needed 31

Thank You! Acknowledgement NSF Grants 1318854, 1129190, and1523041 Your input https://www.owasp.org/index.php/owasp_aside_project#tab=main Search key words: OWASP ASIDE 32