Procedure of Secure Development Tool Adoption Study Introduction This study is designed for us to better understand how developers adopt secure development tools and why some developers refuse to use these tools. (the definition of secure development tools) We define secure development tools as those tools that help find or fix security vulnerabilities reside in source code during the software development life cycle. Example of tools: resharper, Jprofiler, Jprob, Findbugs, FXcut, Valgrind. (2 types of participants adopter & non adopter) During this interview, you will be asked several questions about your past experience with secure development tools. If you have no experience with secure development tool, then our questions will relate to discovering why you have not been exposed to it. (In case of people have trouble understanding me) If you have any difficulty understanding what I am asking, feel free to ask me to repeat myself. (Clarify the privacy issue) During this interview, your voice will be recorded. In any data collected, or in reports or papers that are published, you will not be identified by name. Please be careful not to discuss any sensitive information about the company you work for. If you do mention any, we will do our best to remove it from our transcripts, but better if you don't mention such sensitive information at all. Further Definition of Secure Software and Secure Development Tools (Make sure we are in the same context with the participant) Secure Software Enhancing the Development Life Cycle to Produce Secure Software defines secure software as follows: To be considered secure, software must exhibit three properties: 1. Dependability: Dependable software executes predictably and operates correctly under all conditions, including hostile conditions, including when the software comes under attack or runs on a malicious host. 2. Trustworthiness: Trustworthy software contains few if any vulnerabilities or weaknesses that can be intentionally exploited to subvert or sabotage the software s dependability. In addition, to be considered trustworthy, the software must contain no malicious logic that causes it to behave in a malicious manner. 3. Survivability (also referred to as Resilience ): Survivable or resilient software is software that is resilient enough to (1) either resist (i.e., protect itself against) or tolerate (i.e., continue operating dependably in spite of) most known attacks plus as many novel attacks as possible, and (2) recover as quickly as possible, and with as little damage as possible, from those attacks that it can neither resist nor tolerate.
Secure Development Tools Secure development tools are those tools that help developers make more secure software by finding or fixing security vulnerabilities reside in source code during the software development life cycle. Generally, there are two different types of secure development tools on the market static analysis tools and dynamic analysis tools. Static analysis tools (e.g. Fortify SCA, Armorize CodeSecure, etc.) are used to scan application source code for vulnerabilities. Dynamic analysis tools (e.g. HP WebInspect, IBM AppScan) are used to scan live applications such as web applications or a web service. Do you have any questions about this definition before we continue? Background Check Are you working as a developer in your company, or manager or both? (Our participants are developers or managers. Managers are treated as opinion leader in their company, but they will also be asked questions as developers because they usually either worked as developer before or working as both manager and developer now ) Questions just for managers How many people do you supervise? Can you tell me a little bit about your job's duty? Do you know how decisions are made at your company about tool purchasing? Who makes the decision? Where do you fit into the purchasing chain? What is the most important factor when you consider tool purchasing? Does your company have budget just for tool purchasing? how about security tools? If budget is a big concern, why don t you consider open source security tools? Typically, how do you know about a security tool? What is the information channel that you rely on? How do you trust that channel? After purchased the tool, what is the company s strategy to let people actually use that tool? Was this strategy succeed? (Talk about specific cases, if any) Have you ever adopted any secure development tool for your group? (adoption case) What is the name of the tool you adopted?can you tell me the situation when you adopted that tool? What are the concerns? What is the result of adoption? (main reason for non adoption) What is the main reason you haven t adopted any security tool for your group?
Questions for developers Have you ever adopted any secure development tool?{ Yes. > Go to the Adopter Question part. No. Go to the Non adopter Question part } Non adopter Question (Only ask non adopters) What is the main reason you think that you have not used any of secure development tools? (An open ended question before all the specific questions. Elicit more if possible) (Activity 1: Role Playing before asking awareness question) Awareness Question (Ask both adopter and non adopter) Security sensitive domain What are the domains of the application you have developed? Was security a big concern of the software you have developed? (if the developer claim security is not a concern, ask the following questions to see if we can persuade them that security is a concern, even if it is low) What kind of resource your software accesses? Whether confidentiality, availability and integrity of the resource could be compromised by security bugs in your software? Which programming language you have used? So security is a concern, right? (make them admit) Secure development experience Which programming language you are using? Organizational culture & standards Is developing secure software a big concern in your company? Does your company have any standards to follow in terms of secure development? Reward & punishment system for software security Does your company have any reward and punishment system for software security? or more general for software quality? Organizational structure Does your company have a dedicated security team? Does your company have a dedicated testing team? Perceived responsibility
Do you think you as a developer is responsible for software security? Or the testing team should responsible for software security? Or other dedicated teams? Tool usage observability Can you describe the environment that you usually work in? (share a cubicle with some peers; sitting in a private cubicle but have peers sitting nearby; a private office) Practitioner inquisitiveness Are you interested in exploring new tools and techniques relate to your work? What is your patient level for looking for new tools? added 6/29 Tool advertisement (awareness knowledge) Have you ever seen any secure development tool advertisement? Where did you see it? When? What type of advertisement did you see? Peer influence Has any of your colleague recommended a secure development tool to you before? Does anybody else in your company use the security tool? (Nobody use it? working environment and are people around you using it do make difference. Or management issue? Why doesn t the company introduce security tool to their developers?) Has your manager ever required or encouraged you use any security tools? Education Have you learned about any secure development tools through university courses or company trainings? Does your company provide this kind of training? Is it mandatory or optional? Adopter Question (Only ask adopters) Open questions: (here we ask open questions to discover other factors that are not in our initial model) What is the name of the tool you adopted? Could you please describe the tool to me? Which part of this tool you like most? Which part of this tool you like least? When did you adopt the tool? How did you know about this tool? What type of application you were developing when you adopted this tool? Did this tool help? What made you decide to adopt this tool? What made you decide to try out this tool? Have you ever recommend this tool to people you know?
How did you recommend this tool to others? How was this topic brought up between you and your friend. Did he or she see you using that tool? What is the result of the recommendation? Have them tried it out or adopted the tool? Are you still using this tool?{ Yes. { Have you ever tried other tools that have similar functionality? If so, what makes you continue to use this tool? If I recommend XXX to you, which has more advantages than the one you are using, will you consider discontinue using the current tool? } } No.{ } how long did you use that tool? Why discontinue using that tool? What kind of effort the development team make can change your mind? What do you think is the main reason for the security tool underused problem? added 6/29 Factors related questions: (here we ask questions relate to actors in our initial model) Desired functionality What are the functionalities this tool has? Does that tool have all the desired functionalities you want? Cost & potential gain How much does this tool cost? (financial cost) Was it hard to learn to use that tool? (learning cost) How long it took for you to get familiar with all the operations of that tool? (learning cost) What are the potential benefits if you adopt this tool? Status aspects Will use this tool help you gain status aspects? i.e., treated as more experienced in your company? Do you feel using this tool makes you experienced or advanced developer? Do you feel using this tool makes you superior than other developers who do the same tasks without using this tool? Incentives Did your company provide any incentives for asking you to adopt this tool? Or any punishment if you refuse to adopt it?
Tool advertisement (how to & principles knowledge) How did you learn how to use this tool? How deep you learned to use this tool? Peer influence (ask only when peer recommendation is mentioned by the developer) Did you trust the colleague who recommend this tool to you? Did the situation of that colleague similar to yours? How did the colleague recommend this tool to you? highly recommended? or just mentioned about it? Perceived complexity Was the use interface of the tool complex to you? Was the framework of this tool hard to understand? Perceived compatibility Was this tool compatible with the operating system you are using? Was this tool compatible with the Integrated Development Environment you are using? Was the operation of that tool similar to dominating tools? Did this tool come with a bunch of other tools as a cluster? (technology cluster, eg., HP Fortify Products) Did this tool have some functionalities or strengths that other tools do not have? (Did this tool fit the niche of customers requirement?) Perceived Trialability Did this tool have detailed tutorial? Did this tool have complete documentation? Re Invention Was this tool configurable? Can you customize that tool to better suit your needs? Workflow suitability Did this tool fit into your workflow?
Activities 1. Role Play (Awareness of the tool drives the need of the tool or the need of the tool makes people aware of the tool, which one is the case?) awareness need (Online Advertisement vs interpersonal network ( peers vs opinion leader) (Little concern in this case) One day, you are curious about how to make more secure software, so you google it. This page come out. Will you click the AD inside the red rectangle? (show the picture) A page contains more detailed information come out. Please read it for 2 minutes. Will you try this tool out? Suppose I am your colleague. One day, I said this to you when we met in our company: hi, I am using a tool called CodeSecure. This tool is really good for me to find vulnerabilities in my code, you might want to try it out. Will you try this tool out? Then I said this: remember I were trying to find a bug last time? I asked you to help me. But we did not find anything. The code was just giving weird results. I finally found a bug by using this tool! So I think it might help you out later in similar situations. Will you try this tool out? Suppose I am your manager. One day, I said this to you: Hi, I know a tool named SecureCode. It can make our code more secure. Why don t you try it out? Will you try this tool out? need awareness (trusted peer vs untrusted peer) Suppose you are the guy who posted this post in stackoverflow. Basically you need a tool to help you coding against malicious attacks, e.g. SQL injection. Please read this post for 1 second.
Suppose somebody answered you and posted a link to CodeSecure. Will you try this tool out? 2. Rank the factors/attributes of the secure development tool Can you rank these factors in terms of how important it is when you make your adoption decision? You can drag it to rank them in Google Doc. I will explain the factors one by one. Please let me know if you have any questions. First, let s start with 5 main factors: Perceived Complexity: how complex is the tool? Perceived Compatibility: how compatible this tool with your working environment? Perceived Trialability: how easy can you try this tool out? Perceived Relative advantage: the advantages this tool gives to you over not using any tool or using other tools. Re invention: Can you configure or even customize this tool to more suit your needs? Can you tell me your opinion about these 5 main factors first? Next, we are going to look at more detailed factors. Make a better tool toolsmiths Desired functionality: if this tool have whatever functionalities you want? Cost & potential gain: what are the cost and potential gain if you adopt this tool? is it worth adopting? Compatibility with OS: is this tool compatible with the operating system you are using? Compatibility with IDE: is this tool compatible with the integrated development environment you are using? Operations similar to dominating tools: is the operations of this tool similar to the dominating tools? which reduces the difficulty of learning it. Framework complexity: is the framework of this tool hard to understand? User interface complexity: is the UI hard to understand or hard to use? Tutorial: is the tutorial well written and comprehensive? Documentation: is the documentation complete and helpful? Technology cluster: does this tool come out with other tools as a cluster? (show examples: HP Fortify; Microsoft Security Development Lifecycle Tools). (I do want to emphasize this one) Ideal niche: Does this tool have its special ability that fits in a ideal niche? Configurability: can you configure this tool? Customizability: can you customize this tool? (usually larger changes than configuring). Provide better environment of adopting the tool company managers Organizational culture & standards: is this tool compatible with the company s culture and standards? In other word, does your company care about security (non functional requirement in general)? Does your company have any security requirement that you have to make your code pass?
Status aspects: Will use this tool help you gain status aspects? i.e., treated as more experienced in your company? Incentives: (a factor comes from the company & managers) does anybody provide you incentive to adopt your tool or punishment if you don t? Can you think of other factors that did not mention here but it is important to you to make adoption decision? 3. Brainstorm of desired functionalities (Opinion from novice vs experienced) Can you brainstorm the functionalities you want to have in secure development tool? The functionality can be as fancy as whatever you can think of. Security Experts 1. [Guidance] Drove effort to define company wide usable security design guidance for Microsoft engineers. Can you tell me your experience about designing the company wide usable security design guidance? 1. What are the types of the guidance? 2. Who asked you to design this guidance? 3. What did you do to make ordinary developers follow your guidance? 4. What was the result? Does everybody follow the guidance now? What were the challenges? 5. Does Microsoft have policies to ensure secure coding other than guidance? 2. [Education] Co developed 4 hour course on designing usable security and privacy user experiences that I teach several times per year to Microsoft engineers. 1. What is this course about? 2. Does this course include how to use some specific security tools? 3. Does anybody in the security team teach about using some specific security tools? 4. Is this course optional or mandatory? 5. Who is the audience? Does everybody in Microsoft can attend? 3. [Consulting] Consult with engineering teams as needed on usable security issues 1. What do you do as a consultant? 2. Do the engineering teams interact with you often? 3. Do you do code reviews for new software features? 4. What types of application have the need to consult security experts? 4. [Community building] Maintain a distribution list, bring speakers to campus, and publish a newsletter. 1. What do you do to make developers more aware of the security issues? 5. [Usable security team]
1. Where is this team? Do you sit near ordinary developers? 2. How many people do you have in this team? 3. Do the team members have different expertises? 6. [Company related questions] 1. Do developers in Microsoft use security tools? What are the tools they use? 2. Did Microsoft adopt any security tools in company level? How was this decision made? How was the security evaluated? 3. Is there anybody responsible for searching or developing security tools in Microsoft? 4. Does Microsoft allow developers use outside open source security tools? (Does Microsoft encourage individual level security tool adoption?) 7. [High level questions] 1. Do you think ask developers to use security tools would help them build more secure software? 2. Why security tools are underused? 3. False positive mentioned many times by developers, is there any way to make security tools smart enough to only present the result the users expect? References Goertzel, Karen, Theodore Winograd, et al. for Department of Homeland Security and Department of Defense Data and Analysis Center for Software. Enhancing the Development Life Cycle to Produce Secure Software: A Reference Guidebook on Software Assurance, October 2008.