BASS: A Benchmark suite for evaluating Architectural Security Systems

Transcription

1 BASS: A Benchmark suite for evaluating Architectural Security Systems James Poe 1 and Tao Li 2 Intelligent Design of Efficient Architecture Lab (IDEAL) Department of Electrical and Computer Engineering University of Florida s: 1 jpoe@ufl.edu, 2 taoli@ece.ufl.edu Abstract As software vulnerabilities continue to be exposed on a daily basis and the motivation of cunning adversaries to compromise valuable computer assets grows, vel methods must be developed to ensure security. Recently there has been a growing interest within the computer architecture research community in designing architectural and hardware mechanisms to improve security. Unfortunately, there is currently t a representative set of benchmarks for evaluating the security features of proposed hardware modifications. The frequent result is that great effort is often spent searching for vulnerable programs, and/or evaluations suffer from a lack of diversity. To address this problem, we developed BASS, a benchmark suite to evaluate the security features of proposed architectural solutions under various malicious attack scenarios. BASS v1.0 currently consists of seven benchmarks chosen to cover a diverse range of architectural attack characteristics. To facilitate the use of these benchmarks in architectural security research, we have developed both vulnerable programs and scripts to automatically generate exploits targeting those vulnerable programs across both 32-bit x86 and 64-bit Alpha Linux platforms. The entire BASS framework including documentation, source code, input data sets, and precompiled binaries for the M5 full system simulator is released under the Gnu GPL and can be freely downloaded at 1. Introduction Despite an increased emphasis placed on secure programming models, the number of vulnerabilities reported by CERT has steadily increased in recent years (shown in Figure 1). As the complexity of software continues to grow along with the value of the systems that software is entrusted to protect, both the incentive and opportunity increases for the malicious attacker. A prime example of the levels of complexity modern software engineers are forced to contend with is Microsoft s upcoming Windows Vista operating system [2]. Vista is speculated to have over 50 million lines of code, as many as 50 dependency layers and is one of the largest software projects ever attempted. Completing a software project of this magnitude without introducing vulnerabilities is an almost impossible task, and formal verification of such vast amounts of code is infeasible. Further, purely software methods of ensuring security become impractical because of performance penalties. To address these difficult issues, the computer architecture research community has placed an increased focus on the design of hardware support for enhanced security. In a recent meeting [3] sponsored by the Computing Research Association and the National Science Foundation, a group of panelists that are among Vulnerabilities Reported CERT/CC Statistics /Q2 Estimated Reports Actual Reports Year Figure 1. Number of vulnerabilities reported by CERT (Source [1]) the top researchers in the field cited security as one of the grand challenges in future computer architecture research. Already the number of security papers published in computer architecture conferences and journals is beginning to rise. While the architecture community has a long history of evaluating performance optimizations, the need to evaluate the security of a proposed architectural mechanism is a relatively new challenge. In surveying recent conference publications in this area, we found that authors generally: 1. Download existing vulnerable programs and write exploits for them 2. Provide sensitivity results they feel imply security 3. Provide security validation results, but explain why it theoretically should be secure As is described in more detail in the next section, these methods of validating security have serious weaknesses. To assist architects in evaluating the security of proposed hardware modifications, we have developed the BASS benchmark suite. BASS is a suite of security benchmarks that is comprised of vulnerable C programs and scripts to automatically generate exploits for those programs across both x86 and Alpha platforms. Each benchmark in the BASS suite has been designed to result in different architectural behavior across dimensions such as vulnerability class, memory location of attack, end result of attack, and injected code location. Compared with current approaches, BASS provides a more thorough, yet easy to use method of evaluating the security features of computer architecture ACM SIGARCH Computer Architecture News 26 Vol. 34, No. 4, September 2006

2 The rest of this paper is organized as follows. In Section 2, we discuss our motivation in an effort to explain why such dimensions need to be explored. Section 3 briefly introduces the background and current difficulties architects face. In Section 4, we provide an overview of the intended use, and in section 5 we present the BASS framework and introduce the benchmarks that make up the suite. Section 6 discusses other related work in this area, and finally in Section 7 we conclude with some remarks and outline our future work. 2. Motivation While it would be unacceptable to design a new branch predictor and declare its benefit to performance without thoroughly verifying its contribution through simulation, such practices occur frequently in architectural security research. Most often, authors will download a small set of real programs with kwn vulnerabilities and exploit those in an attempt to verify the security of their modification. Unfortunately, this set is often chosen without regard to the type and breadth of attack dimensions that are being tested across. In some cases the author may reveal the broad classification of attack type (e.g. format string or buffer overflow ), but usually they do t provide more complete details of the exploit (e.g. buffer overflow in the bss section of memory overwriting a functional pointer resulting in a n-authorized modification of instruction flow ). By igring the diversity of attack dimensions when selecting security benchmarks, one can miss serious, difficult to detect flaws in the security implementation. This would be similar to evaluating a performance modification by only running gcc multiple times with different input sets. As a simple example, assume an intrusion detection system that detects amalies by monitoring system calls [4] and generating an alarm whenever a system call occurred that was t within its set of acceptable system call behavior. Figure 2 shows three sample system call traces from the same program. The first set are system calls that were generated during rmal behavior, the second set are system calls generated during a buffer overflow attack in which a functional pointer is overwritten to gain unauthorized access, and the third set are system calls generated during a buffer overflow attack in which shellcode is injected to gain unauthorized access. The amalous system calls are highlighted. Figure 2. System call traces on rmal program and under two different exploit scenarios Since the buffer overflow attack that overwrites the functional pointer introduces additional code, and only executes the native program code in a different order, it results in additional system calls whereas the shellcode injection method results in many amalous system calls. Even though both of these examples are buffer overflows on the same program, it is much more likely that our hypothetical security implementation would catch the shellcode injection form. While the above illustrated attack scenarios are straightforward, there are many more crafty exploits to be found in recent literature where two semantically similar attacks at the software layer result in vastly different architectural characteristics. In [5], Dalton et al described a format string attack that allows an attacker to circumvent all currently available forms of taint analysis [6] by designing malicious strings using data available within the stack as opposed to usergenerated strings. To defeat these increasingly ingenious attacks, it is essential for the future of architectural security research that a baseline collection of different vulnerabilities and exploits be brought together to quickly provide a first round analysis of architectural security solutions. We believe that the BASS suite can offer such a tool to future security research. 3. Background There are many reasons for the inadequacies of security evaluation in the architecture community. First, security focused research is a relatively new phemen in computer architecture. Whereas performance driven evaluation has a long-standing tradition in architecture and authors have widely accepted tools such as the SPEC suite [7] to use for evaluating performance, there really is equivalent in the area of security. If a researcher wants to evaluate the security of a proposed architectural enhancement, he or she either has to design custom pairs of vulnerable programs and exploits, or download real programs with vulnerabilities and write exploits for them. Both of these approaches can require exorbitant amounts of time and effort. In both cases the author needs to be an expert in many forms of attack, either to program the benchmarks or to locate a broad range of different benchmarks. Locating vulnerable real programs can be difficult because in many cases after a vulnerability for a program is discovered, a new version of that program is released, and so finding vulnerable programs often means tracking down old versions of the code. Once a program with a vulnerability is obtained, the researcher must then write an exploit to take advantage of that vulnerability. This entire process is made even more difficult if the researcher is using a simulator built on top of the Alpha ISA which has been widely adopted by the computer architecture research community. In August of 2004, HP anunced that they had plans to release any future versions of the Alpha processor, and thus software is less readily available for processors based on the Alpha ISA. Consequently, there are substantially fewer attacks targeted at software compiled for the Alpha ISA due to its relative obscurity as compared to the ACM SIGARCH Computer Architecture News 27 Vol. 34, No. 4, September 2006

3 x86. Finally, software running on Alpha systems uses a 64-bit memory address space, making attacks an order of magnitude more difficult than they would be for an equivalent x86 (due to the large number of \x00 bytes found within memory addresses). To facilitate simulation based studies, we have implemented automatically generating exploit scripts targeted t only on the 32-bit x86, but also the 64-bit Alpha ISA within the BASS framework. This will allow security architecture researchers to immediately take advantage of the BASS framework on both readily available full system simulation frameworks based on the Alpha ISA (e.g. m5 [9]), as well as real world production machines based on the more common x86 ISA. 4. Intended Use The purpose of the BASS framework is to improve the overall quality of the evaluation of architectural security enhancements while at the same time reducing the amount of overhead that must be spent by the researcher to perform that evaluation. To that end, the BASS framework provides a base line set of diverse security benchmarks that are automatically customized to the x86 or Alpha Linux platform that a researcher wishes to evaluate. To perform the evaluation, all that is required from the researcher is to download the suite ( compile the benchmarks, run the appropriate script to generate the exploit on the target machine, and then execute the exploit to evaluate their designs effectiveness at identifying/stopping the exploit. Since these are synthetic benchmarks targeted at evaluating the security of an architectural modification, they are t intended to provide diverse performance characteristics and thus a researcher interested evaluating the performance of an architectural modification should still use an appropriate performance benchmark suite such as the SPEC suite. 5. The BASS Framework This section describes the BASS framework. Section 5.1 provides an overview of the entire framework. Section 5.2 illustrates how to use the framework. The individual vulnerability/exploit combinations that comprise each individual benchmark are described in Section 5.3. Finally, section 5.4 explains how to interpret the results from running the BASS suite. 5.1 Overview The BASS suite has been designed to facilitate the security evaluation of proposed architectural security solutions while ensuring the thoroughness of evaluation by addressing attack dimension diversity. All attacks are t created equal and even within a broad attack classification such as buffer overflow or format string attack there can be great diversity. This diversity is amplified at the architectural level, and the attacks designed for the BASS suite have been chosen to reflect this diversity in classification, attack location, code injection, payload location, and final result. The BASS suite consists of a set of vulnerable programs that contain vulnerabilities, and automatic exploit generating scripts. All vulnerable programs within the suite are designed to provide some legitimate functionality. This allows the researcher to generate a rmal signature of n-malicious program flow which can be used in the evaluation of amalous intrusion detection systems. Examples of vulnerable programs within the BASS framework include a lottery game, group messaging system, secure logging program and a simple finger program. To facilitate deterministic simulation, all benchmarks are designed to accept input either from the command line or standard input, and thus can be easily scripted within a full-system simulation framework. In addition to providing a set of vulnerable programs, the BASS framework also provides scripts to automatically generate the different exploits needed to attack those programs on different machines and across both the x86 and Alpha platforms. Except where ted, these scripts use only information that would be available to a malicious attacker and debugging information needs to be compiled into any of the vulnerable programs. To allow for direct comparison between different methods of exploitation, there is purposely t a one to one correspondence between exploit and vulnerability. This means that while each rmal program has one primary vulnerability, there may be multiple ways to exploit that vulnerability that result in vastly different architectural scenarios. This also reduces simulation time since the same program rmal signature can be reused to study multiple attack dimensions. Moreover, it is our hope that BASS will also serve as an educational tool by providing detailed descriptions of various types of vulnerability and attack. To this end we have thoroughly documented each of the vulnerable programs with particular emphasis placed on the vulnerability. We have also included a DEBUG_MODE preprocessor which can be passed during compilation that will allow the researcher to view key memory locations before, during, and after an attack. We hope that this will aid future researchers that discover weaknesses within their design based on the results of the BASS evaluation in understanding the nature of the attack and thus allow them to enhance their modification to provide increased security. While the BASS framework has been designed to provide a diverse, first round analysis of proposed security mechanisms, it is important to remember that BASS is comprised of security specific synthetic benchmarks, and thus should be used in conjunction with real programs to fully evaluate security/performance tradeoffs. In addition to this, we also feel that one of the most important contributions of the BASS framework is to provide a centralized collection of different security benchmarks that can be expanded as new, diverse and crafty methods of exploitation are discovered. We hope that this will help to prevent new architectures from sharing weaknesses that were discovered in previous architectures simply because a paper describing a cunning attack was overlooked or forgotten. ACM SIGARCH Computer Architecture News 28 Vol. 34, No. 4, September 2006

4 5.2 Using BASS The BASS suite is targeted primarily for architectural security researchers using either native hardware or full system simulation environments (although software researchers may find it to be of value as well). While many of the benchmarks can be ported to run in a n-full system simulator (e.g. Simplescalar [8]) that faithfully represents the memory structure of a real system, other attacks are intimately tied to the system and exploit features such as environmental variables and other aspects of the system which may t be available in system call emulation simulators. We have tested the framework on a Pentium 4/Ubuntu 6.06/Kernel , a Pentium 3/Fedora Core 3/Kernel , an Alpha EV6/Debian 3.1/Kernel , and on an unmodified version of the M5 full system simulator [9] which boots Linux Kernel Each exploit within the BASS framework is stored in its own directory, and the directories are stored in a tree within the tar archive; where the first level describes the broad class of attack, the second level describes the area of memory that the attack targets, and the third level describes the specific attack. Each directory contains, at a minimum, the vulnerable program source code, a generate.sh script that automatically generates an exploit targeting the vulnerable program customized for the host platform, a runme.sh script which will both generate and immediately execute the attack, and a detailed README file that explains the rmal operation of the program, how to use the generated attack, and the theory behind how that particular exploit works. All BASS framework benchmarks share a common make/generate/run interface to simplify the evaluation process. To use the BASS framework, you simply need to change into the directory of the exploit you wish test, type make to compile the source code for your architecture, and run the generate.sh script on the machine you wish to test to generate the attack for your specific machine/architecture configuration (the script should automatically detect which ISA you are running the script on). The output of the script will be a command you can invoke to exploit the vulnerable program (which can be entered directly or added to a script). For exploits that store shellcode in environmental variables, a shell will be spawned with the appropriate environment and you can invoke the command from within this environment. Also included for all benchmarks is a runme.sh script which will automatically generate and immediately run the attack. Please see the individual exploit README files for more in-depth coverage on each benchmark, as well as troubleshooting tips for specific environments. While the general goal of the suite was to keep the rmal programs as simple as possible (while maintaining a legitimate function) for the sake of compatibility, simulation speed, and easier legibility; we have attempted to design the rmal programs in a limited modular fashion so that vital functions (such as encryption or database accesses for bank accounts) can be swapped out with more complex functions or external applications if they can be supported by a specific system. 5.3 The Benchmarks Table 1. Benchmark Description This subsection discusses the individual benchmarks (listed in Table 1). Since each benchmark consists of a rmal program / exploit script pair, and there may be more than one attack per vulnerable program, we organize this subsection by first discussing the vulnerable program and then the various methods of exploiting it. For more detailed information, we recommend that users refer to the README file accompanying each benchmark. Benchmark Vulnerable Program 01 lottery buffer overflow 02 lottery buffer overflow 03 message_wall buffer overflow 04 small_finger buffer overflow 05 secure_log format string 06 secure_log format string 07 secure_log Vulnerability Attack / End Result Location format string (environment variable) overwrite function pointer / manipulate instruction flow overflow variable / modify bank account total overflow file pointer / add malicious root account overflow return address / spawn root shell read memory location / access cryptographic key write memory location / modify cryptographic key overwrite deconstructor / spawn root shell bss bss heap stack data data dtors Injects Code (shellcode location) yes (stack) yes (environment variable) Lottery This program is designed to emulate a simple lottery game in which a player chooses a number and the computer picks a random number, and if those two numbers match, the player wins a predetermined number of credits. This program contains a buffer overflow in the bss section of memory since the argument supplied from the user is copied directly into a ACM SIGARCH Computer Architecture News 29 Vol. 34, No. 4, September 2006

5 statically declared global buffer of predetermined size. If a malicious user supplies an argument that is larger than that predetermined size; he or she can overwrite variables stored subsequently in the bss section of memory. In the case of our vulnerable program, the next two variables stored within the bss section are a functional pointer to the play function and the variable which stores the user s current bank account total. The first benchmark attack is designed to overwrite the functional pointer that points to the play function, which is responsible for generating a random number and comparing the user s number to that random number. First, the number of bytes required to overflow the buffer must be determined. Once this is determined the functional pointer can be overwritten; however if any arbitrary address is used, it will likely just lead to a segmentation fault. To provide a more interesting result, the attack script performs an object dump on the lottery binary to determine the location of the win function. Once this is done, the script accounts for the correct Endean format of the host, and outputs a string which when passed to our lottery program will overwrite the play function with the win function. The lottery program can w be won every time it is played. This attack is particularly interesting because it achieves the desired result using only the programs original code by simply overwriting a pointer and thus redirecting the rmal control flow. The second benchmark attack is similar to the first attack, except this time instead of overwriting the functional pointer and redirecting the rmal instruction flow, we overwrite the bank variable that stores our total credits directly. This attack results in an even more powerful attack that is harder to detect since we can achieve any account total in a single overflow without modifying any of the instruction sequence Message_wall This program is similar to a global graffiti or message wall that was once popular on many dialup bulletin board systems. Also similar to modern chat systems, it allows many users across the system to post messages to, and read messages from, a single protected file. The program itself acts as a gateway to this file, and thus requires elevated privileges to access the file. When a user posts a message, the message is taken as an argument from the user and the message and the user s login name (queried from the system) are appended to the protected file. This program contains a potential buffer overflow in the heap section of memory. A buffer is allocated for storing the user input and subsequently input is taken directly from the user and copied into that buffer without a bounds check; thus the user has the potential to overflow the space allocated for user input. The third benchmark attack is derived from [10] and presents an interesting scenario in which a malicious root account can be created on the system by overflowing the buffer and overwriting the location of the pointer to the protected file. First, the length required to overflow into the file pointer must be determined, and the file pointer is overwritten to point to the systems passwd file. Since we are only able to append to the file, the data used to pad out the overflow must follow the syntax required by the passwd file, which results in some interesting problems. These issues are explained in depth in the README file. In the end, our attack subverts the program and redirects the output of the program to the passwd file and creates a malicious root account on the system with a blank password Small_finger This program is a simplified version of the finger program found on many systems. Our version of finger allows each user of a system to post a single message that other users of the system can query using the finger program. This program has a stack buffer overflow vulnerability located within the modify_db function which is invoked when a user wants to add or modify his or her message. This vulnerability is again the result of a direct copy from user input into a buffer of predefined length without a bounds check. The fourth benchmark attack is the common stack smash, shellcode injection attack. When a function is called, space is allocated on the stack for the local variables. In this attack, we first determine the length required to overflow the local variable and overwrite the return address stored on the stack. In the case of x86, this will be the return address of the current function, and on the Alpha ISA this will be the return address of the main function (since the stack of an Alpha is similar to SPARC in that the local variables are stored after the local return address). Once this is determined, the buffer is filled with a number of NOPs, assembly code to spawn a root shell (shellcode), and finally an address within the range of NOPs is supplied so that we overwrite the return address with an address located within our own buffer on the stack and force the execution of the code to spawn the root shell. Details on alignment issues, shellcode generation, locating the address of the buffer relative to the stack pointer, and x86 versus Alpha characteristics can all be found within the README file Secure_log This program is designed to provide a gateway for users and programs of a system to generate messages to a global system log. In addition to allowing one way access to the global system log, as an added security measure, the secure_log program also encrypts the message using a cryptographic key stored within the memory space of the program. After a user/program submits a message to secure_log (either in the form of an argument or a file), the program confirms acceptance of the message by repeating the message to the user. This program contains a format string vulnerability because in the confirmation stage the program passes the input from the user directly to the printf function in the form: printf(userinput);. The fifth benchmark attack takes advantage of the fact that the printf function does form of type checking on its input. First, a memory location that the malicious agent would like to read is chosen and added to the end of the string supplied to the printf function. Next, the %s format parameter is added to the beginning of the string. Since printf has means of type checking, it assumes that the %s points to a location further on ACM SIGARCH Computer Architecture News 30 Vol. 34, No. 4, September 2006

6 the stack. This allows us to read arbitrary locations in memory. All that remains is to use direct parameter access to advance the location of the stack pointer to the memory address stored at the end of our string, and the printf function will print out the contents of that location in string format (in the case of our benchmark, the cryptographic key). The sixth benchmark attack works in a similar manner, only using the %n format string parameter to modify a location in memory. Since %n writes the number of bytes output thus far to a location pointed to in the stack, we can write arbitrary values to memory by outputting a carefully chosen number of characters followed by a %n with a direct parameter access number that has been determined to align with the memory address stored at the end of the string. Using this method, which is explained in greater detail in the README file, we can change the cryptographic key to any key of our choice. The seventh benchmark builds upon the method of writing data to arbitrary locations in memory discussed in the sixth benchmark. This time the target memory address is the.dtor section of memory. This writable section of memory, found in all Gnu compiled binaries, contains a pointer to a deconstructor function that is called by the program when it exits. First, shellcode designed to switch to root user mode and spawn a shell is generated and stored into a well aligned environmental variable. Next, the location in memory of this environmental variable is determined, and this address is written to the location of the.dtor section in memory using the %n process discussed above. When the program subsequently exists, it calls the deconstructor function which w points to the environmental variable, and executes shellcode to spawn a root shell. 5.4 Interpreting the Results The evaluation results obtained by running the BASS benchmark suite differ somewhat from the results obtained by running traditional performance oriented benchmarks. Whereas architects are generally most interested in quantitative metrics that vary within a range for performance benchmarks, security evaluation results take a much more boolean approach. Each benchmark within the BASS suite has been designed to exploit the vulnerable program in a unique way, and thus each benchmark can be considered a separate pass/fail test for the targeted security enhancement. A security enhancement is awarded a pass for a benchmark if it is able to effectively prevent the exploit from breaching the security of the vulnerable program in its prescribed way. If the security enhancement is unable to detect or deter the exploit from succeeding, then the enhancement has effectively failed for that benchmark. This allows the researcher to gain a fundamental understanding as to the individual strengths and weaknesses of an architectural security design. While the ratio of the total number of passed to failed benchmarks can be consider a sort of quasi-quantitative metric for describing the overall security of a proposed architectural enhancement, we feel that it is much more beneficial to individually report the results of each benchmark as this provides a much more valuable result. Table 2. Benchmark Exploit Success Conditions Benchmark 01 (lottery) 02 (lottery) 03 (message wall) 04 (small finger) 05 (secure log) 06 (secure log) 07 (secure log) Result of a Successful Attack Output should indicate the lottery game has been won with credits subtracted Output should display current bank account total as $1337 File passwd in same directory should have an additional account line appended to it A new shell should be spawned. If the vulnerable program has the root setuid flag, this new shell should have root privileges The secret cryptographic key should be output to screen (the default cryptographic is VeryWeak ) The secret cryptographic key should be overwritten (this can be verified by passing the DEBUG_MODE ) A new shell should be spawned. If the vulnerable program has the root setuid flag, this new shell should have root privileges Table 2 provides the result of a successful attack for each exploit benchmark (and thus if this result is achieved while a security enhancement is in place, a failure condition for that security enhancement). It is recommended that each benchmark be tested on the host without the security enhancement under evaluation to ensure the attack is functioning properly (the troubleshooting section of the accompanying README files addresses the most common sources of failure across different systems). As is shown in the table above, the result of a successful attack for most of the benchmarks can be easily observed from the standard output of the program or by examining the contents of a single file. The only exception to this is benchmark 06 which overwrites a secret cryptographic key stored in memory. The success of this benchmark can be determined in one of several ways. The least invasive but most difficult method would be to externally examine the memory location that stores the key after the exploit has completed (or alternatively to attempt to decrypt the last line of the generated syslog file using the new malicious key). A far simpler approach is to pass the DEBUG_MODE preprocessor when compiling the secure_log program which will output to standard output the contents of the memory location storing the cryptographic key before and after the attack. ACM SIGARCH Computer Architecture News 31 Vol. 34, No. 4, September 2006

7 6. Related Work The challenge of evaluating the security of an entire computer system is t a new one. The National Institute of Standards and Techlogy (NIST) and the National Security Agency (NSA) have developed the Common Criteria Evaluation and Validation Scheme (CCEVS) [11], and many European governments have agreed to recognize the validity of Information Techlogy Security Evaluation Criteria (ITSEC) [12] evaluations. However these procedures are generally designed to confer assurance upon secure systems and therefore they can t be directly applied at the architecture level. Some of the early work on developing a set of benchmarks to evaluate computer architecture security mechanisms was made by [13]. In this effort, the authors develop a set of benchmarks called SecSpec that can be built to either include or exclude code that contains suspicious behavior for use in testing behavior-based security mechanisms. Our work differs in that the SecSpec suite focuses on the broader area of malware (including spyware) and emulates general suspicious behavior such as copying files, downloading remote code, etc; whereas the BASS suite focuses on exercising actual vulnerabilities. In [5, 10, 14, 15, 16, 17], the authors showcase individual new attacks designed to circumvent current security measures. One of the goals of the BASS suite is to provide a common framework to integrate new forms of attack so that they are t lost over time, and so that new security mechanisms can build upon the oversights of past systems. [10] presents an excellent educational overview of many forms of attack, including theory behind designing future attacks, and several of the unique end results as well as vulnerable programs within our BASS suite were inspired from this work. 7. Conclusions and Future Work In this paper, we have presented BASS, a benchmark suite designed to facilitate the evaluation of architectural security mechanisms while ensuring the quality of evaluation by addressing vulnerability/attack diversity. Our suite has been designed and validated to run on native machines as well as full-system architecture simulators, and provides both vulnerable programs as well as automatically generating exploit scripts which are able to generate exploits targeted at both 32-bit x86 and 64-bit Alpha platforms. The BASS suite currently contains a set of seven diverse vulnerabilities and exploits which all employ an easy to use standard interface. The current BASS framework including documentation, source code, input data sets and pre-compiled binaries for the M5 full system simulator can be freely downloaded at Our future work includes developing other interesting and diverse attacks to expand the suite to cover other forms of vulnerabilities such as race conditions and command injection parsing attacks, as well as to continue to add unique attacks designed to circumvent current architectural solutions. A more long term goal would be to port the current suite into other ISA s and operating systems. We believe that BASS introduces a more thorough, yet easy to use method of evaluating security that will allow the architecture research community to focus its energy on hardware security solutions that will result in successes with this grand challenge akin to those the community has achieved in performance. Ackwledgement This work is supported by a National Science Foundation Graduate Research Fellowship and the Microsoft Research Trustworthy Computing Award No References [1] [2] Microsoft Windows Vista, [3] Revitalizing Computer Architecture Research, CRA Conference on Grand Research Challenges, Dec. 2005, re/home.html [4] Stephanie Forrest, Steven A. Hofmeyr, Anil Somayaji, Thomas A. Longstaff, A Sense of Self for Unix Processes, In Proceedinges of the IEEE Symposium on Research in Security and Privacy, [5] M. Dalton, H. Kannan, and C. Kozyrakis, Deconstructing Hardware Architectures for Security, Fifth Annual Workshop on Duplicating, Deconstructing, and Debunking (held in conjunction with the 33rd International Symposium on Computer Architecture), [6] J. Newsome and D. X. Song, Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. In Proceedings of the Network and Distributed System Security Symposium, [7] SPEC CPU 2000 Benchmarks, [8] SimpleScalar, [9] [10] J. Erickson, Hacking: The Art of Exploitation, San Francisco: No Starch Press, [11] Common Criteria Evaluation & Validation Scheme (CCEVS), National Institute of Standards and Techlogy. [12] Information Techlogy Security Evaluation Criteria (ITSEC), Commission for the European Communities, [13] D. Ye, M. Moffie and D. Kaeli, A Benchmark Suite for Behavior-Based Security Mechanisms, In Proceedings of the Workshop on Software Security Assurance Tools, Techniques and Metrics, [14] T. Oh, Buffer Overflow Exploit in the Alpha Linux, Taeho Oh, ohhara@postech.edu, ACM SIGARCH Computer Architecture News 32 Vol. 34, No. 4, September 2006

8 [15] S. Seo, Format String Attack on Alpha System, [16] J. Rivas, Overwriting the.dtors Section, [17] D. Wagner and P. Soto, Mimicry Attacks on Host-Based Intrusion Detection Systems, In Proceedings of the 9th ACM Conference on Computer and Communications Security, ACM SIGARCH Computer Architecture News 33 Vol. 34, No. 4, September 2006