HEALERS: A Toolkit for Enhancing the Robustness and Security of Existing Applications Christof Fetzer, Zhen Xiao AT&T Labs Research 180 Park Avenue Florham Park, N.J. 07932 christof, xiao @research.att.com Abstract HEALERS is a practical, high-performance toolkit that can enhance the robustness and security of existing applications. For any shared library, it can find all functions defined in that library and automatically derives properties for those functions. Through automated faultinjection experiments, it can detect arguments that cause the library to crash and derive safe argument types for each function. The toolkit can prevent heap and stack buffer overflows that are a common cause of security breaches. The nice feature of the HEALERS approach is that it can protect existing applications without access to the source code. Keywords: reliability, robustness, security, wrapper, fault tolerance 1 Introduction Software reliability is becoming increasingly important as our daily lives become more and more computerized. Many financial transactions nowadays are conducted using computers through the Internet. Big companies (such as AT&T) employ on-line billing to reduce their operational costs. Both ACM and IEEE offer free on-line courses as a flexible way to let people acquire new knowledge during their spare time. Virtual office environment allows people to work comfortably at their homes. We expect this trend to continue in the years to come. At the same time, there is a growing concern on the correct functioning of computer software under exceptional or stressful settings. If a computer crashes, hangs, or gives erroneous output, it may lead to severe disrup- This paper appears in the Proceedings of the IEEE International Conference on Dependable Systems and Networks (DSN), June, 2003. tion of service or loss of productivity. The consequences may be costly due to the increasing reliance on computer technology in our society. Unfortunately, previous studies have found that many existing softwares are not very reliable. The Ballista project, for example, found that many implementations of operating systems and C libraries are brittle with respect to invalid inputs [6]. The authors of [2] studied the bug reports of three open source softwares (Apache, Gnome, and MySQL) and found that many deterministic bugs exist long after their initial releases. One reason for this is that new features are constantly added to these softwares without thorough testing. We believe that such phenomena are common for commercial softwares as well: while many applications desire reliability, they cannot afford to sacrifice functionalities in the current competitive market. Given the limited development budget in today s economy, it may be prohibitively expensive to test all boundary conditions before the official release of the software. A good protection method for Commercial Off-theshelf software should satisfy the following requirements: transparency: The method should provide transparent protection for existing software without requiring recompilation or relinking. Since the source code of commercial software is generally not available, any method that requires modification of the source code is undesirable. adaptivity: Due to the fast software update cycle in today s society, the protection method should be able to adapt quickly to new software releases. cost-effectiveness: The method should be able to eliminate a large class of software failures in a costeffective manner. Note that a significant portion 1
of the cost involved in hardening software is operational cost: the cost of a computer programmer is usually much higher than the cost of a group of high-end PCs. Any method that is labor-intensive will not scale well in a production environment where many pieces of software need to be protected. flexibility: Different applications may have different reliability and security requirements and need different levels of protection. An one size fits all approach would not work. Hence, a good protection method should be flexible so that it can be configured to provide only the specific features an application needs. low overhead: The method should have low overhead during normal operations. In addition, an application should only pay the overhead for the protection it actually needs. Any method that provides excessive or overly restrictive protection may not work well in practice. In this paper, we describe a software toolkit called HEALERS (HEALers Enhanced Robustness and Security) that demonstrably satisfies the above requirements. HEALERS can provide effective protection for existing software without source code access. It has a flexible architecture and can be configured to suit the needs of individual applications. It is highly-automated and can easily adapt to new software releases. Its run time overhead is small for most applications. The rest of the paper is organized as follows. Section 2 describes the technology used in HEALERS. Section 3 gives an outline of our software demonstration. Section 4 concludes this paper. 2 Technology HEALERS is designed to protect applications from a large class of software errors related to C library functions. Its protection method has the following key components: interception: Our software intercepts function calls into the C library. This allows it to provide transparent protection for software that is dynamically linked. Previous studies indicate that a major category of software failures related to C library functions are API failures: function calls with invalid arguments may cause the calling process to terminate abnormally [6]. Intercepting such function calls gives us a convenient way to check the validity of their arguments and hence avoid such failures. automation: HEALERS uses automated faultinjection experiments to discover robustness and security problems in software from third-party vendors [4]. Based on the knowledge gathered from these experiments, it automatically generates a set of fault-containment wrappers to correct a large set of such problems. This proves to be a cost-effective approach for protecting a wide variety of software products and for adapting to new software releases. flexible generation: The HEALERS toolkit is highly configurable and can be tuned to satisfy the individual needs of different applications. It generates a variety of fault-containment wrappers through a set of micro-generators. Such an architecture facilitates code reuse and makes it easy to introduce new functionalities into the existing system [5]. In the rest of the section, we will describe each of these components in detail. 2.1 Interception Our software is implemented as a dynamically loadable C library wrapper. The wrapper sits between an application and the C library. It intercepts every C library function call from the application. On most Unix systems a user interested in using a wrapper can preload it by defining the LD PRELOAD environment variable. This is useful for protecting certain network services. In addition, a system administrator can enable a wrapper on a system wide basis through a dynamic link loader. Since different applications may have different security or reliability requirements, they can use different types of wrappers to satisfy their needs. This is illustrated in Figure 1. For example, a process with root privilege may use a security wrapper that can prevent buffer overflow attacks [3]. Previous studies indicate that such kind of attacks are a major cause of security breaches in operating systems [1]. A user application that desires high availability can use a robustness wrapper that prevents a large class of software failures (e.g. crashes, hangs, or aborts). Some application may want to log the errors that occurred during its operation for later failure diagnosis. In this case, they can use a profiling wrapper that gives a detailed report on what kind of errors occurred, how frequently they occurred, and what were the causes of errors (based on errno). The figure also shows that different applications may share the same wrapper, or may use different wrappers. A nice feature of wrapping dynamic link libraries is that it can protect existing software without source code access. Note that it only works for applications that are 2
root process user application user application user application security wrapper robustness wrapper profiling wrapper Shared Libraries Figure 1. A wrapper sits between an application and its shared libraries. Different applications can use different wrappers, or can share wrappers. dynamically linked. In practice, we found that this is not a problem because the overwhelming majority of applications are dynamically linked. Another limitation of our approach is that it can only protect errors due to C library function calls. However, such errors consist of a large percentage of software errors observed in deployed applications. Hence, we believe it is important to reduce such errors. 2.2 Automation One way to increase the security and robustness of software is by visually examining the source code to detect any loopholes. For example, we can examine the control flow of the program to see whether it ever accesses an invalid pointer or whether it performs boundary checks when performing a string copy operation. This approach, however, is labor-intensive: the human cost for examining a large software product can be huge. Moreover, whenever there is a new software release or update, it has to be examined again. With the price of computer hardware decreasing rapidly, the key to providing cost-effective protection to computer software is automation: let the computers do the work. Our HEALERS system discovers the robustness problems in C libraries through automated faultinjection experiments as illustrated in Figure 2. The system parses the header files and manual pages from C libraries to generate the prototype information for all global functions. Note that the API defined by the prototype of a function is not always robust. For example, the prototype of the strcpy function specifies its first argument to be char *. However, it actually has to be a pointer to a writable buffer with enough space to accommodate the source string. Our system searches for the weakest robust argument types for a function by repeatedly probing the function with a hierarchy of function types until it finds one that does not result in robustness failures. By doing so, it generates the robust API for a shared library. Although some manual editing may be needed, this process is largely automated. 2.3 Flexible Wrapper Generation One challenge in our system design is how to generate a large number of wrappers to suit the needs of different applications. Apparently, it is undesirable to write every wrapper from scratch. Our system provides a flexible framework for a wide variety of wrapper types. This is achieved through a modular approach where the functionality of a wrapper generator is decomposed into a number of features, each supported by a microgenerator. Each micro-generator generates a fragment of the prefix and postfix code of a function. The microgenerators can be combined in a variety of ways to generate new wrapper types. Figure 3 depicts the wrapper function wctrans. This code is generated by combining six microgenerators: prototype, function exectime, collect errors, func errors, call counter, and caller. The prototype and caller are standard micro-generators that generate the prototype of the wrapper function and the call to the original function, respectively. The rest microgenerators generate code to collect various statistics during the program execution. The micro-generator also generates code for collecting the results. Just before the 3
searching robust argument types Shared Library Function Prototypes Automated Fault Injection Experiments Robust API Figure 2. HEALERS discovers robust API for shared libraries through automated fault-injection experiments. application terminates, the collection code is called to send the gathered information to a central server. Since different types of wrappers can be used in a distributed environment, the gathered information sent to the server is in form of a self-describing XML document. The server can extract from the document which functions were wrapped and what kind of information was collected. Such information is then stored for later processing. Figure 5 shows an example of collected data. 3 Demonstration We will demonstrate the following features of our software. 3.1 Wrapping Libraries Our toolkit can list all libraries in the system. A user can select one of the libraries. Our system will then generate a list of all functions defined in the library. In addition, our system will create a XML-style declaration file that describes the prototype of each function in the library. The results of such experiments can be used to generate various wrappers as described in [5]. 3.2 Wrapping Applications Figure 4. HEALERS automatically extracts the list of libraries linked to an application and the list of undefined functions. The previous demo is library-centric. In contrast, this demo is application-centric. It allows a user to browse through the list of files in the current system and select an application program (i.e. an executable file). Our toolkit can automatically extract the list of libraries linked to this application as well as the list of undefined functions in the application. The Web interface for this demo is illustrated in Figure 4. 3.3 Profiling In this demo, we show that our profiling wrapper can gather statistics related to the execution of a program and automatically generate graphics to display the result as shown in Figure 5. A user executes a program in the system with our profiling wrapper. Upon termi- 4
/* Prefix code by micro-gen prototype */ wctrans_t wctrans(const char* a1) wctrans_t ret; /* Prefix code by micro-gen function exectime */ unsigned long long exectime_start; unsigned long long exectime_end; rdtsc(exectime_start); /* Prefix code by micro-gen collect errors */ int collect_errors_err = errno; /* Prefix code by micro-gen func error */ int func_error_err = errno; /* Prefix code by micro-gen call counter */ ++call_counter_num_calls[1206]; /* Postfix code by micro-gen caller */ ret = (*addr_wctrans)(a1); /* Postfix code by micro-gen func error */ if (func_error_err!= errno) if (errno < 0 errno >= MAX_ERRNO) ++func_error_cnter[1206][max_errno]; else ++func error_cnter[1206][errno]; /* Postfix code by micro-gen collect errors */ if (collect errors err!= errno) if (errno < 0 errno >= MAX ERRNO) ++collect errors cnter[max ERRNO]; else ++collect errors cnter[errno]; /* Postfix code by micro-gen function exectime */ rdtsc(exectime end); exectime[no FUNCTION] += exectime end-exectime start; /* Postfix code by micro-gen prototype */ return ret; Figure 3. Wrapped function for abs. 5
We have described the HEALERS toolkit for increasing the robustness and security of existing software without source code access. The toolkit intercepts C library function calls and can provide transparent protection to many software systems. It uses automated fault injection experiments to detect robustness and security problems in a given software. Based on the knowledge gathered from such experiments, it automatically generates software wrappers to fix a large percentage of such problems. The architecture of our system is flexible in that it can be configured to generate a variety of wrappers to satisfy the needs of different applications. References [1] Arash Baratloo, Navjot Singh, and Timothy Tsai. Transparent run-time defense against stack smashing attacks. In Proceedings of USENIX Annual Technical Conference, June 2000. [2] Subhachandra Chandra and Peter M. Chen. Whither generic recovery from application faults? A fault study using open-source software. In Proceedings of the 2000 International Conference on Dependable Systems and Networks, 2000. Figure 5. The profiling wrapper can collect various runtime information. nation, the wrapper generate a XML-style log file that shows the frequency of function calls in this program, the percentage of execution time in each function, the distribution of function errors, the causes of such errors (classified by errnos), etc.. 3.4 Buffer Overflow Prevention In this demo, we show that our security wrapper can prevent buffer overflow attacks. It first shows that an attacker can hijack the control flow of a root privileged program by overflowing a buffer allocated on the heap. This results in a root shell for the attacker. An example of such a program was previously published in [3]. Then we show that our security wrapper can detect such buffer overflows and terminate the attacker s program. [3] Christof Fetzer and Zhen Xiao. Detecting heap smashing attacks through fault containment wrappers. In Proceedings of the 20th IEEE Symposium on Reliable Distributed Systems, October 2001. [4] Christof Fetzer and Zhen Xiao. An automated approach to increasing the robustness of C libraries. In Proceedings of the International Conference on Dependable Systems and Networks, June 2002. [5] Christof Fetzer and Zhen Xiao. A flexible generator architecture for improving software dependability. In Proceedings of the International Symposium on Software Reliability Engineering,, November 2002. [6] Philip Koopman and John DeVale. The exception handling effectiveness of POSIX operating systems. IEEE Transactions on Software Engineering, 26(9):837 848, Sep 2000. 4 Conclusion 6