Memory-Constrained Security Enforcement

Transcription

1 CHAMSEDDINE TALHI Memory-Constrained Security Enforcement Thèse présentée à la Faculté des études supérieures de l Université Laval dans le cadre du programme de doctorat en Informatique pour l obtention du grade de PhilosphiæDoctor (Ph.D.) FACULTÉ DES SCIENCES ET DE GÉNIE UNIVERSITÉ LAVAL QUÉBEC Avril 2007 c Chamseddine Talhi, 2007

2 Abstract With the proliferation of mobile, wireless and internet-enabled devices (e.g., PDAs, cell phones, pagers, etc.), Java is emerging as a standard execution environment due to its security, portability, mobility and network support features. The platform of choice in this setting is Java ME-CLDC. With the large number of applications available for Javaenabled network-connected devices, security is of paramount importance. Applications can handle user-sensitive data such as phonebook data or bank account information. Moreover, Java-enabled devices support networking, which means that applications can also create network connections and send or receive data. However, the considerable efforts of securing Java ME-CLDC are constrained by strict memory limitations of the target devices. This thesis aims at investigating memory-constrained security by analyzing the security of Java ME-CLDC and characterizing enforceable security policies. More precisely, the main objectives of our research are (1) evaluating and improving the security of Java ME-CLDC and (2) characterizing memory-constrained execution monitoring; an important class of security mechanisms. The main results of our research are the following: A security analysis of Java ME-CLDC. The two main contributions of this analysis are a vulnerability analysis and a risk analysis of the platform. The vulnerability analysis revealed the presence of vulnerabilities in the platform and showed how to improve the underlying security model. The risk analysis provided a seriousness estimation of the risks associated with the uncovered vulnerabilities. A characterization of memory-constrained execution monitoring. This characterization covers conventional monitors as well as more powerful monitors. The contribution of this characterization is mainly threefold. First, we defined a new automata class, called Bounded History Automata (BHA), to specify memoryconstrained EM enforcement. Second, we identified a new memory-directed taxonomy of EM-enforceable properties. Third, we investigated the enforcement of local properties using memory-constrained EM. This was performed by identifying BHA-enforceable local properties and explaining how to check whether an EM-enforceable policy is local or not.

3 Résumé Avec l extension des cellulaires, des réseaux sans fil et des périphériques mobiles, Java est devenu incontestablement l environnement d exécution le plus populaire. Cela est dû à ses aspects de sécurité, portabilité, mobilité et réseaux. Dans ce contexte, la plateforme de choix est Java ME-CLDC. Aussi, vu le nombre grandissant d applications Java destinées aux périphériques mobiles, la sécurité est devenue un enjeu crucial à considérer d une manière primordiale. Sécuriser ce type d applications devient plus qu impératif, notamment lorsque celles-ci manipulent des données confidentielles telles que les informations relatives aux transactions électroniques. Plus encore, les périphériques supportant Java se retrouvent souvent interconnectées, ce qui signifie que les applications peuvent ainsi créer des connexions réseaux et faire circuler des données critiques sur les canaux de communications. Cependant, les efforts considérables déployés afin de sécuriser Java ME-CLDC se heurtent à des contraintes de limitations strictes de l espace mémoire disponible, au sein des périphériques en question. Dans cette optique, cette thèse étudie le problème du maintien de la sécurité sous contraintes mémoire, et cela en analysant la sécurité de la plateforme Java ME-CLDC. Plus précisément, les objectifs majeurs de notre sujet de recherche sont (1) l évaluation et l amélioration de la sécurité de Java ME-CLDC et (2) la modélisation du monitoring d exécution (EM) en y introduisant des contraintes mémoire. À vrai dire, EM constitue une classe importante et omniprésente parmi tous les mécanismes de sécurité utilisés dans les plateformes Java. Les principaux résultats auxquels a abouti notre investigation sont les suivants : Une analyse de la sécurité de Java ME-CLDC. Les deux contributions principales qu a engendré cette analyse sont l analyse de vulnérabilité et l analyse des risques de cette plateforme. L analyse de vulnérabilité a révélé la présence de certaines faiblesses dans la plateforme, elle a montré également la manière permettant d améliorer son modèle de sécurité. Quant à l analyse des risques, elle a fourni une estimation de la gravité des risques associés aux vulnérabilités décelées.

4 Résumé iv Une modélisation du monitoring d exécution sous contraintes mémoire. Cette modélisation couvre aussi bien les moniteurs conventionnels que des moniteurs plus puissants. Les contributions principales qui découlent de notre modélisation sont les suivantes: Premièrement, nous avons défini une nouvelle classe d automates, dite Bounded History Automata (BHA) ou automates à historique borné, classe d automate qui permet de spécifier les mécanismes EM opérant sous contraintes mémoire. Deuxièmement, nous avons identifié une nouvelle taxonomie orientée mémoire des propriétés assurées par EM. Troisièmement, nous avons étudié les propriétés localement testables qui peuvent être assurées par des EMs opérant sous contraintes mémoire. Cela est fait en deux étapes: on commence par identifier les propriétés assurées par EMs qui sont de nature locale, ensuite on vérifie si ces dernières peuvent être spécifiées par des BHAs.

5 Acknowledgements I m very thankful to Pr. Nadia Tawbi and Pr. Mourad Debbabi for their advices, ideas and efforts to ensure a continuous supervision of this thesis. Their insights and encouragements have had a major impact on this research. Working with them was a very valuable experience for me. They deserve all my acknowledgements. I would like to thank Pr. Guy Tremblay, Béchir Ktari, and Pascal Tesson who honored me by accepting to review this thesis. I strongly believe that their questions, remarks and suggestions will be helpful for producing the final version of this thesis. I would like to express my ultimate gratitude to my wife Hakima who shared with me this thesis experience. She has been always happy to listen to and provide a positive feedback to my research ideas. She merits all my love and respect. I would like also to express all my acknowledgements to some exceptional colleagues who shared with me the precious years of my thesis: Hamdi Yahyaoui, Abdelouahed Gherbi, Lamia Ketari, and Mohamed Saleh from the Computer Security Laboratory (CSL) at Concordia University, Sami Zhioua, and Mahjoub Langar from the Languages, Semantics and Formal Methods Group (LSFM) at Laval University. I m also grateful to Lynda Goulet and Rachel Lapierre from the secretariat of the Computer science and Software engineering Department at Laval University. They are so kind and helpful to all the students. Finally, I would like to thank my friends Omar Seddiki, Mohamed Aoun-Allah, and Ashraf Badr for their encouragements, support and solidarity.

6 Acknowledgements vi To my parents To my wife Hakima To my daughter Rym To my family, friends and colleagues

7 Contents Abstract Résumé Acknowledgements Contents List of Tables List of Figures ii iii v vii xii xiii 1 Introduction Motivations Objectives Methodology Studying Java ME-CLDC and its Security Security Evaluation of Java ME-CLDC Characterizing Memory-constrained EM Enforcement Contributions Dissertation Structure I Java ME-CLDC: Platform and Security Model 9 2 Java ME-CLDC Introduction Java ME Platform Java Platforms Java ME Platform Overview Java ME-CLDC Virtual Machines Java Virtual Machine Java ME-CLDC JVMs Examples Executing Java ME-CLDC Applications

8 Contents viii MIDlet MIDlet Development MIDlet Packaging MIDlet Installation MIDlet Lifecycle MIDlet Persistent Storage Connected Limited Device Configuration Goals Requirements Scope Target Devices CLDC Specificities CLDC Libraries Mobile Information Device Profile (MIDP) Goals Requirements Scope MIDP Libraries Optional Packages Wireless Messaging API Mobile Media API Java ME Web Services API Location API for Java ME Conclusion Java ME-CLDC Security Introduction Java Security Sandbox Model Language Type Safety Bytecode Verification Security Policy Security Manager & Access Controller Secure Class Loading End-to-End Security Java ME-CLDC Security Bytecode Verification Off-Device Preverification On-Device Verification Sandbox Model Protecting System Classes

9 Contents ix Restrictions on Dynamic Class Loading Security Policy Sensitive APIs and Permissions Protection Domains Function Groups User Interaction Policy Security Policy File Security Policy Enforcement Requesting Permissions for MIDlet Suites Granting Permissions to MIDlets Trusting MIDlet Suites Signing MIDlet Suites Authenticating a MIDlet Suite Certificate Expiration and Revocation Keystores Persistent Storage Security End-to-End Security Security And Trust Services API Conclusion II Security Evaluation of Java ME-CLDC 74 4 Java ME-CLDC Security Analysis Introduction Overview Vulnerabilities Classification Code Inspection Security Testing Standard Security Evaluation Methodologies Enforcing Security Policies Methodology Advantages and Weaknesses of Java ME-CLDC Security Model Permissions Protection Domains Security Policy RMS Protection Reported Java ME-CLDC Flaws Siemens S55 SMS Problem Sun s MIDP Reference Implementation Problems Uncovered Vulnerabilities

10 Contents x KVM Vulnerabilities MIDlet Lifecycle Vulnerabilities Storage System Vulnerabilities Networking Vulnerabilities Thread Management Vulnerabilities Conclusion Java ME-CLDC Risk Analysis Introduction The MEHARI Approach Phase 1: Security Strategic Plan Phase 2: Operational Security Plan Phase 3: Company Operational Plan Application to Java ME-CLDC Security Strategic Plan Security Plan Operational Plan Conclusion III Memory-Constrained Execution Monitoring Enforceable Security Policies Introduction Definitions Security Policies Safety Liveness Security Mechanisms Static Analysis Execution Monitoring Program Rewriting Enforceable Security Policies Characterization Security Automata Edit Automata Abstraction-based Security Automata Computability Classification Discussion Bounded History Automata and Local Testability Introduction

11 Contents xi 7.2 Characterizing Constrained EM Enforcement Execution History Abstraction Formal Characterization Bounded History Automata Bounded Security Automaton Bounded Edit Automata Bounded-History-Based Taxonomy of EM-Enforceable Policies Memory-Constrained EM and Local Testability Local Properties BSA-Enforceable Local Properties BEA-Enforceable Local Properties Local EM-Enforceable Properties BHA-Enforceable Security Policies SHA-Enforceable Policies Bounded Availability Policies Transaction-based Policies Conclusion Conclusion 183 Bibliography 186 Index 198

12 List of Tables 2.1 The javax.microedition.rms Package Function Groups and User Settings Association of Function Groups with MIDP Permissions Association of Function Groups with Wireless Messaging API Permissions Association of Function Groups with Mobile Media API Permissions Actions Taken by the Certificate Verification Algorithm Authorization Actions Taken on the Basis of Authentication Results Impact Reduction Assessment Table Impact Assessment Table Seriousness Assessment Table Resource Classification Security Policy Disaster Scenarios Disaster Seriousness Assessment Summary of Risk Assessment Results for Disaster Scenarios Risk Assessment Results for Java ME-CLDC Vulnerabilities

13 List of Figures 2.1 Java Applications from Development to Execution Java Platform Editions Java ME Platform Java Virtual Machine Basic Components The Development of the midletname MIDlet MIDlet Packaging MIDlet Lifecycle Record Stores in MIDP The Original Java Sandbox Model The Extended Java Sandbox Model Verifying Classes in Java ME-CLDC Example of a Policy File Syntax Example of a Policy File Trusting a MIDlet Suite and Binding it to a Protection Domain Methodology to uncover Vulnerabilities Buffer Overflow Vulnerability KVM Overflow Vulnerability Exploiting the KVM Overflow Vulnerability MIDlet Lifecycle A MIDlet which does not follow the Expected Lifecycle A MIDlet Expecting MIDP 1.0 Exceptions Unfair Management of Persistent Storage A MIDlet Trying to Obtain all the Available Free Persistent Space on the Device Various Abstraction Levels in Mobile Devices Software A MIDlet Creating an Unshared Record Store A MIDlet Deleting an Unshared Record Store Belonging to another MIDlet Transferring MIDlet Jar File Transferring rms.db File The PRand.generateData Method

14 List of Figures xiv 4.16 The updateseed Method SMS Authorization Dialog The two Threads used to Send Unauthorized SMS Unsynchronized Access to Display Screen Different Phases of Application of the MEHARI Method (Source: MEHARI documentation) A scanner An edit automaton and the corresponding Büchi automaton Identifying local properties enforceable by BHA A bounded security automaton enforcing the Two-BA property A bounded edit automaton enforcing a transaction-based property

15 Chapter 1 Introduction 1.1 Motivations With the proliferation of mobile, wireless and internet-enabled devices (e.g., PDAs, cell phones, pagers, etc.), Java is emerging as a standard execution environment due to its security, portability, mobility and network support features. The platform of choice in this setting is Java ME-CLDC (Micro Edition-Connected Limited Device Configuration) [28, 101]. It is an enabling technology for a plethora of services and applications: games, messaging, presence and availability, web-services, mobile commerce, etc. Moreover, Java ME-CLDC allows the development of highly portable, minimum-footprint Java applications for resource-constrained, network-connected devices. The mobile devices targeted by Java ME-CLDC are mainly mobile phones and entry-level Personal Digital Assistants (PDAs). The number of mobile devices implementing Java ME-CLDC witnesses a huge increase. In fact, the latest list of mobile phones supporting this Java platform shows more than 60 phone models from various manufacturers. These numbers are expected to grow. According to IDC, a prestigious market research firm, only during 2006, more than billion Java-enabled cell phones have been deployed in the market which represents a 22.5 percent increase compared with With the large number of applications available for Java-enabled network-connected devices, security is of paramount importance. Applications can handle user-sensitive data such as phonebook data or bank account information. Moreover, Java-enabled devices support networking, which means that applications can also create network connections and send or receive data. Security in all of these cases should be a major

16 Chapter 1. Introduction 2 concern. Malicious code has caused a lot of harm in the computer world, and with phones having the ability to download and run applications there is an actual risk of facing this same threat. Currently, viruses for phones start to emerge (e.g., Cabir, Skulls-A, Lasco.A, Cardtrap.A, etc.), a number of model specific attacks has been reported (e.g., Nokia 6210 DoS, Siemens S55 SMS, etc.), and attacking mobile devices starts to interest the hacker community (e.g., Aware of the new security risks, the architects of Java ME-CLDC have proposed new security APIs [110] and adopted a real security-in-mind design of all new introduced APIs. However, the considerable efforts of securing Java ME-CLDC are constrained by strict memory and battery power limitations. Indeed, the expansion of battery capacity during the last years was modest and the expansion of mobile devices memory size, although considerable, was outpaced by the growing memory needs for system software and media capabilities. In fact, the development team of the latest Java ME- CLDC virtual machine [91] revealed that (1) Moore s Law does not apply to cell phone battery life, which means that so far, no exponential expansion of battery capacity with the passage of years has been observed and (2) Most of the available memory in a current-generation or next-generation handset is needed for system software and media capabilities. From what precedes, the market of Java-enabled phone is facing a delicate situation: On one hand, Java ME-CLDC is providing more capabilities and services to larger mobile users community, thus attracting more malevolent actors to join the existing mobile phone hackers community by providing them more possibilities and entry points to experiment new attacks. On the other hand, the efforts spent to reinforce the platform security are constrained by strict memory limitations. Accordingly, the following questions have to be asked and need to be precisely answered: (1) What are the security policies actually enforced on Java-enabled devices and how they are enforced? (2) What are the security vulnerabilities of the existing Java ME-CLDC implementations? (3) What are the risks involved by these platform vulnerabilities? (4) How can the security of this platform be improved? Answering these questions requires performing a security analysis of Java ME-CLDC. To our knowledge, there is no published work on the security analysis of Java ME-CLDC. The aforementioned questions can be generalized to the following main question: What are the limits of what can be enforced on a memory-constrained platform? Answering this general question represents a logical continuation of the efforts of some pioneer authors contributing to the emergence of a new research field. This nascent research field targets characterizing security enforcement mechanisms and identifying the classes of enforceable security policies. Particular attention has to be given to

17 Chapter 1. Introduction 3 execution monitoring (EM), a ubiquitous technique for enforcing security policies. In fact, this class of enforcement mechanisms has attracted the attention of the majority of researchers studying formal characterization of security policies and enforcement mechanisms. All these facts motivated us to elaborate a research thesis about memory-constrained security enforcement. More precisely, we investigate the security enforcement on Javaenabled memory-constrained devices by analyzing the security of Java ME-CLDC. An other important problem targeted by this thesis is the characterization of memoryconstrained security enforcement. More precisely, we investigate memory-constrained execution monitoring enforcement. 1.2 Objectives The main objective of this thesis is to evaluate and improve the security of memoryconstrained systems. Our investigation concerns two research areas. The first area is concerned with the evaluation and the improvement of memory-limited systems security. The second area is concerned with the characterization of memory-constrained security enforcement and the identification of its limitations. For the first area, we target the security of Java ME-CLDC, the standard Java platform for memory-limited networkconnected devices. For the second area, we target the characterization of memoryconstrained EM enforcement. More precisely, our objectives are the following: Study and evaluate the security of Java ME-CLDC. The output of this security analysis must (1) provide a comprehensive study of the Java ME-CLDC security model, (2) perform a vulnerability analysis of the platform, and (3) conduct a risk analysis. The security analysis must be as complete as possible by investigating standard specification documents, standard reference implementations, development platforms, and actual phones. Also, the analysis must be based on standard security evaluation and risk analysis methods. The performed analysis should point out possible shortcomings and aspects open for improvement. Provide a formal framework to characterize memory-constrained EM enforcement. The framework must allow (1) specifying the security policies that can be enforced by execution monitors constrained by memory imitations, (2) identifying the enforcement limits of memory-constrained EM by providing memory-size-directed taxonomy of EM-enforceable security policies, and (3) providing methodologies

18 Chapter 1. Introduction 4 and/or tools for deciding whether a security policy is enforceable by memoryconstrained EMs. The targeted results should be related as much as possible to language theory. 1.3 Methodology The methodology we adopt to reach the aforementioned objectives is detailed in the sequel. To study and evaluate Java ME-CLDC security, we first conduct a comprehensive study of Java ME-CLDC and the underlying security model, then we perform a security evaluation of this Java platform. The security evaluation consists of two main tasks: a vulnerability analysis and a risk analysis. Then, to characterize memory-constrained EM enforcement, we first study the state of the art related to characterizing enforceable security policies, then we provide our own contribution Studying Java ME-CLDC and its Security The study of Java ME-CLDC and its security was performed mainly by investigating the standard specifications and the available reference implementations. The standard specifications are mainly the Java Specification Requests (JSRs) adopted by the Java Community Process (JCP) for Java ME-CLDC APIs and the available documentation concerning the Java Virtual Machine (JVM). The reference implementations are the APIs implementations and the development platforms provided by Sun Microsystems. The study of Java ME-CLDC is performed by clarifying the differences between this Java platform and the conventional Java architecture (Java SE). The main investigated parts are the JVM, the mandatory APIs, and the optional APIs. The study starts by recalling the conventional Java security model. The main aspects covered by the study are static analysis (bytecode verification), sandbox model, security policy, and security policy enforcement Security Evaluation of Java ME-CLDC The components concerned by our evaluation are the JVM and the APIs recommended as mandatory in the latest revision of the Java Technology for the Wireless Industry

19 Chapter 1. Introduction 5 (JTWI) JSR [117]. The previous phase represents the starting point of the security evaluation of Java ME-CLDC. Our evaluation is performed following three main steps. First, we identify the Java ME-CLDC security model limitations. Then we perform a vulnerability analysis of the platform. Finally, we perform a risk analysis of Java ME- CLDC by estimating the seriousness of the uncovered vulnerabilities. The identified Java ME-CLDC security model limitations are the result of (1) our understanding of this security model, on the basis of the existing specification documents, and (2) a comparison with the conventional Java security model. The vulnerability analysis starts by compiling the known Java ME-CLDC vulnerabilities. Then we perform a static code analysis and a security testing of the investigated reference implementations. To be as efficient as possible in accomplishing these two tasks, we perform a reverse engineering of the investigated reference implementations. While static analysis is limited to the available reference implementations, security testing is performed on reference implementations, development platforms and actual phones. Static analysis is performed by manual code inspection and automatic tools. Security testing is performed by executing test suites implementing attack scenarios. Each attack scenario is designed to test some functional component of Java ME-CLDC. Risk analysis aims at structuring the uncovered vulnerabilities and assess the underlying risks according to a well-established and standard framework. The MEHARI method [30] is used to achieve this objective. The MEHARI s criteria are used to structure the uncovered vulnerabilities into an appropriate classification. Afterwards, the seriousness of each vulnerability is assessed based on the MEHARI s risk analysis methodology guidelines. As a downstream result of this phase, a reasonable and efficient set of security requirements is elaborated to harden the security of Java ME-CLDC platform implementations Characterizing Memory-constrained EM Enforcement To provide a formal characterization of memory-constrained EM-enforcement, we started by studying the state of the art related to characterizing security enforcement and classifying enforceable security policies. In particular, we investigated those contributions targeting constrained EM enforcement. EM mechanisms can be classified into conventional EMs and rewriting-based EMs. EMs belonging to the former class are usually specified by security automata [123] while those belonging to latter are commonly specified by edit automata [11]. To cover both conventional and rewriting-based EMs, our characterization is built on

20 Chapter 1. Introduction 6 top of both security automata and edit automata. To characterize memory-constraints, we instantiated and extended Fong s abstraction approach to deal with memory-limitations. The adopted abstraction characterizes the information tracked by an execution monitor by a bounded history representing the memory space which contains the history information tracked by the monitor. The abstraction definition is extended so (1) it covers finite and infinite executions and (2) it applies to both CEM-enforceable properties and RWEM-enforceable properties. Our characterization allows the specification of those security policies that can be enforced by memory-constrained EMs. The specification is based on a class of automata that we call bounded history automata. Also, we identified a new taxonomy of EM-enforceable properties. This taxonomy is directed by the size of the space used by execution monitors to save execution history. To provide methodologies and tools for deciding whether a given security policy is enforceable by memory-constrained EMs, we investigated the specification of locally testable properties by bounded history automata. Namely, we identified BHAenforceable local properties and showed how to check whether an EM-enforceable policy is local or not. The latter is performed by selecting automata-based algorithms which decide wether an EM-enforceable property is local or not. 1.4 Contributions Our contribution is mainly threefold: 1. We carefully studied and evaluated Java ME-CLDC security. To this end, we investigated both published specifications and available implementations. From the specifications, we completed a comprehensive study of Java ME-CLDC security, pointing out possible shortcomings and suggestions for improvements. Concerning the implementations, we extracted details of the security architecture (that are not well explained in the specification) and by black box testing and code inspection, we analyzed the security vulnerabilities of the platform. We performed our vulnerability analysis on reference implementations, development platforms, phone emulators, and actual phones. A useful result of the vulnerability analysis is a test suite that can be used to evaluate any Java ME-CLDC implementation. Finally, we performed a risk analysis of Java ME-CLDC. The analysis was performed using the MEHARI [30] method and it provides a seriousness estimation of the uncovered vulnerabilities. 2. We provided a characterization of memory-constrained EM enforcement. Namely,