JAVA 2 Network Security

JAVA 2 Network Security M A R C O PISTOIA DUANE F. RELLER DEEPAK GUPTA MILIND NAGNUR ASHOK K. RAMANI PTR, UPPER http://www.phptr.com PRENTICE HALL SADDLE RIVER, NEW JERSEY 07458

Contents Foreword Preface The Team That Wrote This Redbook Comments Welcome v xix xix xxi Part 1. Introduction to Java and Security 1 Chapter 1. An Overview of Java and Security 3 1.1 Java Is Not Just a Language 3 1.2 What Java Does 3 1.3 Java Is Not an Island: Java as a Part of Security 5 1.3.1 Safety and Security 7 1.3.2 Java as an Aid to Security 8 1.3.3 Java as a Threat to Security 9 1.3.4 Writing Secure Java 10 1.3.5 Staying One Jump Ahead 11 1.3.6 The Vigilant Web Site 12 1.4 Understanding Java 2 Security 12 1.4.1 An Example of Applet Security in Java 2 14 1.4.2 An Example of Application Security in Java 2 26 1.5 Summary 33 Chapter 2. Attack and Defense 35 2.1 Components of Java 35 2.1.1 The Development Environment 36 2.1.2 The Execution Environment 44 2.1.3 Interfaces and Architectures 50 2.2 Java 2 and Cryptography 53 2.2.1 Cryptographic Tools in Brief 54 2.2.2 Java Cryptography Architecture 56 2.2.3 United States Export Rules for Encryption 57 2.2.4 Signed Code 58 2.2.5 The Other Side of the Coin - Access Control 59 2.3 Attacking the World of Java 59 2.3.1 Perils in the Life of Remote Code 59 2.3.2 Vulnerabilities in Java Applications 66 2.4 Summary 68 Chapter 3. The New Java Security Model 69 3.1 The Need for Java Security 69 ix

3.2 Evolution of the Java Security Model 70 3.2.1 The JDK 1.0 Sandbox Security Model 70 3.2.2 The Concept of Trusted Code in JDK 1.1 72 3.2.3 The Fine-Grained Access Control of Java 2 74 3.2.4 A Comparison of the Three Java Security Models 78 3.3 Java 2 Protection Domain and Permissions Model 80 3.4 New Class Search Path 83 3.4.1 Boot Class Path 84 3.4.2 Extensions Framework 86 3.4.3 Application Class Path 88 3.4.4 Class Search Paths in Summary 89 3.5 Java 2 Class Loading Mechanism 89 3.5.1 Run-Time Access Controls 91 3.6 The Policy File 93 3.6.1 The Default System-Wide Policy File 96 3.7 Security Manager vs Access Controller 98 3.8 Security Management with Java 2 98 3.8.1 Applying a Security Manager to Applets and Applications 99 3.8.2 Applying a User-Defined Security Policy 99 3.8.3 Java Security Debugging 100 3.9 Summary 106 Part 2. Under the Hood 107 Chapter 4. The Java Virtual Machine 109 4.1 The Java Virtual Machine, Close Up 109 4.1.1 The Class Loader 110 4.1.2 The Class File Verifier 112 4.1.3 The Heap 112 4.1.4 The Class Area 112 4.1.5 The Native Method Loader 113 4.1.6 The Security Manager 113 4.1.7 The Execution Engine 113 4.1.8 Just-in-Time Compilers 113 4.2 Summary 115 Chapter 5. Class Files in Java 2 117 5.1 The Traditional Development Life Cycle 117 5.2 The Java Development Life Cycle 119 5.3 The Java 2 Class File Format 124 5.3.1 Decompilation Attacks 126 5.4 The Constant Pool 129 5.4.1 Beating the Decompilation Threat 134 X Java 2 Network Security

5.5 Java Bytecode 136 5.5.1 A Bytecode Example 136 Chapter 6. The Class Loader and Class File Verifier 145 6.1 Class Loaders 145 6.1.1 Loading Classes from Trusted Sources 146 6.1.2 Loading Classes from Untrusted Sources 147 6.1.3 Beyond What the JVM Provides 148 6.1.4 The Class Loading Process 150 6.1.5 Should You Build Your Own Class Loader 155 6.2 The Class File Verifier 168 6.2.1 An Example of Class File Verification 169 6.2.2 The Duties of the Class File Verifier 175 6.2.3 The Four Passes of the Class File Verifier 176 6.3 The Bytecode Verifier in Detail 180 6.3.1 The Data Flow Analyzer 181 6.4 An Incompleteness Theorem for Bytecode Verifiers 183 6.5 Summary 184 Chapter 7. The Java 2 SecurityManager 187 7.1 What SecurityManager Does 187 7.2 Operation of the Security Manager 190 7.2.1 Interdependence of the Three JVM Security Elements 192 7.3 Attacking the Defenses of Java 192 7.3.1 Types of Attack 193 7.3.2 Malicious Applets 195 7.4 Avoiding Security Hazards 204 7.4.1 How to Test 205 7.5 Examples of Security Manager Extensions 206 7.5.1 First Example - Overriding checkwrite() 206 7.5.2 Second Example - Overriding checkpermission() 211 7.5.3 Third Example - Overriding checkread() and checkwrite()... 218 7.6 Summary 224 Chapter 8. Security Configuration Files in the Java 2 SDK 225 8.1 A Note on Java.home and the JRE Installation Directory 225 8.2 Keystores 230 8.2.1 The Certificates KeyStore File cacerts 233 8.3 The Security Properties File, Java.security 234 8.4 Security Policy Files 242 8.4.1 keystore Entry 242 8.4.2 grant Entries 243 8.5 An Example of Security Settings in the Java 2 Platform 248 8.5.1 The Count Application Source Code 248 xi

8.5.2 A Sample Text File 249 8.5.3 Compiling the Application 249 8.5.4 Running the Application without a Security Manager 250 8.5.5 Running the Application with the Default Security Manager... 250 8.5.6 Policy File Modification 250 8.6 File Read Access to Files in the Code Base URL Directory 252 8.7 Security Properties and Policy File Protection 252 8.8 How to Implement a Policy Server 252 Chapter 9. Java 2 SDK Security Tools 259 9.1 Key and Certificate Management Tool 259 9.1.1 keytool Syntax 259 9.1.2 Store and Private Key Password 261 9.1.3 Commands and Options Associated with keytool 262 9.1.4 An Example of keytool Usage 269 9.2 Java Archive Tool 270 9.2.1 Options of the jar Command 271 9.2.2 Running a JAR File 274 9.3 JAR Signing and Verification Tool 275 9.3.1 jarsigner Scenario 280 9.3.2 Observations on the jarsigner Verification Process 284 9.3.3 Tampering with a Signed JAR File 286 9.4 Policy File Creation and Management Tool 288 9.4.1 Observations on the Use of the Policy Tool 295 Chapter 10. Security APIs in Java 2 297 10.1 The Package Java.security 297 10.1.1 Principals 297 10.1.2 Guard Interface and GuardedObject Class 298 10.1.3 Providers 299 10.1.4 The Security Class 301 10.1.5 Access Control APIs 304 10.1.6 Key Management 305 10.1.7 Message Digests and Digital Signatures 311 10.1.8 Secure Random Number Generation 316 10.1.9 The SignedObject Class 316 10.1.10 Permission APIs 317 10.1.11 Code Source 318 10.1.12 Protection Domain 321 10.1.13 Policy 321 10.1.14 Secure Class Loader 322 10.1.15 Algorithm Parameters 322 10.2 The Package Java.security.spec 322 Java 2 Network Security

10.3 The Package Java.security.cert 323 10.4 Package Java.security.interfaces 324 10.5 The Package Java.security.acl 324 10.6 Examples Using the Java 2 Security APIs 325 10.6.1 Signature and Signature Verification 325 10.6.2 Using Keystores 332 10.7 The Permission Classes 339 10.7.1 How to Create New Permissions 344 10.7.2 Working with Signed Permissions 348 10.8 How to Write Privileged Code 350 10.8.1 First Case - No Return Value, No Exception Thrown 351 10.8.2 Second Case - Return Value, No Exception Thrown 352 10.8.3 Third Case - Return Value, Exception Thrown 353 10.8.4 Accessing Local Variables 353 10.8.5 An Example of Privileged Blocks Usage 354 10.8.6 General Recommendations on Using the Privileged Blocks.. 358 Chapter 11. The Java Plug-In 359 11.1 Main Features of Java Plug-In 360 11.2 What Does the Java Plug-In Do? 364 11.3 Java Plug-In HTML Changes 364 11.3.1 Changes Supported by Navigator 364 11.3.2 Changes Supported by Internet Explorer 365 11.3.3 Changes Supported by Both Navigator and Internet Explorer. 366 11.3.4 All the Web Browsers 367 11.3.5 Java Plug-in Software HTML Converter 369 11.4 Java Plug-In Control Panel 370 11.4.1 The Basic Panel 370 11.4.2 The Advanced Panel 371 11.4.3 The Proxies Panel 373 11.5 Java Plug-In Security Scenario 374 11.5.1 First Step - Without Using the Java Plug-in 374 11.5.2 Second Step - Using the Java Plug-in 377 Chapter 12. Java Gets Out of Its Box 385 12.1 JAR Files and Applet Signing 385 12.1.1 Manifest File 387 12.1.2 Signature File 392 12.1.3 Signature Block File 392 12.2 Signed Code Scenario in JDK 1.1 and Sun HotJava 393 12.2.1 Creating the CA Key Database 393 12.2.2 Creating the Server Key Database 395 12.2.3 Creating and Signing a JAR File 397

12.2.4 Running the Applet 399 12.2.5 Creating the Client Key Database 399 12.3 Signed Code Scenario in Java 2 SDK, Standard Edition, V1.2.... 400 12.3.1 Creating a Keystore for Certification Authorities 401 12.3.2 Creating the Server Certificate 402 12.3.3 Creating and Signing a JAR file 406 12.3.4 Granting the Permissions and Running the Applet 407 12.4 Signed Code Scenario in Netscape Communicator 409 12.4.1 Using the netscape.security Package 410 12.4.2 Installing Keys and Certificates in Netscape Communicator.. 415 12.4.3 Signing JAR Files with Netscape Signing Tool 418 12.5 Signed Code Scenario in Microsoft Internet Explorer 437 12.5.1 First Example with Signed CAB Files 438 12.5.2 A More Complex Signed CAB File Example 450 12.6 The JAR Bug - Fixed In Java 2 SDK, Standard Edition, V1.2.1... 461 12.6.1 The Solution in Java 2 SDK, Standard Edition, V1.2.1 470 12.7 Future Developments 470 Part 3. Beyond the Island of Java - Surfing into the Unknown 473 Chapter 13. Cryptography in Java 2 475 13.1 Security Questions, Cryptographic Answers 475 13.1.1 Public Key Certificates 478 13.2 The Java Cryptography Architecture Framework 480 13.2.1 JCE and United States Export Considerations 481 13.2.2 Relationship between Java 2 SDK, JCA and JCE APIs 482 13.3 JCA Terms and Definitions 483 13.3.1 The Provider Concept in the JCA 484 13.3.2 Engine Classes 487 13.3.3 Algorithms 489 13.4 Java Cryptography Extension 493 13.4.1 JCE - Packages and Their Contents 493 13.4.2 The Cipher Class 495 13.4.3 The Cipher Stream Classes 496 13.4.4 Secret Key Interfaces and Classes 499 13.4.5 The KeyGenerator Class 499 13.4.6 The Key Agreement Class 500 13.4.7 The SealedObject Class 501 13.5 Java Cryptography in Practice 503 13.5.1 First Scenario 503 13.5.2 Second Scenario 509 13.6 Asymmetric Encryption with the Java 2 SDK and JCE 1.2 516 13.6.1 Using Asymmetric Encryption 516 xiv Java 2 Network Security

13.7 How to Implement Your Own Provider 520 13.7.1 Write the Service Implementation Code 521 13.7.2 Give the Provider a Name 522 13.7.3 Write a Master Class 522 13.7.4 Compile the Code 523 13.7.5 Install and Configure the Provider 524 13.7.6 Test if the Provider Is Ready 524 13.7.7 Algorithm Aliases 524 13.7.8 Dependencies on Other Algorithms 525 13.7.9 Default Initializations 526 13.7.10 A Sample Master Class 526 Chapter 14. Enterprise Java 527 14.1 Browser Add-On Applets 527 14.2 Networked Architectures 527 14.2.1 Applying the Java 2 Access Control Mechanisms 528 14.2.2 Two-Tier Architecture 529 14.2.3 Three-Tier Architecture 529 14.2.4 Network Security 532 14.3 Secure Clients and Network Computers 535 14.4 Server-Side Java 536 14.4.1 The Cost of Server-Side Java 537 14.5 Servlets 538 14.5.1 Advantages of Servlets 540 14.5.2 Servlets and CGI-BINs 541 14.5.3 Java Servlet APIs 542 14.5.4 Servlet Life Cycle 544 14.5.5 IBM WebSphere Application Server 546 14.5.6 A Sample Servlet 548 14.5.7 The Current Servlet Security Model 556 14.6 Distributed Object Architectures - RMI 563 14.6.1 Stubs and Skeletons 565 14.6.2 RMI Registry 566 14.6.3 A Sample RMI Program 568 14.6.4 The Security of RMI 579 14.7 Enterprise JavaBeans 580 Chapter 15. Java and Firewalls - In and Out of the Net 583 15.1 What Is a Firewall? 583 15.2 What Does a Firewall Do? 584 15.2.1 Inside a TCP/IP Packet 584 15.2.2 How Can Programs Communicate through a Firewall? 587 15.3 Detailed Example of TCP/IP Protocol 588 xv

15.3.1 DNS Flow (UDP Example) 588 15.3.2 HTTP Flow (TCP Example) 590 15.4 Proxy Servers and SOCKS Gateways 596 15.4.1 Proxy Servers 596 15.4.2 What Is SOCKS? 597 15.4.3 Using Proxy Servers or SOCKS Gateways 600 15.5 The Effect of Firewalls on Java 601 15.5.1 Downloading an Applet Using HTTP 601 15.5.2 Stopping Java Downloads with a Firewall 601 15.5.3 Java Network Connections through the Firewall 604 15.6 Java and Firewall Scenarios 606 15.6.1 URL Connection 608 15.6.2 Socket Connection 616 15.6.3 Conclusions 624 15.7 Remote Method Invocation 625 15.8 Summary 628 Chapter 16. Java and SSL 629 16.1 What Is SSL? 629 16.2 Using SSL from an Applet 634 16.2.1 Using SSL URLs with Java 635 16.3 Java and SSL with Sun Microsystems 635 16.3.1 The javax.net Package 636 16.3.2 The javax.net.ssl Package 636 16.3.3 The javax.security.cert Package 638 16.4 How to Use Java and SSL 639 16.4.1 Skeleton Program without SSL 640 16.4.2 Using SSL with the Sun Microsystems API 649 16.5 Java and SSL with IBM SSLite 659 16.5.1 Extensions to the SSL Protocol 661 16.5.2 SSLite Key Ring Management Tools 661 16.5.3 SSL Server Authentication with IBM SSLite for Java 665 16.6 Conclusions 667 16.7 Summary 668 Chapter 17. Epilogue 669 17.1 Future Directions of Java 669 17.1.1 Java 2 SDK-The Path Ahead 669 17.1.2 Resource Consumption Management 670 17.1.3 Java Authentication and Authorization Service 670 17.1.4 Java RMI Security Extension 671 17.1.5 Arbitrary Grouping of Permissions 671 17.1.6 Object-Level Protection 671 XVI Java 2 Network Security

17.1.7 Subdividing Protection Domains 672 17.1.8 Running Applets with Signed Content 672 17.1.9 Java 2 Platform, Enterprise Edition 673 17.2 Conclusion 673 Appendix A. Getting Internal System Properties 675 A.1 Program GetAIIProperties 675 A.2 Program GetProperty 678 Appendix B. Signature Formats 681 Appendix C. X.509 Certificates 683 C.1 X.509 Certificate Versions 684 Appendix D. Sources of Information about Java Security 685 D.1 Companies 685 D.1.1 JavaSoft 685 D.1.2 Sun 686 D.1.3 IBM 686 D.1.4 Microsoft 687 D.1.5 Reliable Software Technologies 688 D.1.6 JavaWorld 688 D.1.7 JCE Providers outside the United States 688 D.2 Universities 689 D.2.1 Princeton 689 D.2.2 Yale 689 D.2.3 Others 690 Appendix E. What's on the Diskette? 691 E.1 How to Access the Diskette 691 E.2 How to Get the Same Software Material from the Web 691 Appendix F. Special Notices 693 Appendix G. Related Publications 697 G.1 International Technical Support Organization Publications 697 G.2 Redbooks on CD-ROMs 697 G.3 Other Publications 697 How to Get ITSO Redbooks 699 IBM Redbook Fax Order Form 700 xvii

Glossary 701 Index 703 ITSO Redbook Evaluation 713 XVIII Java 2 Network Security