Host-based Intrusion Prevention on Windows and UNIX. Dr. Rich Murphey White Oak Labs

Transcription

1 Host-based Intrusion Prevention on Windows and UNIX Dr. Rich Murphey White Oak Labs

2 Acknowledgements Niels Provos OpenBSD s systrace DT suggested this thread last year Greg Hoglund insights md5 at da ghettohackers reviews 2 DEFCON XI 8/3/2003

3 What is Intrusion Prevention? To a netsec person it looks like a firewall. Messages Rules Messages (Packets) 3 DEFCON XI 8/3/2003

4 What is Intrusion Prevention? To a AV person it looks like an AV. Messages Signatures Messages (File IO) 4 DEFCON XI 8/3/2003

5 What is Intrusion Prevention? Network-based Packets Signatures Host-based API() Signatures 5 DEFCON XI 8/3/2003

6 What is Intrusion Prevention? Packets Signatures Consider personal firewalls that combine host and network based filtering. Socket() Signatures 6 DEFCON XI 8/3/2003

7 How is IP different? Rather than rules, it uses signatures. But these aren t the same signatures you might run in an Intrusion Detection Systems (IDS) Signatures as access controls. API() Signatures 7 DEFCON XI 8/3/2003

8 How is it different? Consider SNORT/Hogwash.. Signature-based Firewall Packets Signatures IDS vendors call this Gateway IDS 8 DEFCON XI 8/3/2003

9 What is Intrusion Prevention? It's complementary to AV & Firewall API() Signatures Filters messages between applications and the kernel. Uses signatures to recognize payload behavior or injection mechanisms. 9 DEFCON XI 8/3/2003

10 Why the heck should we care? Encryption, fragmentation and reencoding, can prevent application layer filtering on the wire. Data resides in the clear in the application layer. So do exploits. 10 DEFCON XI 8/3/2003

11 Why the heck should we care? Visibility into the application layer provides capability for better contextual discrimination. = Stops certain kinds of exploits. 11 DEFCON XI 8/3/2003

12 So, why do we need another tool? Network security App level Firewalls Attack through services/daemons. crunchy on the outside? Application state is complex. State of memory, disk, clients 12 DEFCON XI 8/3/2003

13 IP Signatures For signatures that: Are application state specific Are system state specific Use contextual clues Block from the inside 13 DEFCON XI 8/3/2003

14 Well, OK, so how does it work? Consider architectural layers in the OS. Let s take a look at: Layers in Windows architecture Layers in UNIX 14 DEFCON XI 8/3/2003

15 Win2K System Architecture Win32 App Win32 Subsystem OS/2 App OS/2 Subsystem Ntdll.dll Posix App Posix Subsystem IO File Systems Security Reference Monitor Device Drivers Executive Services Interface IPC Virtual Object Hardware Process Micro Kernel GDI Window Hardware Abstraction Layer (HAL) Window Graphics Device Drivers 15 DEFCON XI 8/3/2003

16 Win2K System Architecture Win32 App Win32 Subsystem IO Mgr. File Sys. Security Ref. Monitor Device Drivers OS/2 App OS/2 Subsystem Executive Services Interface IPC Mgr. Ntdll.dll Virtual Process Mgr. Mgr. Object Hardware Micro Kernel GDI Window Mgr. Hardware Abstraction Layer Posix App Posix Subsystem Window Mgr. Graphics Device Drivers Two distinct user-land layers: Binary compatible app layer OS specific, native layer Three distinct kernel layers: Executive services Object Hardware Abstration Layer 16 DEFCON XI 8/3/2003

17 Linux System Architecture Win32 App Mac App Games Linux App Wine or Win4Lin Basilisk MAME Libc.so System call interface Scheduler Process Virtual Allocator VFS Interface Character Devices File Systems Block Devices Socket Network Protocols Network Drivers Hardware 17 DEFCON XI 8/3/2003

18 Linux System Architecture One common abstraction layer for kernel. One common abstraction layer for user-land. Is fewer layers bad? Linux App Scheduler Process Win32 App Wine or Win4Lin Virtual Allocator Libc.so Mac App Basilisk System call interface VFS Interface Character Devices Hardware File Systems Block Devices Games MAME Socket Network Protocols Network Drivers 18 DEFCON XI 8/3/2003

19 Linux System Architecture Each layer provides context info. Linux App Win32 App Wine or Win4Lin Libc.so Mac App Basilisk Games MAME What can that information provide? Given this information, what can we distinguish? How is this different than other forms of filtering? Scheduler Process Virtual Allocator System call interface VFS Interface Character Devices Hardware File Systems Block Devices Socket Network Protocols Network Drivers 19 DEFCON XI 8/3/2003

20 Architecture Comparison Win32 App Win32 Subsystem OS/2 App OS/2 Subsystem Posix App Posix Subsystem Linux App Win32 App Wine or Win4Lin Mac App Basilisk Games MAME Ntdll.dll Libc.so Executive Services Interface System call interface IO Mgr. File Sys. Security Ref. Monitor Device Drivers IPC Mgr. Virtual Process Mgr. Mgr. Object Micro Kernel GDI Window Mgr. Hardware Abstraction Layer Window Mgr. Graphics Device Drivers Scheduler Process Virtual Allocator VFS Interface Character Devices File Systems Block Devices Socket Network Protocols Network Drivers Hardware Hardware 20 DEFCON XI 8/3/2003

21 Linux System Architecture Even considering the app. as a black box we can observe at multiple layers: Linux App Win32 App Wine or Win4Lin Libc.so Mac App Basilisk System call interface Games MAME API calls System calls Instruction level Scheduler Process Virtual Allocator VFS Interface Character Devices Hardware File Systems Block Devices Socket Network Protocols Network Drivers 21 DEFCON XI 8/3/2003

22 Linux System Architecture Win32 App Mac App Games Exploits: focus on escaping IDS often cause exceptional behavior. Rather than fix IDS, let s take an entirely different approach. Linux App Scheduler Process Wine or Win4Lin Virtual Allocator Libc.so Basilisk System call interface VFS Interface Character Devices Hardware File Systems Block Devices MAME Socket Network Protocols Network Drivers 22 DEFCON XI 8/3/2003

23 Linux System Architecture Win32 App Mac App Games Because it acts at these layers it is independent of: Method of transport Method of injection. It is specific to: App state or behavior Payload behavior Linux App Scheduler Process Wine or Win4Lin Virtual Allocator Libc.so Basilisk System call interface VFS Interface Character Devices Hardware File Systems Block Devices MAME Socket Network Protocols Network Drivers 23 DEFCON XI 8/3/2003

24 Examples Yes, let's consider a specific payload (and exploit for that matter) on FreeBSD. How does it get in and how do we distinguish it's behavior. Why is this important DEFCON XI 8/3/2003

25 Demo of IP on FreeBSD Background of exploit: release date, authors, impact, subsequent variations. firewall coverage anti-virus coverage IDS and AV coverage 25 DEFCON XI 8/3/2003

26 Demo of successful exploit 26 DEFCON XI 8/3/2003

27 Review of injection and payload Code excerpts: Vulnerability Injection Payload 27 DEFCON XI 8/3/2003

28 How can we prevent it? Unique aspects of payload? How we can recognize it.. Unique aspects of injection? How we can recognize it.. 28 DEFCON XI 8/3/2003

29 Demo of blocked exploit 29 DEFCON XI 8/3/2003

30 Review of contextual information network identity user identity application state authentication workflow orthogonally to other methods 30 DEFCON XI 8/3/2003

31 What s the state of the industry? Leading products in the market? What differentiates them? How do the integrate with others? Success stories? 31 DEFCON XI 8/3/2003

32 The End Updated slides and tools will be available at: I can be contacted at: [email protected] 32 DEFCON XI 8/3/2003