Server Forensics: Linux

Transcription

1 Server Forensics: Linux Part II.A. Techniques and Tools: Computer Forensics CSF: Forensics Cyber-Security Fall 2015 Nuno Santos

2 Summary } Linux forensics } Overview of Linux (Unix) systems } System subversion with rootkits } Rootkit detection techniques 2

3 Operating systems we will be focusing on } Desktop platforms: Windows Previous class } Server platforms: Linux Today! } Mobile platforms: Android 3

4 Our main focus when analyzing a server s OS } On desktops, we were mostly concerned about evidence left by local users activities } On servers, we will look for evidence left by external agents that resulted in security violations to the OS attack } Such external agents typically consist of stealthy malware called rootkits rootkit 4

5 Overview of Linux systems 5

6 Linux belongs to the big family of Unix systems 6

7 Unix systems bootstrap sequence } At a high-level, Unix systems have same boot sequence 7

8 Unix systems architecture } Unix systems also have same arch, at a high-level } The kernel can be extended with loadable modules 8

9 System subversion with rootkits 9

10 What s wrong with this picture? [root@dobro bin]# ls - a. dd igawk nisdomainname tar.. df ipcalc pgawk tcsh.. dmesg kbd_mode ping touch alsaunmute dnsdomainname kill ping6 tracepath arch doexec ksh ps tracepath6 ash domainname link pwd traceroute ash.static dumpkeys ln red traceroute6 aumix- minimal echo loadkeys rm true awk ed login rmdir umount 10

11 Let s take a closer look [root@dobro]# ls - a. dd igawk nisdomainname tar.. df ipcalc pgawk tcsh.. dmesg kbd_mode ping touch How can there be two.. directories?... 11

12 How did this happen? bin]# mkdir..\ } This is actually: mkdir <dot><dot><backslash><space><enter> } It creates a directory named dot-dot-space 12

13 What s in this mystery directory? [root@dobro bin]# cd..\ [root@dobro.. ]# ls - l total 24 - rw- r- - r- - 1 root root 0 Dec 15 12:19 rootkit_file_01 - rw- r- - r- - 1 root root 0 Dec 15 12:19 rootkit_file_02 - rw- r- - r- - 1 root root 0 Dec 15 12:19 rootkit_file_03 - rw- r- - r- - 1 root root 0 Dec 15 12:19 rootkit_file_04 - rw- r- - r- - 1 root root 0 Dec 15 12:19 rootkit_file_05 - rw- r- - r- - 1 root root 0 Dec 15 12:19 rootkit_file_06 Here s a simple trick to hide malicious files in plain sight 13

14 Rootkits } The behavior of the operating system can be affected by the presence of rootkits } Rootkits are a category of malware which has the ability to hide itself and cover up traces of activities 14

15 Sample scenario } A rootkit runs a password logger on a compromised system Password Logger Compromised system with sniffed passwords encoded using some text-based steganography techniques 15

16 Another example } A rootkit installs a backdoor on the compromised system Shell Backdoor Compromised system communication with backdoor (covert channel in HTTP or DNS) 16

17 Rootkit goals 1. Remove evidence of original attack and activity that led to rootkit installation 2. Hide future attacker activity (files, net connections, processes) and prevent it from being logged 3. Enable future access to system by attacker 4. Install tools to widen scope of penetration 5. Secure system so other attackers can t take control of system from original attacker 17

18 Rootkits include several attacker tools } Attack tools installed after an intruder gained access: } Backdoor programs } Network sniffers } Log cleaners } File / process / user hiding tools } etc. 18

19 Rootkit tools: Backdoor programs } Backdoor is an unauthorized way of gaining access to a program, online service or an entire computer system } Let attackers log in to the hacked system w/o using an exploit again Examples Login Backdoor Telnetd Backdoor Services Backdoor Description Modify login.c to look backdoor password before stored password Trojaned the in.telnetd to allow attacker gain access with backdoor password Replacing and manipulate services like ftp, rlogin, even inetd as backdoor to gain access Cronjob backdoor Backdoor could also be added in crontjob to run on specific time for example at 12 midnight to 1 am Library backdoors Kernel backdoors Shared libraries can be backdoor to do malicious activity including giving a root or administrator access This backdoor is basically exploiting the kernel 19

20 } Packet sniffers Rootkit tools: Sniffers and wipers } Packet Sniffer is a program and/or device that monitor data traveling over a network, TCP/IP or other network protocol } Used to listen or to steal valuable information off a network; many services such as ftp and telnet transfer their password in plain text and it is easily capture by sniffer } Log-wiping utilities } Log file are the lists actions that have occurred, e.g., in UNIX, wtmp logs time and date user log in into the system } Log file enable admins to monitor, review system performance and detect any suspicious activities } Deleting intrusion records helps prevent detection of the intrusion 20

21 Rootkit tools: Miscellaneous attacker tools } DDOS program } To turn the compromised server into a DDOS client such as, trinoo } IRC program } Connects to some remote server waiting for the attacker to issue a command (e.g., to trigger a distributed denial of service attack) } System patch } Attacker may patch the system after successful attack; this will prevent other attacker to gain access into the system again } Password cracker } Vulnerability scanners } Hiding utilities } Utilities to conceal the rootkit files on compromised system 21

22 Rootkit types } Command-level rootkits } Replace user-level programs } Library-level rootkits } Replace system libraries } Kernel-level rootkits } Modify the kernel itself 22

23 Command-level subversion } Command-level rootkits hide the presence of malware by making changes to system commands } Based on principle: To suppress bad news, silence the messenger } Table shows typical command-level rootkit modifications 23

24 Evolution of command-level rootkits } First rootkits came with network sniffers to collect user names and passwords, e.g., the esniff program } Later versions came with remotely controlled agents for distributed denial-of-service attacks, e.g., T0rn 24

25 Command-level rootkit evasion } To evade detection, early rootkits not only replaced system utility sw but also erased records in system logs } Some rootkits even gave modified system utilities the same file time stamps and cyclic redundancy check (CRC) values as the original files } Later command-level rootkits simply install modified programs that hide the presence of malware 25

26 Installation concealment } Use a subdirectory of a busy system directory like /dev, / etc, /lib, or /usr/lib } Use dot files, which aren t in ls output } Use spaces to make filenames look like expected dot files:. and.. } Use filenames that system might use } /dev/hdd (if no 4 th IDE disk exists) } /usr/lib/libx.a (libx11 is real Sun X-Windows) } Delete rootkit install directory once installation is complete 26

27 Rootkit types } Command-level rootkits } Library-level rootkits } Kernel-level rootkits 27

28 Library-level subversion } Instead of replacing system utilities, rootkits can hide their existence by making changes at the next level down in the system architecture: the system run-time library } A good example is redirecting the open() and stat() calls } The purpose of these modifications is to fool file-integrity-checking software that examines executable file contents and attributes } By redirecting the open() and stat() calls to the original file, the rootkit makes it appear as if the file is still intact } However, execve() executes the subverted file 28

29 Example of library-level subversion } Redirect specific open() system calls real_syscall3() is a macro (not entirely shown) that modifies the standard _syscall13() macro real_syscall3 is defines our real_open() function that invokes Sys_open system call 29

30 Rootkit types } Command-level rootkits } Library-level rootkits } Kernel-level rootkits 30

31 Kernel-level subversion } Just like user-level rootkits, kernel-level rootkits are installed after a system s security has been breached } Kernel-level rootkits compromise the kernel } Kernel runs in supervisor processor mode } Thus, the rootkit gains complete control over the machine } Advantage: stealth, e.g., } Runtime integrity checkers cannot see rootkit changes } All programs in the system can be affected by the rootkit } Open backdoors/sniff network without running processes 31

32 Kernel rootkit operation } The purpose of many kernel rootkits is to hide malware processes, files, and network ports (& the rootkit itself) } There are to sides to information hiding } Output: the kernel must censor the output from system calls that produce a list of processes, files, network ports, etc. } Input: any attempts to manipulate a hidden process, file, network port, and so on must fail as if the object did not exist } May also redirect system calls to subvert operation of software that checks integrity of executable files 32

33 Methods to inject rootkit code into a kernel 1. Loading kernel module into a running kernel } Uses official LKM interface, hence it s easier to use } Hide module names from external (/proc/ksyms) & internal tables } Might also intercept syscalls that report on status of kernel modules 2. Injecting code into the memory of a running kernel that has no support for module loading } Involves writing new code to unused kernel memory via /dev/kmem and activating the new code by redirecting, e.g., a system call 3. Injecting code into the kernel file or a kernel module file } These changes are persistent across boot, but require that the system is rebooted to activate the subverted code 33

34 Early kernel rootkit architecture } Based on system-call interposition: Early kernel rootkits subvert syscalls close to the process-kernel boundary 34

35 Rootkit interposition code } To prevent access to a hidden file, process, and so on, rootkits redirect specific system calls to wrapper code } To prevent rootkit disclosure, syscalls that produce lists of files, etc., are intercepted to suppress info to be hidden 35

36 System-call interposition: good and bad } Advantage: The code is easy to understand } The change is made at a point that is close to the user of those system calls } Disadvantage: Many system calls need to be intercepted } E.g., to hide the existence of a file, need to intercept all system calls that have file name arguments, about 50 syscalls } Alternative: subvert the kernel at a level that is closer to the information being hidden 36

37 Later kernel rootkit architectures } Based on file system interposition: Rootkits based on interposition at object interfaces subvert several FS methods 37

38 Hiding file names } Of all the operations provided by the VFS interface the lookup() operation is of particular interest } lookup() translates the pathname into underlying FS object } Redirect can hide a file from all syscalls that access a file by name } Doesn t work for syscalls that create a new directory entry } E.g., link(), mkdir(), socket(), etc. } Unless the malware does not handle this, operation will fail 38

39 Rootkit detection techniques 39

40 Detection techniques for each rootkit type } Command-level rootkits } Library-level rootkits } Kernel-level rootkits 40

41 Approaches to detecting command-level rootkits 1. If a rootkit listens for connections, the network port will be visible to an external network port scanner 2. Some tools can reveal the names of all directory entries, including hidden or deleted files 3. Corrupted versions of ps and similar hide malware processes, but these can still be found using, e.g., the /proc file system 4. Deleted login/logout records in the wtmp file leave behind holes that can be detected using an appropriate tool 5. Ifconfig might report that a network interface is not in sniffer mode, but we can query the kernel for the interface status 6. CRC checksums reported by compromised cksum, can be detected using MD5 or SHA1 7. When examining a low-level copy of the file system on a trusted machine, all hidden files and modifications will be visible 41

42 Example of exposing a command-level rootkit 1/7 } We examine two utilities of the T0rn rootkit for Linux } This rootkit was in widespread use in 2001 } First, search the /bin/ls executable file using strings and grep } The file name /usr/src/.puta/.1file looks very suspicious 42

43 Example of exposing a command-level rootkit 2/7 } If we try to list the /usr/src/.puta directory, the ls command hides the name, as we would expect: } However, the directory name still shows up when we use the echo command, together with the.* wildcard expansion feature that is built into the command shell 43

44 Example of exposing a command-level rootkit 3/7 } File.puta/.1file is a rootkit configuration file } Has list of file and directory names that must remain hidden } Contain the malware program files, config files, and data files 44

45 Example of exposing a command-level rootkit 4/7 } To detect modified process status check against /proc } Table shows that ps command is hiding with ID

46 Example of exposing a command-level rootkit 5/7 } The system utilities that were replaced by the rootkit do a good job of hiding process 153 } Not only it is censored by process status tools like ps, but also it does not show up with network status tools such as netstat } However, it does not replace the lsof command, which can help reveal the purpose of process

47 Example of exposing a command-level rootkit 6/7 } File name /usr/sbin/nscd suggests that it is a system program, but comparison with uncompromised systems shows this program exists only in later Linux versions } nscd: name service cache daemon 47

48 Example of exposing a command-level rootkit 7/7 } Connecting with telnet to TCP port on the local machine confirms it is a backdoor process } In this case we are welcomed by the opening banner of what appears to be an SSH server } Find more about the T0rn rootkit at: 48

49 Automated approaches for detection } Each rootkit differs slightly in its approach to hiding the presence of malware and therefore requires us to take a slightly different approach to detect it } An example of software that automates the search for known rootkits is the Chkrootkit toolkit } } } Runs on a dozen different UNIX platforms Recognizes more that fifty different rootkits Looks for deleted login / logout records, signatures of replaced system utilities, rootkit configuration files and directories, missing processes, and signs of kernel-level subversion 49

51 Detection of library-level rootkits } Strawman: check library hashes. But, would an MD5 or SHA-1 hash reveal library modifications? } Not necessarily: while the runtime linker uses the lowlevel open() system call when it accesses the modified library file, md5sum use the fopen() library routine } Therefore, can be redirected to the unmodified library file } Workaround: rootkit detection tools need to carry their own trusted copy of the system library routines 51

53 Detection of kernel-level rootkits } Kernel rootkits may be exposed because they introduce little inconsistencies into a system } Some may show up externally, in the results from system calls that manipulate processes, files, kernel modules, etc. } Others show up only internally, in the contents of kernel data structures } E.g., hidden objects occupy some storage even though the storage does not appear in kernel symbol tables 53

54 Inconsistencies that may reveal kernel rootkits } Output of tools that bypass the file system can reveal information that is hidden by compromised FS code } E.g., TSK } Oversight } E.g., the modification time of an important system directory is changed, not to the contents of that directory } Inconsistencies in the results from process-manipulating system calls and from the /proc file system } E.g., in reporting a process as not found ) 54

55 Inconsistencies that may reveal kernel rootkits } Unexpected behavior of some system calls } E.g., when the Adore rootkit is installed, setuid() change process privileges will report success for some parameter value even though the user does not have sufficient privileges } E.g., when the Knark rootkit is installed, settimeofday() set the system clock will report suffess for some parameter values even though is should always fail when invoked by an unprivileged user } Directory hard link count inconsistencies } The hard link count of a directory, as reported by stat(), should equal the number of subdirectories, as reported by getdents() } If a directory is being hidden, it may show up as a missing hard link 55

56 Inconsistencies that may reveal kernel rootkits } Modifications to kernel tables, such as system call table or the virtual FS table } May be detected after the fact by reading kernel memory via /dev/kmem } Or by examining kernel memory from inside with a forensic kernel module such as Carbonite } Modifications to kernel tables or kernel code may be detected using a kernel module that samples critical data structures periodically 56

57 Inconsistencies that may reveal kernel rootkits } Checking against raw data sources } Modifications that hide files can show up as inconsistencies between information from the raw disc device and information returned by the kernel file system code } Modifications that hide network ports, processes, or kernel modules may be exposed by reading kernel memory and comparing the contents of kernel data structures with results from system calls 57

58 Example of kernel toolkit detector tool } Findrootkit: examines kernel mem through /dev/kmem and checks consistency from multiple sources: } The /dev/ksyms symbol table, with kernel function and data addresses and sizes } The in-kernel module list, with executable code and data segment addresses and sizes of loaded kernel modules } The in-kernel text arena table, with executable code segment addresses and sizes } The function addresses in the system call jump tables and in file system operation jump tables } The executable code and data segment sizes, as specified in the symbol tables of kernel module files 58

59 Example of kernel toolkit detector tool } Findrootkit can produce modification reports } Report example for a Solaris kernel } Changes to (a) the file system operations table, and (b) the systemcall jump table 59

60 Conclusions } Many attacks to operating systems are performed through rootkit software } There are three different types of rootkits depending on the OS layer targeted by the rootkit: command-, library-, and kernel-level rootkits } Depending on the rootkit, the forensic analyst needs to employ different rootkit-detection techniques 60

61 References } Primary bibliography } Dan Farmer, Wietse Venema, Forensic Discovery, Chapter 5 61

62 Next class } Mobile forensics: Android 62