Malware. Prof. Tom Austin San José State University Spring 2014

Transcription

1 Malware Prof. Tom Austin San José State University Spring 2014

2 Or: The Cat & Mouse Game Attackers and Defenders Play

3 1971

4 "I'M THE CREEPER : "I'M THE CREEPER : CATCH ME IF YOU CAN." CATCH ME IF YOU CAN." "I'M THE CREEPER : CATCH ME IF "I'M YOU THE CAN." CREEPER : "I'M THE CREEPER : CATCH ME IF YOU CAN." CATCH ME IF YOU CAN." "I'M THE CREEPER : CATCH ME IF YOU "I'M CAN." THE CREEPER : "I'M THE CATCH CREEPER ME IF : YOU CAN." CATCH ME IF YOU CAN."

5 Creeper The world's first computer virus (probably) Created by Bob Thomas as an experiment in "mobile" self-replicating programs Not malicious, just annoying Very annoying.

6 Reaper A second self-replicating program soon appeared. It 1. Deleted Creeper 2. Spread to other machines 3. Deleted itself Arguably the first anti-virus software and the second virus. "Core War" game inspired by Creeper/Reaper.

7 The ongoing struggle The contest between malware writers and anti-malware writers has continued ever since

8 Malware has gone from a lonewolf's tool for petty revenge

9 ...to the realm of the professional criminal

10 even to weapons for nation states.

11 What is malware? Malicious software Covers a variety of hostile or intrusive software

12 Where does malware live? Boot sector Take control before anything else Memory resident Stays in memory Applications, macros, data, etc. Library routines Compilers, debuggers, virus checker, etc.

13 Types of Malware Trojan horse: Appears benign, but has some malicious behavior. Backdoor: Allows unauthorized access.

14 Yay, free music! To listen: 1. Double click on the file 2. itunes opens But then things get weird

15 Mac Trojan Double click on freemusic.mp3 itunes opens (expected) Wild Laugh (not expected) Message box (not expected)

16 Trojan Example How does freemusic.mp3 trojan work? This mp3 is an application, not data This trojan is harmless, but could have done anything user could do (like delete all of your files)

17 More Types of Malware Rabbit: exhausts system resources. Worm: Actively spreads over the network. Many more exist: Rootkits, Adware, Keyloggers

18 Trojan Rabbit? These categories often overlap. A Trojan Horse might attempt to exhaust system resources A worm might install a backdoor on its victim

19 One type of malware I have not defined so far Computer Viruses In the vernacular, 'virus' is often a synonym for malware. So how do computer scientists use the term?

20 The Many Definitions of "Virus" "A virus is a program that is able to infect other programs by modifying them to include a possibly evolved copy of itself." --Frederick Cohen "A computer virus is a program that recursively and explicitly copies a possibly modified version of itself." --Peter Szor

21 Infection process (viruses under Cohen's definition) Attacker creates an initial germ file. The germ infects other programs. When infected programs run, they spread the virus to other programs.

22 How a file gets infected original code infected code

23 How a file gets infected original code infected code JMP JMP

24 History of Famous Worms

25 November 2, 1988 Apple and Micro$oft were fighting for control of the PC market. Windows 3.0 had not yet been released.

26 Robert Tappan Morris Cornell grad student. Allegedly, he wanted to gauge the size of the Internet. He wrote a program that would: 1. Determine where it could spread 2. Transmit itself 3. Hide, and then repeat the process.

27 How Morris Worm Spread Obtained access to machines by User account password guessing Exploit buffer overflow in fingerd Exploit trapdoor in sendmail Flaws in fingerd and sendmail were wellknown, but not widely patched

28 Bootstrap Loader Once Morris worm got access Bootstrap loader sent to victim 99 lines of C code Victim compiled and executed code Bootstrap loader fetched the worm Victim authenticated sender Don t want user to get a bad worm

29 How to Remain Undetected? If transmission interrupted, code deleted Code encrypted when downloaded Code deleted after decrypt/compile When running, worm regularly changed name and process identifier (PID)

30 Whoops Morris's program had a bug (or so he claimed). It would re-infect the same systems It acted like a fork-bomb (a type of rabbit) and took out many systems

31 The Internet had been designed to survive a nuclear war but apparently not one grad student.

32 Morris Worm Aftermath Estimate of damages: between $100,000 and $10 million Morris became the first person convicted under the 1986 Computer Fraud and Abuse Act The Computer Emergency Response Team Coordination Center (CERT/CC) was created as a direct result of this attack Increased awareness of computer security

33 2001 We had survived Y2K. The world did not end. Windows XP began its long reign.

34 HELLO! Welcome to Hacked By Chinese! In 15 hours, 250,000 machines were infected

35 Code Red Worm Eventually infected 750,000 out of about 6,000,000 vulnerable systems Exploited buffer overflow in Microsoft IIS server software Then monitor traffic on port 80, looking for other susceptible servers

36 Code Red: What it Did Day 1 to 19 of month: spread its infection Day 20 to 27: distributed denial of service attack (DDoS) on Later version (several variants) Included trapdoor for remote access Rebooted to flush worm, leaving only trapdoor Some say it was beta test for info warfare But no evidence to support this

37 January 25, 2003 As impressive as Code Red was, 2 years later we saw an even more effective worm.

38 SQL Slammer Infected 75,000 systems in 10 minutes! At its peak, infections doubled every 8.5 seconds Spread too fast so it burned out available bandwidth

39 Why was Slammer Successful? Worm size: one 376-byte UDP packet Firewalls often let one packet thru Then monitor ongoing connections Expectation was that much more data required for an attack So no need to worry about 1 small packet Slammer defied experts

40 The Worms of the Future?

41 Warhol Worm In the future everybody will be world-famous for 15 minutes Andy Warhol Warhol Worm is designed to infect the entire Internet in 15 minutes Slammer infected 250,000 in 10 minutes Burned out bandwidth Could not have infected entire Internet in 15 minutes too bandwidth intensive Can rapid worm do better than Slammer?

42 A Possible Warhol Worm Seed worm with an initial hit list containing a set of vulnerable IP addresses Depends on the particular exploit Tools exist for identifying vulnerable systems Each successful initial infection would attack selected part of IP address space

43 Flash Worm Can we do better than Warhol worm? Infect entire Internet in less than 15 minutes? Searching for vulnerable IP addresses is the slow part of any worm attack Searching might be bandwidth limited Like Slammer Flash worm designed to infect entire Internet almost instantly

44 Flash Worm Predetermine all vulnerable IP addresses Depends on details of the attack Embed these addresses in worm(s) Results in huge worm(s) But, the worm replicates, it splits No wasted time or bandwidth! Original worm(s) 1st generation 2nd generation

45 Flash Worm Estimated that ideal flash worm could infect the entire Internet in 15 seconds! So how could we defend against this attack? Any defense must be fully automated

46 We have reviewed different types of malware. How do we stop its spread? Protect against the attacks Detect when malware has spread

47 Stopping the Spread of Malware

48 Attack vectors Spread by network (worms) Drive by downloads Code injection vulnerabilities Cross-site scripting (XSS) attacks Buffer overflow vulnerabilities USB sticks

49 Stopping Trojan Horses and Backdoors

50 Defending against a Trojan horse with cryptographic hashes A Trojan horse relies on a user downloading it Using checksums provided by a cryptographic hash verifies that a file has not been tampered with.

51 Sandboxing Applications written for the web browser in JavaScript, Flash, Java, etc. rely on sandboxing. Code run in the sandbox has reduced privileges. Provided it does not escape from its sandbox.

52 App Stores Applications are reviewed by a trusted party. Therefore, applications can be given greater permissions. Examples include Apple's app store, Mozilla's Firefox addon gallery, etc. Labor intensive

53 Firewalls Great tool for fighting viruses: Stop malware from entering the network Detect messages coming from infected machines on your network (botnet)

54 The Weakest Link A system is only as strong as its weakest point. Often, the weak point is the user

55 The Dancing Pigs Problem "Given a choice between dancing pigs and security, users will pick dancing pigs every time." --Edward Felten & Gary McGraw "While amusing, this is unfair: users are never offered security" --Mark Pothier

56 USB Drives Common method of transmitting data between machines. Potentially dangerous: Pass out infected drives to compromise machines Convince users to plug drives into infected machines

57 Why is it dangerous to use USB stick? The difference between inserting a CD into your computer and connecting a hard drive? CDs can autoplay! U3 created USB sticks with a portion that pretended to be CDs (more or less) Allowed applications to run automatically USB Hacksaw ( exploits this trick to infect a machine automatically

58 Parking lot attack "Computer disks and USB sticks were dropped in parking lots 60% of the people who picked them up plugged the devices into office computers. And if the drive or CD had an official logo on it, 90% were installed."

59 Parking lot attack, continued There's no device known to mankind that will prevent people from being idiots. --Mark Rasch [That is] the right response if 60% of people tried to play the USB sticks like ocarinas... But not if they plugged them into their computers. That's what they're for. Quit blaming the victim. They're just trying to get by. --Bruce Schneier

60 Defenses? Disable autoplay on your machines Patch vulnerabilities on your system If you are really paranoid, try to formally analyze your code (this way lies madness) Protecting against vulnerabilities is not enough

61 Malware Detection

62 Methods of Detecting Viruses Signature detection looks for byte patterns in executables to identify an infected file. Low false-positive rate (no files mistakenly identified as infected) Unable to detect unseen viruses Heuristic approaches Integrity checking Behavior blocking

63 Signature Detection Earliest approach used Looks for byte patterns identifying the virus Still the primary technique for virus detection

64 Identifying New Malware Zero-day malware is a previously unseen malicious program. Antivirus vendors use a variety of techniques to discover zero-day malware. Honeypots are vulnerable systems monitored for malicious activity. Once detected, they can be added to a signature database.

65 Signature Detection Example Known Virus Signatures: 0A7E1F FE08C4 300A7B file to test 010A7E1F

66 Signature Detection Example Known Virus Signatures: 0A7E1F FE08C4 300A7B file to test 010A7E1F VIRUS!!!

67 Signature Detection Example Known Virus Signatures: 0A7E1F FE08C4 300A7B file to test F3AE701B No matches: benign

68 Encrypted Viruses Virus writers encrypted the virus code to evade signature detection. While the code remains essentially unchanged, the signature is entirely different.

69 Encrypted Virus Example Known Virus Signatures: 0A7E1F 010A7E1F FE08C4 300A7B 811F7G20 MISSED A KNOWN VIRUS!!!

70 Identifying Encrypted Viruses While the virus payload was encrypted, the decryption code had a signature as well. Virus scanners identified these viruses by looking for the encryption code itself.

71 Polymorphic Viruses Virus writers attempted to hide the encryption code by mutating the code. Creates code functionally equivalent to the original Signature detection cannot be applied to morphed code Obfuscation techniques include dead code insertion NOOP insertion block shuffling changing the instructions

72 Code Mutation Example Original MOV ecx, 0 Morphed version NOP OR eax, 0 XOR ecx, ecx

73 Using Code Emulation to Detect Polymorphic Viruses Most antivirus products were incapable of handling the early polymorphic viruses. Antivirus software detects these viruses through code emulation: 1. The code is run in a virtual machine 2. The virus decrypts itself 3. Signature detection is then used

74 Metamorphic Viruses Mutates entire virus code Variants are hard to distinguish Signature detection becomes extremely difficult

75

76 Some Interesting Viruses Next Generation Virus Construktion Kit (NGVCK) Advanced virus GUI interface create a virus with little technical expertise needed MetaPHOR highly metamorphic virus 90% of its code focuses on its metamorphism has many more

77 Inspecting MetaPHOR MetaPHOR is well commented (including detailed notes about the virus author's politics) Created by The Mental Driller WARNING!!! THIS IS A REAL VIRUS!!! DON'T COMPILE ON YOUR MACHINE!!! METAPHOR.ASM

78 Points to examine in MetaPHOR METAPHOR.ASM How does it find files to infect? What is the payload of the virus? What are its metamorphic techniques? How does it keep from growing too large? Does it rely on encryption at any point?

79 Goats You might have noticed that MetaPHOR is choosy about which files it infects. Why? To analyze viruses, antivirus writers will use goat files. (Conceptually similar to a honeypot). MetaPHOR trys to avoid goats.

80 Alternate Detection Techniques Signature detection is reaching its limits. Other strategies Change detection Anomaly detection

81 Change Detection Viruses must live somewhere If you detect a file has changed, it might have been infected How to detect changes? Hash files and (securely) store hash values Periodically re-compute hashes and compare If hash changes, file might be infected

82 Change Detection Advantages Virtually no false negatives Can even detect previously unknown malware Disadvantages Many files change and often Many false alarms (false positives) Heavy burden on users/administrators If suspicious change detected, then what? Might fall back on signature-based system

83 Anomaly Detection Monitor system for anything unusual or viruslike or potentially malicious or Examples of unusual Files change in some unexpected way System misbehaves in some way Unexpected network activity Unexpected file access, etc., etc., etc., etc. But, we must first define normal Normal can (and must) change over time

84 Advantages Anomaly Detection Chance of detecting unknown malware Disadvantages No proven track record Trudy can make abnormal look normal (go slow) Must be combined with another method (e.g., signature detection) Also popular in intrusion detection (IDS)

85 Other Approaches Newer approaches are being developed, but these are still experimental. Control flow graphs [Krügel et al. 2005, Bruschi et al. 2006] Statistical approaches [Wong & Stamp 2006, Filiol & Josse 2011] MWOR: Worm created at SJSU to evade some of these techniques