A Survey on Malware Analysis and Detection Techniques

A Survey on Malware Analysis and Detection Techniques 1 Farhood Norouzizadeh Dezfouli, 2 Ali Dehghantanha, 3 Ramlan Mahmod, 4 Nor Fazlida Binti Mohd Sani, 5 Solahuddin bin Shamsuddin, 6 Farid Daryabar 1,6 Faculty of Computer Science and Information Technology, University Putra Malaysia, {farhood1990, farid0fx}@gmail.com 2.3,4 Faculty of Computer Science and Information Technology, University Putra Malaysia, {alid, ramlan, fazlida} @fsktm.upm.edu.my 5 Cyber Security Malaysia, Solahuddin@cybersecurity.my Abstract Nowadays the use of the internet has become an integral part. Parallel to this, we have seen an overwhelming flood of new malware. And malware detectors are the primary tools in defense against malware. The quality of such a detector will be decided by the techniques it used. This paper discusses the background of the types of malware and technological innovations that led to the present day computing environment. We have also identified inadequacies in the signature-based, behavior-based detection methods. Keywords: malware, types of malwares, malware detection techniques, signature-based, behaviourbased 1. Introduction Malware is any malicious software that is used by the attacker to enter the system as unauthorized access, harm other programs or alter data and spread it to another device through internet. The combination of malicious and software is the creation of the term of malware. Therefore, malware has another name called malicious code. According to [1], malware is used to set on the computer and affect the performance of the computer to give the remarkable controls to the attackers. This kind of control can be overwriting the owner of the user. Nowadays malware is considered as worldwide threats in the computing world. It continues to grow and evolve in complexity. Some malware can be very easy to detect and remove from the computer or other device such as mobile phone thought antivirus software. This antivirus software is used to check all the files in the computer and find out those files which contain any virus signature. A quality of malware detector is based on the techniques that are used in the detector. The condition that a good malware detection technique must have includes identifying the malware code either it has been hidden or embedded inside those files. Besides that it should be able to look for others unknown or new malicious code. This paper includes 3 sections. The first section discusses malware and its different types. The second section discusses various existing malware detection techniques. Finally, in the last section we conclude the paper and summarize the findings. 2. Malware There are six types of malware which are virus, worm, backdoor, trojan horse, spyware and adware. According to [2], there are three characteristics that are used to associate with those malware types. 1) Self-replicating malware will keep creating the new copies of the code when it is being broadcast. Besides that, malware can also be broadcasted passively via the file that has been infected and copied by users, but this kind of malware cannot self-replicate. 2) According to [2], because of the first characteristics, the population growth of malware describes the overall change in the number of malware. Malware that doesn t self-replicate will always have a zero population growth may self-replicate. 3) In order to exist, parasitic malware requires some executable code that can be executed automatically and stealthy in any machine. Those executable codes are boot block code on a International Journal of Advancements in Computing Technology(IJACT) Volume5, Number14, October 2013 42

disk or on a USB thumb drive, binary code in applications, interpreted code and others. Besides that, it also involves the source code such as application scripting languages, but this kind of code might have to compile before it being executed. Table 1 shows different types of malicious code [3]. Type of Malicious Code Virus Worm Backdoor Trojan horse Malicious mobile code Table 1. Types of Malicious Codes Defining Characteristics Significant Examples Infects a host file such as executable, word processing document. Self-replicates. Usually requires human interaction to replicate through opening a link that sent by unknown sender, opening a program or install software that includes the code inside the software. Can be broadcast through the network connection. Self-replicates. Normally can be broadcast automatically. Bypasses normal security controls to give an attacker access. Disguises itself as a useful program while masking hidden malicious purpose. Consists of lightweight programs that are downloaded from a remote system and executed locally with minimal or no user intervention. Typically written in Javascript, VBScript, Java or ActiveX. Michelangelo, CIH. Morris Worm, Code Red, SQL Slammer. Netcat and Virtual Network Computing (VNC) Both of it can be used legitimately as remote administration tools and illegitimately as an attack tools. Setiri, Hydan. Cross Site Scripting. Malware have different types of payloads to perform different functionalities that challenge detection. Many types of malware perform different activities; however, this paper will focus into following categories: 2.1. Virus A virus is a computer program that is hidden into a executable file when the infected program is executed. It is one of the types of malware. According to [4], when the virus code executes, it attempts to replicate itself into other executable code; when it succeeds, the file is considered infected by the virus. Those files that have been infected by the virus code can infect the new code in turn. This kind of action is called self-replication and is one of the characteristic of malware. On MS-DOS system, files that have the extensions.exe,.com,.bat and.sys can be infected easier. The most harmless viruses are only able to self-replicate and spread to new programs from infected files. Besides that, the virus can damage the data file, copy data continuously until the capacity of the computer is full, alter the original data and keep shutting down or open the same program in the shortest time. Boot sector virus infects the boot sector or master boot record on a computer system. The first step to infect the computer system is to alter the original boot code and replace the original code with infected boot code. And from that infected boot code spread the virus to other codes. Normally, boot sector virus is hard to detect, because when a computer is being switched on, the first process that is going to load will be the boot sector. After computer system has been infected, the viruses have the full control of the computer system and are able to do anything on the computer system based on the virus code. Figure 1 shows that boot sector viruses target MBR or PBS instructions that are executed during the PC s boot-up sequence [4]. 43

Figure 1. Viruses targeting MBR and PBS. The objective is to infect as much executable files as possible with the extensions.com,.exe,.ovl and other overlay files. According to [4], there are a few examples of software products that support macros- commands embedded into documents for the official purpose of enhancing the application, interacting with user, or automating tasks. Those software products are Microsoft Office, WordPerfect Office, StarOffice and AutoCAD. Figure 2 shows that the visual basic editor, built into Microsoft Office, allows users to embed executable instructions into Office document [5]. 2.2. Backdoor Figure 2. Visual basic editor allowing embedding executables in Office Documents. According to [6], a backdoor is a way that bypasses a normal security check through the device. Backdoor is not only used for illegitimate purposes but it is also used for legitimate reason as well, such as sometimes programmers will create backdoor to skip an authentication process when debugging a network server. Figure 3. An example of using backdoor for legitimate purposes. 44

There is another special situation for legitimate purpose of backdoor called RAT, which is stand for Remote Administration Tool or Remote Access Trojan. The function of this backdoor is similar to other remote computer software. Once RAT is installed on a computer, another computer can monitor and have full control over that computer. According to [6], user may install these to access a work computer from home, or to allow help desk staff to diagnose and fix a computer problem from afar. According to [7], backdoor are focused on giving the attacker access to the target machine. These kinds of access have many different forms. It depends on the attacker s aim and the function of backdoor. Table 2 shows several types of access by using backdoor [7]. Backdoor Access Local Escalation of Privilege Remote Execution of Individual Commands Remote Command- Line Access Remote Control of the GUI Description Table 2. Access types using backdoor This kind of backdoor lets attackers which are having an account on the system suddenly change their privilege level to the highest privilege level such as administrator. With these special privileges, the attacker is able to access any file that is stored on the victim machine without any limit. This kind of backdoor allows attacker to send a message or command to the victim machine and execute the command on the victim machine. After the backdoor executes the command from the attacker, it will return the result to the attacker automatically. This kind of backdoor is also known as remote shell. It is able to allow attacker to access victim machine and type a command that attacker wants to execute on a command prompt through the network connection. Besides that, attacker is able to use all of the functions of command such as open a program through command prompt, copy or alter data that are in the victim machine and delete the data. These kind of backdoors are more harmful than other backdoor access. This kind of backdoor will allow attacker to see the GUI of the victim computer, able to control the victim mouse movements and enter keystrokes through the network connection. Besides that, attackers are able to monitor all of the victim s activity. 2.3. Worm A worm is a self-replicating piece of code that spreads via networks and usually doesn't require human interaction to propagate [8]. Worm is quite similar to virus as both of them share some common characteristics. Worms can be identified by their obvious characteristics which is self-replicating just like a virus. However, self-replication of a worm is distinct from virus as its self-replication process can be done standalone without dependent to executable code. Table 3 shows the comparison and differences between virus and worm. Table 3. Comparison between virus and worm Type of Malware Self- Way of Propagation User required in propagation replication Virus Yes Infecting a file in computer, such as a document file. Worm Yes Spread from machine to machine across networks. process User interaction is required for virus propagation, such as opening a document file in computer. Worm can propagate in either present or absent of user interaction. Humans are slow, compared to computers and computer networks. Worm thus has the potential to spread very, very quickly, because human doesn t have to be involved in the process of worm propagation [9]. After a worm attacks a targeted system, the targeted system will be taken over and used by worm as a staging ground. From there, the worm will start its scanning for other vulnerable 45

systems and conquer other vulnerable systems found. The existence of a network is very important for worm propagation because worms cannot spread from one machine to another if there is no network. Whether user interaction is present or absent, it doesn t affect the propagation of worm. Worm is able to propagate in both situations. A worm using email which is called email worm is one of the worm examples that can propagate with user interaction. For example, email worm arrives to user s computer as an email attachment and tricks the user to run it or open it. After it is executed or opened by the user, the worm will harvest email addresses off the machine and mails itself to those addresses. Worm that can propagate without user interaction doesn t have to trick user to open an email or email attachment. This type of worm will find the vulnerability of a system and attack the targeted system based on the system weakness. A user interaction is not required at all if the worm spread using buffer overruns between long-running network server processes on different machines. Once a computer or system is being attacked by a worm, it will cause many very bad effects. Due to worm can self-replicate itself on the system, it could send out thousands of copies of itself and cause a huge devastating effect. The worm will use up the system or computer processing speed and space. The worms on a system will consume too much system memory and cause the system process to be accomplished slower and even can cause system to stop responding. Some worms have been designed to tunnel into user system. This will allow malicious users to control tunnelled system remotely. So, the malicious users can hijack internet access or email of the victim computer and use these services for illegal activities. Worm will also cause data corruption on the computer or system. Some worms are designed to perform vandalism actions on computer data. The data on victim computer will be corrupted or erased once it is attacked by this type of worms. The result of such worm attacks is that the computer owner will be unable to run a program or access data on their computers. Worm also causes damage to human psychology. When a computer operation is affected by worm attack, it not only takes personal resources and times to fix the damage, but it also will leave the victim with unsafe and vulnerable feeling. 2.4. Trojan Horse Trojan horse is named after the Greek Army invasion method to the city of Troy where the Greek army was hiding in a gift of a horse hollow statue. This also applies to a type of computer virus that attacks the victim computer in a similar way, the Trojan horse virus will act as a gift or part of a computer and invades the computer silently. A Trojan horse is a program that appears to have some useful of benign purpose, but really masks some hidden malicious functionality [10]. The Trojan horse will appear as a normal or useful software at the first glance but will actually do damage once it is installed or run on the computer. Users who receive a Trojan horse will be tricked to open or run it because Trojan horse appears like legitimate software or files from a legitimate source. In this case, unsuspecting users are the vulnerability and become entry point for the malicious software to successfully deploy on the computer. There are different techniques used by attackers to create a Trojan horse virus. For the simplest level of Trojan horse techniques, an attacker just needs to alter the malicious code name on a system. With this action, the Trojan horse will appear to belong to that machine. Another technique is naming a backdoor program the same name with normal program in the computer. This allows an attacker to operate undetected. One simple Trojan horse naming technique which is popularly used by attackers against systems operated based on Windows, is done by creating a file named with a bunch of spaces in it to obscure the file's type, for example (text.txt.exe). The program is still executable with.exe at the end after all the spaces, but careless or unwary user might not notice the.exe suffix. When user looks at this file with Window Explorer file viewer, it will confuse the user as the program will appear in normal text file look. Besides that, Trojan horse will mask itself as some normal process or program running on a machine. So, Trojan horse will look like a program that belongs to the system and is hard to detect by users. Users will continue their daily activity without being aware of the intrusion of Trojan horse. Key logging is one of the main effects caused by Trojan horse. Once Trojan horse invades a machine, it will implement a key logger program on that machine. This program will record all buttons that are being pressed by the user on their keyboards and send a report of it to the hacker. Private and confidential information such as banking password, online purchases and online transaction will be tracked through key logger program and sent to hackers. Some hackers who might be interested in vandalism will 46

create a Trojan horse to attack victim s computer. This type of Trojan horse can corrupt delicate data of the operating system and can even cause an operating system to crash. There is another type of Trojan horse that will install a remote access program on victim s computer which is known as backdoor. Backdoor allows hackers to access victim s computer system freely and modify the personal files inside. It will cause the victim s private and confidential information stored inside the computer to be leaked and compromised. 2.5. Spyware A new category of malware has gained momentum and it is called Spyware. Spyware can be defined as Any software that monitors user behavior, or gathers information about the user without adequate notice, consent, or control from the user [11]. Another definition of Spyware is given as Software that gathers information about use of a computer, usually without the knowledge of the owner of the computer, and relays the information across the Internet to a third party location [12]. Spyware is sometimes known as scumware, stealware, theftware and occasionally mixed up with computer virus and worm [13]. Spyware as the term appeared on 16 October 1995 in a Usenet post in which hardware which can be used for espionage was given this name. In 2000 the founder of Zone Labs, Gregor Freund used Spyware as a term in a press release for their firewall product [14]. Many Spyware usually come as part of a software (pre-installed software, downloaded from the web or software purchased). Parallel to this, Spyware may come via e-mail as an attachment, part of the message or may through a hyperlink on email. Spyware can be capable of capturing keystrokes, taking screen shots of user activity, saving or storing personal information. Unfortunately, it can lead to financial loss as in identity theft and credit card fraud [15]. 2.6. Adware In 1987, the first publicly recorded use of the word adware appeared on the Internet in the Usenet Newsgroup comp.sys.mac. Amusingly, the post refers to a Macintosh application rather than normal application. Until 15 years later, Permissioned Media, INC released a software program that sent a link to itself to everyone in the Outlook contact list [16]. Adware and spyware have several similarities, where both try to gather information from the user. However, Adware focuses on marketing, advertisements (by using popup) or tries to redirect a user to other web pages rather than Spyware. 3. Malware Detection Techniques A malware detector (commonly known as virus scanner) is the implementation of some malware detection techniques. The malware detector attempts to determine whether a program has malware behavior [17]. The malware detectors use a pattern matching approach which is susceptible to obfuscations used by hackers. A malware detector identifies and contains malware before it can reach a computer or network. Current malware detectors are based on scan string or signatures, suspicious byte sequences of instructions and data. The antivirus software will compare their database with the files on hard disk, USB as well as within RAM. Malware detectors take two inputs: Knowledge of the malware signature or behavior and the other is the program under inspection. Once the malware detector has the knowledge of what is considered malicious behavior and the program under inspection, it can employ its detection technique to decide if the program is malware or benign [17]. Besides that malware detector is usually a part of Intrusion Detection Systems (IDS). Techniques used for detecting malware have two main categories as illustrated in figure 4: behaviorbased detection and signature-based detection. 47

3.1. Behavior-based Detection Figure 4. Malware detection Techniques. Behavior based detection has a big difference with surface scanning method. Behavior based detection detects malware by identifying actions performed by the malware, while surface scanning method detects malware by binary pattern. Behavior-based detection doesn t rely on virus signature database to detect malware. The programs having the same behavior but different syntax are collected. Thus, various samples of malware can be identified with this single behavior signature. Behavior based detection can help to detect the malware that keep on generating new mutants. The new mutants generated will always use the services and system resources in the same similar manner. The behavior based detection is basically made up of three components, which are data collection, interpretation and matching algorithm. For the data collection, this component is used to collect the captured static and dynamic information. For interpretation, this component s main function is to convert collected raw information into intermediate representations. For matching algorithm, this component will compare each representation with the behavior signature. Figure 5 illustrates the behavior malware detection technique. Figure 5. Behavior detectors [18] 48

3.2. Specification-based Detection Specification-based detection is one of the types of behavior-based detection. However, the limitation of specification-based detection system is that it is usually difficult to accurately specify the entire set of valid behaviors a system should exhibit. Dynamic Specification-based Detection: use behavior observed at runtime to determine the maliciousness of an executable. Static Specification-based Detection: The malware is determined with PUI s structural properties. Hybrid Specification-based Detection 3.3. Signature-based Detection According to [19], each virus will have the signature of the developer and the signatures usually are a sequence of bytes within the malware code to declare that the program scanned is malicious in nature. Malware are categorized into three groups: basic, polymorphic and metamorphic malwares. Figure 3 shows that the program entry points have been changed and the malicious code has been added behind the program s original code. Figure 6 shows that basic kind of virus [19]. Figure 6. Basic kind of virus [19] The difference between basic malware and polymorphic is that polymorphic malware have involved the encrypted malicious code with the decryption code. Inside the virus code, there is a polymorphic engine to enable the polymorphic virus. Each time when the polymorphic virus is executed, the polymorphic engine will produce or mutate the new virus. According to [19], using signature-based detection to detect the virus is such a difficult task because each transformed virus will auto generate a new signature. Therefore, it is the difficult part for signature-based detection. Figure 7 shows polymorphic virus [19]. Figure 7. Polymorphic virus [19] Metamorphic malware is self-replicated. It will be able to reprogram itself by using the certain obfuscation techniques. When each new variant is produced, the signature will be different with the previous signature. Besides that, it is hard to store each signature that comes from the same malware after it is generated. According to [19], after disassembly, the engine will alter the program code and create a new code that keeps the previous functionality, but the interface will be different between the original and the new code. Figure 8 shows metamorphic malware [19]. Figure 8. Metamorphic virus [19] In addition, there were several works on analysis of cloud and virtualized environments [20-22], privacy issues that may arise during forensics investigation [23-28], mobile device investigation [29-30] and greening digital forensics process [31-32]. 49

4. Conclusion and future work The malware threats faced by computer users have been increasing from year to years. Malware nowadays is getting more intelligent and able to attack user computer in many different ways. It is impossible to detect and eliminate all malware as new malware is being created every day; there are no ways a computer user can escape from threat of these malwares. For sure, the information security and malware prevention technologies need to be constantly updated as the malware evolves continuously. The defensive communities against malware need to continuously improve their capabilities and invent new defensive and security technologies that can help to give a better protection to computer user against malware threats. Keeping up-to-date about the newest malware threat is essential for every computer user if they don t want to fall inside the trap of malware and become the next victim of malware. The law should be reinforced and heavier penalties or punishments need to be enforced to those who are involved in the illegal activity over internet. 5. Acknowledgments This work has been supported by the MOSTI- Science Fund project 01-01-04-SF1677. The views and opinions expressed in this article are those of authors alone and not the organizations with whom authors are or have been associated/supported. 6. References [1] ED Skoudis, Lenny Zeltser, Malware: Fighting Malicious Code, Pearson, 2004. [2] John Aycock, Computer Viruses and Malware, Springer, pp.25-26, 2006. [3] ED Skoudis, Lenny Zeltser, Malware: Fighting Malicious Code, Pearson, Chapter 1, 2004. [4] John Aycock, Computer Viruses and Malware, Springer, pp.28-29, 2006. [5] ED Skoudis, Lenny Zeltser, 2004, Malware: Fighting Malicious Code, Pearson, Chapter 2 Viruses. [6] John Aycock, 2006, Computer Viruses and Malware, Springer, pp.27-28. [7] ED Skoudis, Lenny Zeltser, 2004, Malware: Fighting Malicious Code, Pearson, Chapter 5 Backdoors. [8] Lenny (2003) Malware Fighting Malicious Code. Prentice Hall PTR. [9] P. Ferrie and F. Perriot. Detecting complex viruses. SecurityFocus, 6 December 2004. [10] Lenny (2003) Malware Fighting Malicious Code. Prentice Hall PTR. [11] M. Boldt, B. Carlsson, Privacy-Invasive Software and Mechanisms, Systems and Networks Communications, 2006. International Conference, pp 21, Oct. 2006. [12] Lenny (2003) Malware Fighting Malicious Code. Prentice Hall PTR. [13] Richard H. Stern, FTC cracks down on Spyware and PC hijacking, but not true lies, IEEE Computer Society, 2005. [14] Mathias Klang, Spyware: Paying for Sofware with our Privacy, International Review of law Computers & Technology, vol. 17, no. 3, pages 313-322, November. 2003. [15] Zone Alarm, Press Release 2000, http://www.zonealarm.com (Date access: 1-5-2012). [16] Spyware. www.us-cert.gov/reading_room/spywarehome_0905.pdf (Date access: 2-5-2012) [17] Eric Chien, Techniques of Adware and Spyware, Symantec security, 2005. [18] Greoigre Jacob,Herve Debar,Eric Fillol, Behavioral detection of malware:from a survey towards an established taxonomy,springer-verlag France 2008 [19] Vinod P.,Survey on Malware Detection Methods. [20] Farid Daryabar, Ali Dehghantanha, Farhood Norouzi, Farbod Mahmoodi, Analysis of Virtual Honeynet and VLAN-Based Virtual Networks, in the 2011 IEEE International Symposium on Humanities, Science & Engineering Research (SHUSER2011), 5-8 Jun 2011, Kuala Lumpor- Malaysia. [21] Farid Daryabar, Ali Dehghantanha, Nur Izura Udzir, Nor Fazlida binti Mohd Sani, and Solahuddin bin Shamsuddin, "Towards secure model for SCADA systems", In Cyber Security, Cyber Warfare and Digital Forensic (CyberSec), 2012 International Conference on, pp. 60-64. IEEE, 2012. [22] Farhood Norouzizadeh Dezfouli, Ali Dehghantanha, Ramlan Mahmoud, Nor Fazlida Binti Mohd Sani, and Solahuddin bin Shamsuddin, "Volatile memory acquisition using backup for forensic investigation", In Cyber Security, Cyber Warfare and Digital Forensic (CyberSec), 2012 International Conference on, pp. 186-189. IEEE, 2012. [23] M. Damshenas, A. Dehghantanha, R. Mahmoud, S. Bin Shamsuddin, Forensics investigation challenges in cloud computing environments, Cyber Warfare and Digital Forensics (CyberSec), pp. 190-194, 2012. [24] S. H. Mohtasebi, A. Dehghantanha, Defusing the Hazards of Social Network Services, International Journal of Digital Information, pp. 504-515, 2012. 50

[25] A. Dehghantanha, R. Mahmod, N. I Udzir, Z.A. Zulkarnain, User-centered Privacy and Trust Model in Cloud Computing Systems, Computer And Network Technology, pp. 326-332, 2009. [26] A. Dehghantanha, Xml-Based Privacy Model in Pervasive Computing, Master thesis- University Putra Malaysia 2008. [27] C. Sagaran, A. Dehghantanha, R Ramli, A User-Centered Context-sensitive Privacy Model in Pervasive Systems, Communication Software and Networks, pp. 78-82, 2010. [28] A. Dehghantanha, N. Udzir, R. Mahmod, Evaluating user-centered privacy model (UPM) in pervasive computing systems, Computational Intelligence in Security for Information Systems, pp. 272-284, 2011. [29] A. Dehghantanha, R. Mahmod, UPM: User-Centered Privacy Model in Pervasive Computing Systems, Future Computer and Communication, pp. 65-70, 2009. [30] S. Parvez, A. Dehghantanha, HG. Broujerdi, Framework of digital forensics for the Samsung Star Series phone, Electronics Computer Technology (ICECT), Volume 2, pp. 264-267, 2011. [31] S. H. Mohtasebi, A. Dehghantanha, H. G. Broujerdi, Smartphone Forensics: A Case Study with Nokia E5-00 Mobile Phone, International Journal of Digital Information and Wireless Communications (IJDIWC),volume 1, issue 3, pp. 651-655, 2012. [32] Nor Fazlida Binti Mohd Sani, Solahuddin bin Shamsuddin, "A Data-centric Model for Smartphone Security", IJACT, Vol. 5, No. 9, pp. 9-17, 2013. 51