Scan Sequence and Action in Microsoft Forefront Protection 2010 for Exchange Server

Size: px
Start display at page:

Download "Scan Sequence and Action in Microsoft Forefront Protection 2010 for Exchange Server"

Transcription

1 Scan Sequence and Action in Microsoft Forefront Protection 2010 for Exchange Server Published: October, 2009 Software version: Forefront Protection 2010 for Exchange Server Carolyn Liu

2 Introduction... 1 Exchange and Forefront hook... 3 Scan Processes... 4 Scan Process Type... 5 Actions for Malware Scans and Filters... 6 Action Table... 7 Scan Job and Filter Types... 9 Scan Sequence...10 Message Header Scan and Action Sequence Message Scan and Action Sequence Summary...12

3 Introduction Microsoft Forefront Protection for Exchange Server (FPE) is a leading solution for securing your messaging environment. Its multi-engine antimalware solution is a proven security product that has helped many customers to secure their system. With the introduction of a Premium Antispam solution and seamless integration with Exchange Hosted Filtering, FPE will bring protection for Exchange to the next level. Users familiar with FPE know that besides malware scanning, there are various filtering options. This article provides insight into the scanning options, as well as the FPE process sequence for malware scanning and filtering. Administrators can leverage this knowledge to maintain a secure and sophisticated messaging system. The concept of server roles was introduced in Exchange Server Server roles enable Exchange to clearly classify different functionalities within Exchange and enable administrators to categorize one or more roles on different servers and locations in the organization. Exchange Server 2007 introduced the following five roles: Edge Transport, Hub Transport, Client Access,, and Unified Messaging. There is also a combined Hub Transport/ role. For more detail about these server roles, see the following article: On Edge and Hub Transport roles, Microsoft Exchange provides a Transport Agent framework. This is a plug-in architecture that enables Exchange message security vendors to supply their own agent to process messages passing through the transport pipeline. An agent processes messages based on SMTP events and communicates to the Exchange Transport pipeline for processing results and actions, such as discarding a spam message or adding a legal disclaimer footer when a message leaves an organization. The SMTP events processing sequence is shown in the diagram below: OnConnect OnEhloCommand OnHeloCommand OnAuthCommand OnEndOfAuthentication OnMailCommand OnRcptToCommand OnDataCommand OnEndOfHeaders OnEndOfData OnRsetCommand OnReject OnDisconnect OnopCommand OnHelpCommand Figure 1 SMTP Events Processing in Exchange Transport The processing sequence moves from left to right.

4 Based on different mail processing requests and the mail delivery status, each agent may intercept different SMTP events. For example, the OnConnect event is often processed by the antispam agent. For more information about the Exchange Transport architecture and detailed SMTP events, see the following article: In the Categorizer (see Figure 2), the routing agent processes the routing events and categorizes and routes messages already received by the organization to proper mail store(s) or other organization(s). On the Edge and Hub Transport roles, Forefront provides real-time protection via the Exchange Transport framework. This is processed in several stages. First, Forefront Antispam agents process s at the Edge role via comprehensive mechanisms (IP block list, Sender ID, SMTP filtering, Content Filtering), stopping spam s before they enter an organization. Next, the Forefront Antimalware routing agent passes the messages to Forefront scanning proceses for Malware and filtering processing. The Forefront routing agent in the Categorizer intercepts messages that are passing through in real-time and routes the data to one of the Forefront scanning processes using an Inter-Process Communication mechanism for malware scanning and various filtering operations. Figure 2, below, describes the SMTP events going through an Exchange Edge role and different process points by Transport agents. EdgeTransportSvc.exe Smtp Receive Connector Selection Tarpitting IP Connection throttling MEx Event Dispatch Inbound TLS Inbound MLS Transport SMTP Receive Agents OnConnect OnEhloCommand OnHeloCommand OnAuthCommand OnEndOfAuthentication OnMailCommand OnRcptToCommand OnDataCommand OnEndOfHeaders Header Firewall OnEndOfData OnRsetCommand OnReject OnDisconnect OnopCommand OnHelpCommand Mex Event Dispatch Connection Filtering Agent AddressRewritingInbound Agent Edge Rule Agent Sender ID Agent priority Recipient Filter Agent Sender Filter Agent Content Filtering Agent on restart Stranded Mail Scanner fork/create Protocol Analysis Agent Attachment Filtering Agent create Messages Jet Figure 2 Exchange Transport

5 Exchange and Forefront hook On the Exchange role, Exchange provides a virus scanning API (VSAPI) that enables antivirus vendors to scan messages passing through the Exchange Mail Store (mailbox databases). When a mail client such as Outlook accesses mail, FPE provides real-time protection via the Exchange VSAPI plug-in to intercept messages and route the data to one of the FPE scanning processes for malware scanning and filtering. This is an additional layer of protection. Because the Mail Store can be very heavily loaded, we advise customers to deploy their messaging system and protection solution carefully. For example, FPE has a virus stamp feature that stamps a message when it is scanned on the Edge or Hub role so that a redundant scan is not performed when the message is stored in the mailbox. Internet Inbound Inbound Outbound Outbound FSE-protected Edge Inbound Outbound Inbound Inbound FSE-protected Hub Outbound Inbound Inbound Outbound FSE-protected Hub Outbound Figure 3 Exchange and Forefront Topology

6 Scan Processes For all Exchange roles that have FPE installed, FPE uses a similar common entity to perform malware scanning and filtering: a scan process that communicates to the hook agent and works independently to avoid disruption of any Exchange processes. A scan process analyzes messages and applies appropriate file navigation, filters, and malware scans for each part of a message. There are multiple scanning processes per scan job type (default number is 4), configurable by the administrator, which enable concurrent processing of multiple messages and reduce the direct impact of the scanning process on the core Exchange process (preventing, for example, the possibility of crashing due to the deep content inspection of potentially malicious code). Currently, the FPE scan process encompasses the following scanning technologies: Malware scan (viruses, spyware, and worms) Filters, which include: o Sender-domain: This filter examines an from particular senders or domains. o Subject line: This filter examines the subject line of s. o File: This filter examines file names, file size, file types, or file extensions based on file content. o Keyword: This filter compares words and phrases in the message body of an e- mail. o Allowed senders: This filter is similar to the sender-domain filter but allows the administrator to bypass any content protection filters. Figure 4 Forefront Security for Exchange Server Transport Scan Process Figure 4 describes the Forefront scan process basic diagram in Exchange Edge and Hub roles.

7 Exchange Transport Antispam Agents Forefront Antimalware Agent Other Agents File Navigators Scan Process Scan Process AntimalwareEngine Adapters Keyword and Filtering Engines Quarantine and Actions Figure 5 Forefront Security for Exchange Server Scan Process on Role Figure 5 describes the Forefront scan process basic diagram on Exchange role Exchange VSAPI Framework Forefront VSAPI hook agent File Navigators Scan Process Scan Process AntimalwareEngine Adapters Keyword and Filtering Engines Quarantine and Actions. SCAN PROCESS TYPE There are four scan process types: Transport, Realtime, Scheduled, and On-demand.

8 Transport Scan Job The Transport Scan process (FSCTransportScanner.exe) is installed on the Exchange Edge/Hub Transport role, and scans messages as they arrive from the Exchange Transport Service (Edge- Transport.exe) and are intercepted by the FPE transport routing agent (FSEAgent.dll). Realtime Scan Job The Realtime Scan process (FSCRealtimeScanner.exe) is installed on the Exchange role and scans messages when a user accesses mail via the mail client (such as Outlook or Outlook Web Access Client). The messages are intercepted by the FPE VSAPI hook agent. Scheduled Scan Job The Scheduled Scan process (FSCScheduledScanner.exe) is architecturally the same as the Reatime Scan Job, except the trigger is different. The Scheduled scan job is scheduled via the Windows Task Scheduler and leverages Exchange background scanning a separate task thread that traverses through items in the Exchange store database looking for instances of items that have not been scanned. On-Demand Scan Job The On-Demand Scan process has been architecturally redesigned for the this release due to Exchange Server 2010 architecture changes. For Exchange Server 2010, the on-demand scan leverages EWS (Exchange Web Services) from the Exchange Client Access Server (CAS) Role. Ondemand scanning in Exchange Server 2007 installations will still use the older design (ADO). ACTIONS FOR MALWARE SCANS AND FILTERS When malware is found or a filter is matched, the FPE scan process will take necessary actions on the relevant message part. It is necessary to have a clear understanding of each action taken by each FPE scan process. The action definitions are: Clean A message part (which could be a message body or an attachment) is cleaned. This option only applies to virus scans. If cleaning is successful, the original part will be replaced by the cleaned part and reassembled into the original format of the message. For example, an contains the attachment a.zip. This zip file contains two files: b1.doc and b2.exe. If b1.doc is infected but cleaned by FPE and b2.exe is clean, a modified a.zip that contains the cleaned b1.doc and the original b2.exe will arrive in the user s inbox. A message part is deleted and replaced with custom defined deletion text. For example, an e- mail contains the attachment a.zip. This zip file contains two files, b1.doc and b2.exe. If b1.doc is infected, it will be deleted, and a modified a.zip that contains the deletion text b1.txt and the original b2.exe will arrive at the user s inbox. Deletion Text b1.txt contains the following text by default: Forefront Security for Exchange Server detected b1.doc to be infected.

9 The FPE administrator can customizethe Deletion Text. For more information on customizing tion Text, refer the FPE Operations Guide. The entire message is deleted and will not be delivered to the recipient(s). This option always applies to worms (a special virus type). This option is supported in realtime (Exchange ) scanning as well. In VSAPI 2.6, the VIRSCAN_DELETE_MESSAGE error code will indicate that the top level message is deleted, effectively purging the message. See Table 1 and Table 2 for what this action applies to. Identify A user-defined word or phase will be pre-pended to the subject line. other action is taken on the message. This is supported in filtering. It is available for keyword filtering, file filtering, subject line filtering, and sender-domain filtering. For example, if a keyword is matched within an message body, text defined by the FPE administrator will be pre-pended to the subject line, indicating that a matching keyword was found. The default pre-pended-text is SUSPECT: FPE administrators can also use this option to add a MIME message header so that it can be identified later for processing into folders at a user s inbox or for other purposes identified by the FPE administrator. By default, X-Junk-Mail is written to the header. When the option is selected, an incident log entry will be created indicating the infection and filtering information, and the rest of the scanning and filtering process continues. ACTION TABLE The following table shows the action options within FPE filters and default actions among various scan job types. Filter Type Allowed Subject Line Sender Scan Job Type Hub Transport or Edge Transport File Filter Identify Keyword Filter Identify N/A 1 Identify Sender- Domain Identify Default: Default: Default: Identi- Default: Identi-

10 Realtime Identify fy fy N/A N/A 1 Scheduled Default: Default: Skip (detect N/A N/A 1 Default: Skip (detect On-Demand Default: Default: Skip (detect N/A N/A 1 Default: Skip (detect Default: Default: Skip (detect Default: Skip (detect Table 1 te: 1. The Allowed Sender List is used to identify sender address/domains that are allowed to bypass the configured filters (File Filter, Keyword Filter, Subject Line Filter, Sender-Domain Filter). The following table shows the action choices in FPE among various scan job types for malware scans. Malware Type Virus Spyware Scan Job Type Edge Transport or Hub Transport Clean Default: Clean Clean Default:

11 Realtime Scheduled On-Demand Default: Clean Clean Default: Clean Clean Default: Default: Default: (2) Table 2 SCAN JOB AND FILTER TYPES The following table shows correlation between the scan job and filter types. Filter Type Scan Job Type File Keyword Allowed Senders Subject Lines Sender-Domain Hub Transport or Edge Transport Realtime Scheduled On-Demand Table 3

12 Scan Sequence When a message is scanned by an FPE scan process, it is processed by antimalware engines and filtering engines in one pass. This is done by navigating each part of the encoded message or compressed files in a recursive manner. This maximizes the performance and increases the complexity of the process. The following diagrams depict the logic flow of the scan and action sequence for the scan process. MESSAGE HEADER SCAN AND ACTION SEQUENCE Process message headers Antimalware/Filtering Agent Message Header Scanning Does message match an allowed sender list for subject or sender filtering? Does message match a sender/domain filter? Does message header match a subject filter Is the action purge? [Transport] Is the action identify? Is the action purge? [Transport] Is the action identify? Message removed from pipeline Tag(s) added to header(s) Message removed from pipeline Tag(s) added to header(s) MESSAGE SCAN AND ACTION SEQUENCE The following diagrams depict the logic flow of the scan and action sequence for the message body and attachments. te: The scan sequence is a recursive operation based on file navigation flow. End of execution means to go back to the last level of execution of the recursive action. For example, a message contains a.zip as an attachment, and a.zip contains b.exe and c.doc. If b.exe

13 is spyware but not a virus, and the spyware scan action is, file b.exe will be replaced with Deletion Text b.txt, and the execution will end for b.exe and the flow will go back to the scan of the next container subpart, c.doc. Process all file parts from message Antimalware/Filtering Agent File Filtering Worm Keyword Filtering Virus Spyware Check if is container [Transport] Is this file a message body? Does file contain a worm? Does file contain a virus? Does message contain spyware? Does sender match an allowed sender list for keyword filtering? Does sender match an allowed sender list Does file name or type for file filtering? match a file filter? If container, have all subparts been scanned yet? Process all file parts from container ; action is skip Does message body match a keyword filter? Is the action clean? ; action is skip Is the action delete? Is the action purge? ; action is skip Is the action purge? Is the action purge? Was part of a container? Is the action delete? Was part of a container? ; action is skip [Transport] Is the action identify? [Transport] Is the action identify? Is the action delete? Was part of a container? Can file be rebuilt? Was clean successful? Deletion text inserted Can file be rebuilt? Message removed from pipeline Deletion text inserted Can file be rebuilt? Message removed from pipeline Tag(s) added to header(s) Message removed from pipeline Message removed from pipeline Tag(s) added to header(s) Deletion text inserted Treated as corrupted compressed file Treated as corrupted compressed file New container replaces old Treated as corrupted compressed file New container replaces old New container replaces old Was file a subpart of a container? End of execution Continue to workload pipeline

14 Summary We summarized some of the core functionalities in Forefront Protection for Exchange Server and provided detailed views of malware scanning and filtering. This should give you an in-depth understanding of the product to leverage the superior protection provided by FPE. The vision behind this product line is to maximize protection by building a solution that is componentized and is adaptive to current and future scanning technologies. We are working hard towards that goal. Your feedback is critical for improving the existing product and building more successful ones in the future.