Resolving problems with SMTP Security Server and CVP operating in Check Point NG

PROFESSIONAL SECURITY SYSTEMS Resolving problems with SMTP Security Server and CVP operating in Check Point NG by Mariusz Stawowski CCSA/CCSE (4.1x, NG) The Check Point FireWall-1 Next Generation (NG) security system provides powerful capabilities in terms of detailed and restrictive electronic mail control. It is performed by means of a dedicated security component SMTP Security Server. In terms of electronic mail control it has, among others, the following capabilities: commands correctness and application data format control, blocking of prohibited MIME subtypes and attachments types (e.g. executable Visual Basic files and EXE files), erasing of prohibited scripts in HTML messages, blocking messages which are too large, blocking unauthorized Mail Relaying attempts, sending messages and attachments for anti-virus check on external servers CVP (Content Vectoring Protocol). Subjects related to IT system protection against threats related to using electronic mail services are covered in the document titled Practical methods for electronic mail protection. When solving problems related to operation of the SMTP Security Server and CVP, the basic principles of their operation should be taken into consideration. SMTP Security Server consists of two processes asmtpd and MDQ (Mail Dequeuer). Application control of electronic mail stem from Firewall-1 security policy settings (using of SMTP Resource). SMTP connections, which according to security policy are subject to application control are received by the asmtpd process (SMTP Security Server acts as a target mail server). The asmtpd process performs a detailed check if the connection comply with the SMTP protocol standard. After receiving the message, the asmtpd process stores it on disk on the Firewall machine under the $FWDIR/spool directory in the d_state subdirectory. Another process MDQ is responsible for periodical reading (every 2 seconds by default) of messages stored in the spool directory, performing further examination and processing of these messages, and sending them to the target mail server. MDQ reads e-mail messages from the d_state directory and performs required processing (e.g. removes prohibited attachments, sends messages to the CVP server). Upon completion of this check, MDQ stores messages in the d_sender directory, where they will be sent from to the target mail server. In case of failure to send messages (e.g. the target mail server is unreachable) they will be stored in the d_resend directory. Again, the MDQ process will periodically try to send the messages stored in this directory. CLICO Ltd. Al. 3-go Maja 7, 30-063 Kraków, Poland; Tel: +48 12 6325166; +48 12 2927525 Fax: +48 12 6323698; E-mail: support@clico.pl, orders@clico.pl; Ftp.clico.pl; http://www.clico.pl

1. Problem identification When solving e-mail related problems, we should analyze the whole process of e-mail messages processing in Check Point FireWall-1 security system (see figure): 1. Mail message is received from the sender, through the asmtpd (SMTP Security Server introduces itself as a target mail server). 2. asmtpd stores received message in the spool/d_state directory. 3. MDQ reads the message and performs check according to the FireWall-1 security policy (e.g. transfers it for CVP check). 4. After the check, the message goes to the spool/d_sender directory. 5. MDQ sends the message to the target mail server. 6. Undelivered messages go to the /spool/d_resend directory, from where the MDQ will try to send them again in specific time. Mail sender FireWall-1 Gateway Mail server 1 asmtpd MDQ 2 5 spool d_resend d_sender d_state 6 3 4 CVP server SMTP Security Server processes (asmtpd, MDQ) are activated when the FireWall-1 security policy requires application control of SMTP protocol. The number of running processes and the port on which SMTP Security Server listens to should be configured on the Firewall machine in the $FWDIR/conf/fwauthd.conf directory (see the product documentation): 25 fwssd in.asmtpd wait 0 In case of problems to run SMTP Security Server control, the settings in the fwauthd.conf file should be examined. 2002 CLICO LTD. ALL RIGHTS RESERVED 2

1.1 Firewall status analysis When we notice problems in handling electronic mail in the Check Point Firewall-1 security system, in the first place we should examine the following elements: Firewall machines and CVP status e.g. free disk space, particularly free space in the $FWDIR/spool directory, system directory and directory for temporary files (/tmp), RAM memory status (fw ctl pstat), processor load, Firewall network interfaces status (fw ctl iflist), If FireWall-1 and CVP server versions are up to date (e.g. hofixes installed). 1.2 Firewall logs analysis The problem cause can be provisionally identified on the basis of information contained in Check Point FireWall-1 security system logs. Using Check Point Log Viewer (SmartView Tracker) tool we read Info field in logs containing information about unsuccessful mail message delivery (Action = reject, Service = smtp). Log content reason Connection to Content Security Server failed reason Content Security Server has approved the requested resource reason Connection to Final-MTA failed Meaning no connectivity with the CVP server CVP control works properly; the problem exists on FireWall-1 or the target SMTP server no connectivity with the target SMTP server 1.3 Network communication verification Network communication with the CVP server can be performed from the Firewall machine using the following command: telnet <CVP server IP address> <CVP port> - 18181/TCP is the default CVP server Network communication with the target SMTP server can be performed from the Firewall machine using the following command: telnet <SMTP server IP address> 25 Network communication with the FireWall-1 SMTP Security server can be performed from an external machine (on the message sender side) using the following command: telnet <SMTP server IP address> 25 In order to check if required communication link exists (e.g. between MDQ and CVP server), the network traffic in the specific network segment can be read with a sniffer program when needed. 2002 CLICO LTD. ALL RIGHTS RESERVED 3

Depending on the configuration, mail delivery through the SMTP Security Server can be done using DNS (i.e. FireWall-1 searches DNS for MX record of a specific mail domain and reads IP address of the target SMTP server). The way of delivering mail by FireWall-1 can be configured in the SMTP Resource General Mail Delivery definition. Network communication with the DNS server can be performed from the Firewall machine using the following command: nslookup Server: <DNS server name> Address: <DNS server IP address> > set type=mx > we submit the name of the DNS target domain (e.g. checkpoint.com) <domain name> MX preference = X, mail exchanger = <mail server name > <mail server name> internet address = <mail server IP address> 1.4 The spool directory content review Mail messages examined in the FireWall-1 security system are temporarily stored in the $FWDIR/spool directory or the directory pointed out in the Firewall object configuration (Gateway Advanced SMTP Spool Directory). When viewing content of the spool directory, we can notice, that undelivered mail messages are stored in the d_resend subdirectory. In the $FWDIR/spool directory we can find files of three types. The filenames of these files start with the T, R and E letters: T (Temporary) - temporary file containing a message which has not been delivered completely, R (Ready) - file containing a message ready to be delivered, E (Error) - file containing a message which can not be sent. We should take note of the number of files of E type. 2002 CLICO LTD. ALL RIGHTS RESERVED 4

2. Configuration verification/correction Stopping e-mail messages in the security system FireWall-1 usually results from configuration restrictions (e.g. message size limit). The basic restrictions are set up in the SMTP Resource settings and in the Firewall object definition (Gateway Advanced SMTP). 2002 CLICO LTD. ALL RIGHTS RESERVED 5

In case of problems with mail delivery, the FireWall-1 configuration should be examined in this respect. If search for a target mail server is performed through an MX (Mail exchanger) the DNS configuration on the Firewall machine should be examined. In the typical FireWall-1 configuration the Mail Relaying is disabled so as to prevent users from unauthorized use of the server (e.g. for sending spam). It may happen that improper configuration of the SMTP Resource with this respect will cause rejection of mail messages. 2002 CLICO LTD. ALL RIGHTS RESERVED 6

Disruptions in CVP control usually result from the fact, that FireWall-1 configuration had not been properly prepared for use with CVP server. Because of that, the guidelines provided by a specific CVP server manufacturer should be used, and in case when no such guidelines exist, the typical settings should be implemented. Typical settings include turning CVP control on only for messages content and attachments (see figure). Deployment of the CVP control in the FireWall-1 security system can only be performed with a product which has OPSEC compliance certificate issued by Check Point. Configuration with other products will not be accepted by the Technical Support. OPSEC certificates have been issued, among others, for the Trend Micro VirusWall and Aladdin esafe. 2002 CLICO LTD. ALL RIGHTS RESERVED 7

It is also important to properly define OPSEC server (i.e. Server/Client Entities, CVP Options) to be used in FireWall-1 security policy (see figure). 2002 CLICO LTD. ALL RIGHTS RESERVED 8

Details of FireWall-1 configuration parameters can be found in the $FWDIR/conf/objects_5_0.C file. These parameters can be adjusted when needed. Changes in the objects_5_0.c file should be done on Management Server with the dbedit application or GUI console Check Point Database Tool (GuiDBedit). Typical FireWall-1 parameter settings in terms of SMTP protocol control include: enabling of local encoding for attachment names (e.g. polish characters ) :smtp_encoded_content_field (true) enabling of handling messages from mail clients not using RFC 821 standard :smtp_rfc821 (false) In case of problems with MDQ operation, the multithread support for this process can be disabled: mdq_run_multi_threaded (false) 2002 CLICO LTD. ALL RIGHTS RESERVED 9

3. Corrective actions 3.1 System restart Depending on the identified problem source, in the first place we should restart the FireWall-1 unit (fwstop/fwstart) or CVP server, and to observe if this action has resolved the problem. After receiving of first SMTP messages, the processes responsible for their support in the FireWall-1 security system should be started. This can be checked in $FWDIR/tmp directory, where we should find in.asmtpd.pid and mdq.pid files. 3.2 Increase of the frequency of sending undelivered messages In case of a great number of undelivered messages in the $FWDIR/spool/d_resend directory, it is recommended that the time Mail resend period in the Firewall object definition (Gateway Advanced SMTP Mail resend period) be increased (e.g. up to 20 seconds). After the parameter has been changed, the security policy should be installed and when needed, the FireWall-1 module should be restarted (see para 3.1). 2002 CLICO LTD. ALL RIGHTS RESERVED 10

3.3 Manual clearing of the spool directory Disruption of SMTP Security Server or CVP server operation may take place during inspection of a particular SMTP message or MIME attachment. Such cases are very rare but they might happen. In such situation the SMTP message causing problems should be identified and erased from the spool directory. This can be done in the following way: manually move all message files from the spool directory to the other backup directory, so as to empty the spool directory, increase the value of the Mail resend period parameter (see p. 3.2), restart FireWall-1 unit (see p. 3.1), copy messages from the backup directory to the spool directory (5-10 messages at a time), FireWall-1 will automatically distribute messages from the spool directory to the mail servers, continue copying messages until messages remain in the spool directory for a significant period of time, identify and separate the message causing problems from the others, continue operation until all the messages from the backup directory have been sent. After the mail message causing problems has been identified it should not be copied into the spool directory again. From the message header we can read the e-mail address of the message sender and ask him to send the message again. This is very likely, that the message contains a virus. Hence, this is not advisable, that the message be manually transferred to the internal network without CVP check. 3.4 Temporary disabling of SMTP application protections In the critical situation when all the SMTP messages received by the Firewall go into the spool directory and are not sent further, the CVP control in SMTP Resource definition should be disabled. Then, we should install the Firewall security policy and check if the situation has improved. If so, we should verify the configuration and CVP server status or contact Technical Support. If despite enabling of CVP control, the system operation has not improved, the SMTP Security Server application control should be disabled. To do so, we modify the rule which enables relaying/receiving of electronic mail in the FireWall-1 security policy. In the rule, we disable SMTP application control (i.e. we change the setting Service = smtp->resource into Service = smtp). Next, we should verify the security system configuration again or contact Technical Support. 4. Debug tests Debug tests of the FireWall-1 security system should be performed under the guidance of Check Point Technical Support. Enabling logons in debug mode for SMTP protocol can be done in many ways depending on version of a system (FP) and an operating system, e.g. NG FP2 version for Windows 2000: fw debug in.asmtpd on FWASMTPD_DEBUG=3 fw debug mdq on MDQ_DEBUG_LEVEL 3 Debug results can be found in the mdq.elg and asmtpd.elg files. 2002 CLICO LTD. ALL RIGHTS RESERVED 11