Lumension AntiVirus Technical Notification December 19, 2012

Transcription

1 Lumension AntiVirus Technical Notification December 19, 2012 Frequently Asked Questions Copyright 2009, Lumension

2 Lumension deeply regrets any inconvenience and disruption of business operations suffered as a result of this incident with Lumension Antivirus on the Lumension Endpoint Management and Security Suite (L.E.M.S.S.). We have placed the full weight of our company resources behind getting this issue resolved immediately and we are working with our customers to assist them in getting their operations back up and running as quickly as possible. If you have been impacted and we have not already spoken with you, please contact our support line. Together with our third party AV partner, we identified the root cause of this issue and jointly deployed a replacement definition file. We have also implemented processes to ensure that the issue does not happen again. Q: What happened? A: At 4:30 a.m. EST Saturday, December 15, 2012, Lumension received an antivirus (AV) definition file (v ) from our third party AV partner. That definition file falsely detected Windows systems and application files as viruses. Depending on your Real-Time Monitoring and/or Recurring AV Scan policies, this caused system or application files to be either deleted or quarantined. This in turn caused impacted systems either to become unstable. Updated definition files were released at approximately 2:30 p.m. EST Saturday, December 15 via the Lumension Content Distribution network. The new AV definition files (v or higher) will be pulled by the L.E.M.S.S. server based on the AV Subscription Service Configuration setting at the next update. By default, the L.E.M.S.S. server will look for and syndicate updated content every 60 minutes. These new AV definition files will then replace the faulty definition file automatically. The Lumension Support team is working 24x7 with customers whose systems have been adversely affected and assisting them to get back up and running. 2

3 Q: What should you do if are unsure if you have been affected or not?» Customers will know they have been impacted if their Virus and Malware Event Alerts include Heuristic_Anomaly.A virus. Customers can check for this in L.E.M.S.S. by navigating to Review > Virus and Malware Event Alerts. The view can be filtered by entering "Heuristic_Anomoly.A" into the Virus or malware name filter at the top of the page. If no alerts appear with this name, the customer has not been affected by this issue.» If your L.E.M.S.S. AV endpoints were off at the time this faulty definition file was sent and have remained off during the time that it took to send a new definition file, you will not be affected by this issue. When your endpoints check in, they will receive the latest definition files.» An updated definition file (v ) has been released so it is imperative you check that the AV definition file on your L.E.M.S.S. server is v or higher. New definition files are released typically twice a day.» To check the AV definition file on your L.E.M.S.S. Server, select Tools --> Subscription Updates --> Configure button --> AntiVirus tab. Validate that the AV definition file is v or higher. If it is not, click on the Update Definitions button to receive the latest version. To check the AV definition file version on your endpoints, first go to Manage --> Endpoints and select the AntiVirus tab; sort using the AV Definition Version column, if necessary.» Lumension recommends that you disable Real Time Scans and Scheduled Scan until you are sure the AV definition files have been updated. Lumension highly recommends you select a setting of Attempt to clean, then quarantine. Once the latest AV definition file (v or higher) is in place, it is safe to return to normal scanning mode(s).» Notify any remote users that they must connect to your corporate network to receive updated virus definitions.» If you feel that you have been affected, please contact Lumension Support. Q: Has the root cause been determined? A: Yes. The root cause has been determined between Lumension and our third party AV provider. In October, our 3 rd party AV partner updated their NSE7 engine to improve their ability to implement generic detections. This was done to improve the quality of detection and the speed of the AV engine. This new signature (Heuristic_Anomaly.A) works as designed for engines released after the October engine change, including all OEM s. On Friday December 14 th, 2012 our AV partner implemented a new heuristic signature of the NSE7 engine to replace an existing sandbox detection, and to improve the quality of detection and speed of the engine. As a result of the Lumension incidents experienced, it was discovered that OEM engine NSE v generates severe False Positives ( FP), or Windows files detected as malicious, with the new (Heuristic_Anomaly.A) signature. Our third party AV partner has now been able to reproduce the 3

4 exact incident experienced by Lumension customers, and is confident that the root cause has been identified. Q: How could Lumension s AV partner release these faulty definition- files to the public? A: Our third party AV partner has a systematic and advanced FP testing mechanism to make sure that new definition file releases do NOT generate severe false positives. FP testing is a systematic procedure that takes well over three hours of computing time, at every definition file release, twice a day. The partner s current FP system utilizes engines NSE v and NSE v These engines worked as planned with the updated signatures and the definition file release procedures did not detect any problems whatsoever with the definition files released to the public on Saturday. However, with engine NSE v that Lumension was using, the definition file released on Saturday results in the false positives experienced by some Lumension customers. OEM customers of our third party AV supplier are normally exempted from general public releases as they have a standard 14-day verification period to ensure compatibility before a new engine release is distributed into their release tracks. As a result, OEMs of our third party AV supplier, of which Lumension is one, could still be running the NSE AV engine on December 14, The current FP testing procedures run in the third party AV supplier s lab was run on the latest engine version, and as a result no errors were detected with NSE Unfortunately, these current testing procedures did not include all engines in release to the general public and OEM partners of the supplier, but only the latest version of the main release. As a result, Lumension s version of the third party supplier AV engine (NSE ) was not part of the FP test performed after the (Heuristic-Anomaly. A) signature was added. Q: What is being done by Lumension s AV partner to prevent this from happening again? A: Effective December 18th, 2012 following the release of NSE v to all NSE7 customers and discontinuation of NSE v , Lumension s third party AV partner has implemented new standardized procedures to its definition file release process to ensure that distributed definition files undergo the FP testing process with all engines that are in release to the general public. Q: What is being done by Lumension to prevent this from happening again? 1. A: Effective December 17, Lumension has implemented a 2 to 12-hour delay for all AV definition files received from its 3 rd party AV partner. During this delay time, these files will undergo a round of production testing at our Galway, Ireland facility before being distributed to customers via the Lumension Content Distribution Network for syndication across L.E.M.S.S. servers. In addition to the current testing we do, Lumension will also add false positive testing for critical system files. 4

5 2. Effective December 18, 2012 NSE engine, v was released to OEM customers. This ensures that Lumension customers have the same engine that Norman is using and testing. This engine has been distributed to all L.E.M.S.S. servers after testing was completed in our Galway office. a. Lumension s AV partner has committed that all versions of the engine in field will always be part of the acceptance testing. 3. Lumension is planning to add a capability which will allow customers to receive, test, and then complete a controlled release of new virus definitions to the endpoints. This new capability is being scoped right now as a top priority for the next feature release. 4. In the next maintenance release of L.E.M.S.S. AntiVirus (v.7.2 Update 2), targeted for early January 2013, new AntiVirus policies will be defaulted to be First attempt to clean, then quarantine when a new virus is found. Existing policies will not be changed automatically, although customers can easily change their configurations in their current policies today. Q: Has this ever happened before? A: Lumension has not previously experienced an issue like this. While events of this nature are rare, they are not of unheard of in the industry. For instance, recently several major AV companies have also grappled with issues arising from faulty virus definition files. Q: When will we find out more? A: Customers can find out the latest information by checking with Lumension Support. We will make every attempt to provide accurate and relevant data as soon as it becomes available through Lumension KnowledgeBase article KB 910, Customer Support team, the Lumension Customer Portal and the Lumension Community. 5