Nathan Turajski Jamz Yaneza Best Practices for Evaluating Anti-spam Solutions VB 2005, Dublin, Ireland
Benchmarking Validation Methodologies Accurate Comprehensive Fair Filtering Techniques Pattern matching, Heuristics, IP blocking, Whitelist/Blacklist, Challenge/Response, Community..
Anti-spam Solutions Current Solutions Software Appliance Services Legislation Methods Catch rate (effectiveness) Error rate (accuracy)
Content Defined Spam UCE, commercial bulk mail Consumers: well defined Enterprise: borderline Non-spam Appropriate, predictable, traceable Graymail Inappropriate to environment Requires exception capability
Factors for Evaluating Solutions Primary Effectiveness Accuracy Resiliency Secondary Administration Integration
Testing Failures Confused spam type classification Non real-world environment Short-term testing cycle Fixed regional origins Fixed language type Non-relative industry. Etc.
Spam Trends Estimates vary, but the total amount was usually agreed to have passed 40% by the beginning of 2002 Email was 50% SPAM by January of 2003 65% of all email was SPAM by 2004 Almost 80% of all email is currently either unwanted advertising or virus-ridden
Evaluation Guidelines Valid vs. illegitimate mail sampling over time period spam/month
30% Monthly Spam Growth (2005) Total Spam Mails Received 250000 232,194 200000 150000 163,425 202,867 183,269 100000 50000 0 March April May June
Evaluation Guidelines Predominant language English vs. Non-english New Spam Mails Received June, 46% May, 62% April, 51% March, 66% English Non-English March, 34% April, 49% May, 38% June, 54%
Evaluation Guidelines What Country does Spam like the Most? 2.97% 3.83% 2.77% 3.94% 2.21% 2.05% 1.84% 10.34% 21.42% 21.78% China United States Republic of Korea Brazil Japan France Spain Taiwan Israel Germany Point of origin broad mixed sampling Spam Countries Brazil 4% Japan 3% Spain 3% Germany 2% Poland 2% Republic of Korea 31% Switzerland 4% France 5% http://www.trendmicro.com/spam-map/default.asp United States 20% China 26%
Industry definitions Evaluation Guidelines overlap of needs vs. excess 8% Spam Categories 10% 18% 24% 36% 4% 23% 1% Adult 19% Bad Samples Commercial 8% 3% Financial Health 14% Others Non-English 32%
Chinese Language (traditional) Traditional Chinese (snapshot) Work Spiritual 0.3% Sexual 7% Health 4% Financial 22% Education 4% Spiritual Sexual Health Financial Education Commercial Other Work 23% Other 2% Commercial 38% Summary: 38% commercial offers, 23% work related, 22% financial, 7% sex related
Chinese Language (simplified) Sexual 5% Spiritual 0.04% Health 2% Simplified Chinese (snapshot) Financial 17% Education 4% Work Spiritual Sexual Health Financial Education Commercial Work 1% Other 2% Other Summary: Commercial 69% 69% commercial offers, 17% financial, 7% sex related, 4% education
German Language Health 1% Financial 1% Commercial 12% German (snapshot) Sexual Health Financial Commercial Other Sexual 15% Other 71% Summary: 15% sex related, 12% commercial, 71% mixed offers
Evaluation Guidelines Timeliness update frequency distribution strain on network/system correction efficiency
Evaluation Guidelines Summary Efficiency and accuracy dependent on spam classification and audience Used testing samples to be valid and fixed Overall results used for evaluation False positive graymail vs. legitimate mail Unmodified message delivery
Other Considerations Product configuration and tuning Out of the box state Vendor recommended tuning Tolerance rating based on audience target Long-term testing timeframe
Other Considerations Filter technique testing Signature matching Focus: catch efficiency and update timeliness Heuristic rules Focus: false positive rate and mitigation tools Hybrid techniques Focus: accuracy and update timeliness IP filtering Focus: delivery efficiency and mitigation tools
Other Considerations Performance Deployment time Management reporting tools Update overheard Message latency
SUMMARY Comprehensive evaluation includes scalability and resiliency long term performance customer specific goals exception handling minimal administration
Questions?
Mass-mailing malware spam Malware Tracking Center # of reported infections (uniqe) 2,000,000 1,500,000 1,000,000 500,000 0 January February March April May June July August September October November December 2002 2003 2004 Bulk-mailing Malware Summary: 2003, due to Mimail, Blaster, and Sobig 2004, due to Bagle, Mydoom, Netsky, and Sasser