Best Practices for Evaluating Anti-spam Solutions

Transcription

1 Nathan Turajski Jamz Yaneza Best Practices for Evaluating Anti-spam Solutions VB 2005, Dublin, Ireland

2 Benchmarking Validation Methodologies Accurate Comprehensive Fair Filtering Techniques Pattern matching, Heuristics, IP blocking, Whitelist/Blacklist, Challenge/Response, Community..

3 Anti-spam Solutions Current Solutions Software Appliance Services Legislation Methods Catch rate (effectiveness) Error rate (accuracy)

4 Content Defined Spam UCE, commercial bulk mail Consumers: well defined Enterprise: borderline Non-spam Appropriate, predictable, traceable Graymail Inappropriate to environment Requires exception capability

5 Factors for Evaluating Solutions Primary Effectiveness Accuracy Resiliency Secondary Administration Integration

6 Testing Failures Confused spam type classification Non real-world environment Short-term testing cycle Fixed regional origins Fixed language type Non-relative industry. Etc.

7 Spam Trends Estimates vary, but the total amount was usually agreed to have passed 40% by the beginning of was 50% SPAM by January of % of all was SPAM by 2004 Almost 80% of all is currently either unwanted advertising or virus-ridden

8 Evaluation Guidelines Valid vs. illegitimate mail sampling over time period spam/month

9 30% Monthly Spam Growth (2005) Total Spam Mails Received , , , , March April May June

10 Evaluation Guidelines Predominant language English vs. Non-english New Spam Mails Received June, 46% May, 62% April, 51% March, 66% English Non-English March, 34% April, 49% May, 38% June, 54%

11 Evaluation Guidelines What Country does Spam like the Most? 2.97% 3.83% 2.77% 3.94% 2.21% 2.05% 1.84% 10.34% 21.42% 21.78% China United States Republic of Korea Brazil Japan France Spain Taiwan Israel Germany Point of origin broad mixed sampling Spam Countries Brazil 4% Japan 3% Spain 3% Germany 2% Poland 2% Republic of Korea 31% Switzerland 4% France 5% United States 20% China 26%

12 Industry definitions Evaluation Guidelines overlap of needs vs. excess 8% Spam Categories 10% 18% 24% 36% 4% 23% 1% Adult 19% Bad Samples Commercial 8% 3% Financial Health 14% Others Non-English 32%

13 Chinese Language (traditional) Traditional Chinese (snapshot) Work Spiritual 0.3% Sexual 7% Health 4% Financial 22% Education 4% Spiritual Sexual Health Financial Education Commercial Other Work 23% Other 2% Commercial 38% Summary: 38% commercial offers, 23% work related, 22% financial, 7% sex related

14 Chinese Language (simplified) Sexual 5% Spiritual 0.04% Health 2% Simplified Chinese (snapshot) Financial 17% Education 4% Work Spiritual Sexual Health Financial Education Commercial Work 1% Other 2% Other Summary: Commercial 69% 69% commercial offers, 17% financial, 7% sex related, 4% education

15 German Language Health 1% Financial 1% Commercial 12% German (snapshot) Sexual Health Financial Commercial Other Sexual 15% Other 71% Summary: 15% sex related, 12% commercial, 71% mixed offers

16 Evaluation Guidelines Timeliness update frequency distribution strain on network/system correction efficiency

17 Evaluation Guidelines Summary Efficiency and accuracy dependent on spam classification and audience Used testing samples to be valid and fixed Overall results used for evaluation False positive graymail vs. legitimate mail Unmodified message delivery

18 Other Considerations Product configuration and tuning Out of the box state Vendor recommended tuning Tolerance rating based on audience target Long-term testing timeframe

19 Other Considerations Filter technique testing Signature matching Focus: catch efficiency and update timeliness Heuristic rules Focus: false positive rate and mitigation tools Hybrid techniques Focus: accuracy and update timeliness IP filtering Focus: delivery efficiency and mitigation tools

20 Other Considerations Performance Deployment time Management reporting tools Update overheard Message latency

21 SUMMARY Comprehensive evaluation includes scalability and resiliency long term performance customer specific goals exception handling minimal administration

22 Questions?

23 Mass-mailing malware spam Malware Tracking Center # of reported infections (uniqe) 2,000,000 1,500,000 1,000, ,000 0 January February March April May June July August September October November December Bulk-mailing Malware Summary: 2003, due to Mimail, Blaster, and Sobig 2004, due to Bagle, Mydoom, Netsky, and Sasser