TLD Data Analysis. ICANN Tech Day, Dublin. October 19th 2015 Maarten Wullink, SIDN. Klik om de s+jl te bewerken

Transcription

1 Klik om de s+jl te bewerken Klik om de models+jlen te bewerken Tweede niveau Derde niveau Vierde niveau TLD Data Analysis Vijfde niveau ICANN Tech Day, Dublin October 19th 2015 Maarten Wullink, SIDN Wie zijn wij? Mijlpalen Het huidige internet Missie - Visie Diensten Referen@es SamenvaJng 1

2 SIDN Domain name registry for.nl cctld > 5,6 million domain names 2,46 million domain names secured with DNSSEC SIDN Labs is the R&D team of SIDN

3 DNS > 3.1 million resolvers > 1.3 billion query's daily > 300 GB of PCAP data daily

4 ENTRADA ENhanced Top-Level Domain Resilience through Advanced Data Analysis Goal: data-driven improved security & stability of.nl and the Internet at large Problem: for analyzing network data do not work well with large datasets and have limited Main requirement: high-performance, near data warehouse Approach: avoid expensive pcap analysis: Convert pcap data to a performance-op@mized format (key) Perform analysis with tools/engines that leverage that

5 Use Cases Focussed on increasing the security and stability of.nl Visualize DNS pagerns (visualize traffic pagerns for phishing domain names) Detect botnet Phishing (stats.sidnlabs.nl) research with Dutch support for DNS operators

6 Example DNS security scoreboard Resolver

7 DNS Security Scoreboard Goal: Visualize DNS pagerns for malicious How: Combine external phishing feeds with DNS data

8 Architecture Security feed I new event Security feed II new event Hadoop Event Analyzer save enriched event PostgreSQL REST API Web UI retrieve event data

9 Traffic

10 Resolver (RESREP) Goal: Try to detect malicious by assigning scores to resolvers How: resolver behaviour

11 RESREP Concept ISP Resolvers.nl Registry DNS.nl Malicious Spam-runs Botnets like Cutwail agacks DNS and responses

12 RESREP Architecture Root operator.nl Privacy Board ISP network Resolvers RESREP service RESREP Privacy Policy ENTRADA Plaqorm AbuseHUB Abusedesk User HTTP Child operator (example.nl)

13 ENTRADA Architecture DNS big data system Goal: develop and services that further enhance the security and stability of.nl, the DNS, and the Internet at large ENTRADA main components and services Plaqorm and data sources Privacy framework Plaqorm + privacy framework = ENTRADA plumbing

14 ENTRADA Privacy Framework Part of the ENTRADA plumbing Juridisch%en%organisatorisch% ENTRADA%data%plaKorm%(technisch)% toepassingssilos% ENTRADA%privacyraamwerk% Key concepts privacy policy Privacy Board Enforcement Points R&D% licenje% Template% Auteur% (Ontwikkelaar% toepassing%t1)% Aanpassingen% Concept% policy% voor%t1% Privacy% Board% Policy% voor%t1% T1% T2% TN% PEP#G% PEP#A% PEP#O% PEP#G% PEP#A% PEP#O% PEP#G% PEP#A% PEP#O% Security%en%stability%% services%en%dashboards% Data#analyse%% algoritmes% Database%queries% Opslag% DNS%packets%(PCAP)% Policy elements include PEP#V% PEP#V% PEP#V% Verzameling% Purpose Data used.nl%nameservers% Filters DNS%query s%en%responses% period resolvers% Type of (R&D vs.

15 ENTRADA Technical Architecture ENTRADA-specific components Workflow Services DNS Library PCAP Conversion ENTRADA plaqorm Open source Hadoop (generic components) Support IMPALA HDFS Parquet

16 Workflow name server PCAP staging PCAP decode X Y Join Filter Hadoop Impala Analyst Enrich Parquet Monitoring Metrics Import Query data available for analysis within 10 minutes

17 Performance Example query, count # ipv4 queries per day. select concat_ws( -,day,month,year), count(1) from dns.queries where ipv=4 group by concat_ws( -,day,month,year) Query 1 Year of data is 2.2TB Parquet ~ 52TB of PCAP

18 ENTRADA Status Name server feeds 2 Queries per day ~320M Daily PCAP volume(gzipped) ~70GB Daily Parquet volume ~14GB Months opera@onal 18 Total # queries stored > 74B Total Parquet volume > 3TB HDFS (3x replica@on) > 9TB Cluster capacity ~150B-200B tuples

19 Conclusions Technical: Hadoop HDFS + Parquet + Impala is a winning combina@on! Contribu@ons: Research by SIDN Labs and universi@es Iden@fied malicious domain names and botnets External data feed to the Abuse Informa@on Exchange Insight into DNS query data

20 Future Work Combine data from.nl name server with scans of the complete.nl zone and ISP data. Get data from more name servers and resolvers Expand Open Data program

21 and Feedback Maarten Wullink Senior Research hgps://stats.sidnlabs.nl