netflow-indexer Documentation

Transcription

1 netflow-indexer Documentation Release Justin Azoff May 02, 2012

2 CONTENTS 1 Installation Install prerequisites Install netflow-indexer Configuration Example Configuration files Cron Daily compaction Example Session Indexing data Searching the index Specifying output columns Dumping data API Searching with the API Example File metadata Searching with pynfdump Indices and tables 10 Index 11 i

3 netflow-indexer Documentation, Release Netflow-indexer is a program that uses xapian to index the flat file databases used by nfdump or flow-tools. Contents: CONTENTS 1

4 CHAPTER ONE INSTALLATION 1.1 Install prerequisites netflow indexer uses the python xapian bindings. The IPy module is used for some subnet calculations to support CIDR searching. On debian you can install all the dependencies using: # apt-get install python-pip python-xapian xapian-tools python-ipy 1.2 Install netflow-indexer I recommend installing netflow-indexer into a virtual environment: # pip install -s -E /usr/local/python_env/ netflowindexer tar.gz Then modify your path or source the activation script: # PATH=/usr/local/python_env/bin/:$PATH 2

5 CHAPTER TWO CONFIGURATION Netflow-indexer uses a small configuration file that setups the type of indexer to use and the location of the files on disk. It has the following settings: indexer - the type of indexer to use. nfdump, nfdump_full, flowtools, or flowtools_full dbpath - the path to save the indexes to fileglob - the shell glob that will expand to the flow data files for the current hour allfileglob - the shell glob that will expand to all flow data files pathregex - a regular expression or simple string used to extract metadata from flow file paths. 2.1 Example Configuration files nfdump using full indexing(recommended) [nfi] indexer = nfdump_full dbpath = /data/nfdump_xap flowpath = /data/nfsen/profiles/live/podium fileglob = %(flowpath)s/nfcapd.%(year)s%(month)s%(day)s* allfileglob = %(flowpath)s/nfcapd.* pathregex = /profiles/:profile/:source/nfcapd nfdump [nfi] indexer = nfdump dbpath = /data/nfdump_xap flowpath = /data/nfsen/profiles/live/podium fileglob = %(flowpath)s/nfcapd.%(year)s%(month)s%(day)s* allfileglob = %(flowpath)s/nfcapd.* pathregex = /profiles/:profile/:source/nfcapd flow-tools 3

6 netflow-indexer Documentation, Release [nfi] indexer = flowtools dbpath = /usr/local/var/db/flows/nfi flowpath = /usr/local/var/db/flows/packeteer fileglob= %(flowpath)s/%(year)s/%(year)s-%(month)s/%(year)s-%(month)s-%(day)s/ft-v05.%(year)s-%(month allfileglob = %(flowpath)s/*/*/*/ft-v05.* 2.2 Cron Netflow-indexer should be run from cron 5 minutes after every hour when using the nfdump indexer and every 5 minutes when using the nfdump_full indexer: MAILTO=root PATH=/usr/local/python_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 45 0 * * * cd /data/nfdump_xap/ &&./daily_compact > /dev/null */5 * * * * sleep 30;netflow-index-update /data/nfdump_xap/nfdump.ini 55 0 * * * netflow-index-cleanup /data/nfdump_xap/nfdump.ini -d 2.3 Daily compaction xapian allows you to compact an index for read-only use. Compaction yields disk usage and speed improvements. daily compaction is a work in progress examples/daily_compact.sh #!/bin/sh DAY= date +"%Y%m%d" -d "60 minutes ago"./xap_compact ${DAY}.db examples/xap_compact.sh #!/bin/sh orig="$1" tmp=tmp_$$.db tmp2=tmp2_$$.db if [ -e $orig/.compacted ] ; then exit 0 fi xapian-compact -F $orig $tmp && mv $orig $tmp2 && mv $tmp $orig && rm -rf $tmp2 && touch $orig/.compa 2.2. Cron 4

7 CHAPTER THREE EXAMPLE SESSION 3.1 Indexing data Tell the netflow indexer to index the current netflow files For this example I deleted todays index so it can be re-created netflow@nf:~$ netflow-index-update /data/nfdump_xap/nfdump.ini read /data/nfsen/profiles/live/podium/nfcapd in 2.4 seconds ips. read /data/nfsen/profiles/live/podium/nfcapd in 2.5 seconds ips. read /data/nfsen/profiles/live/podium/nfcapd in 3.8 seconds ips. read /data/nfsen/profiles/live/podium/nfcapd in 2.7 seconds ips.... read /data/nfsen/profiles/live/podium/nfcapd in 1.3 seconds ips. read /data/nfsen/profiles/live/podium/nfcapd in 1.3 seconds ips. read /data/nfsen/profiles/live/podium/nfcapd in 1.2 seconds ips. read /data/nfsen/profiles/live/podium/nfcapd in 1.2 seconds ips. Flush took 7.4 seconds.... read /data/nfsen/profiles/live/podium/nfcapd in 7.4 seconds ips. read /data/nfsen/profiles/live/podium/nfcapd in 7.1 seconds ips. read /data/nfsen/profiles/live/podium/nfcapd in 5.7 seconds ips. Flush took 28.9 seconds. Running the indexer when more data is available does an incremental update: netflow@nf:~$ netflow-index-update /data/nfdump_xap/nfdump.ini read /data/nfsen/profiles/live/podium/nfcapd in 3.7 seconds ips. read /data/nfsen/profiles/live/podium/nfcapd in 3.7 seconds ips. read /data/nfsen/profiles/live/podium/nfcapd in 4.2 seconds ips. Flush took 7.0 seconds. netflow@nf:~$ netflow-index-update /data/nfdump_xap/nfdump.ini netflow@nf:~$ When performing an index for the first time you should use the full-index or -f option to index all the data. By default netflow-indexer only tries indexing files that match fileglob: netflow-index-update /data/nfdump_xap/nfdump.ini --full-index 3.2 Searching the index Search the index for : 5

8 netflow-indexer Documentation, Release # is an address that just scanned us remote@nf:~$ time netflow-index-search /data/nfdump_xap/nfdump.ini /data/nfdump_xap/ db :35: :40: :45: :50: :55: :00: :05: :40: :45: :50: :55:00 real 0m0.072s This output shows that it was present in the index in 11 5 minute chunks. Searching the 30 day index takes only slightly longer and returns the same results: remote@nf:~$ netflow-index-search-all /data/nfdump_xap/nfdump.ini Searching for an IP that does not exist in the index is very fast: remote@nf:~$ time netflow-index-search-all /data/nfdump_xap/nfdump.ini real 0m0.097s 3.3 Specifying output columns netflow-index-search and netflow-index-search-all support a -c option which selects what columns should be output. By default only time is output. The other built-in field is filename. Additional fields are made available by using the pathregex configuration option. Columns can be selected by using two methods: -c time -c filename or: -c time,filename 3.4 Dumping data netflow-index-search and netflow-index-search-all support a -d option which automatically runs the appropriate netflow tool for you: remote@nf:~$ time netflow-index-search-all /data/nfdump_xap/nfdump.ini d head :38: TCP : > : :38: TCP : > : :38: TCP :22 -> : :38: TCP : > : You can also use the -f option to pass an additional filter: remote@nf:~$ netflow-index-search-all /data/nfdump_xap/nfdump.ini d -f not port Specifying output columns 6

9 CHAPTER FOUR API 4.1 Searching with the API class netflowindexer.main.searcher(ini_file) Create a new searcher instance. Call with the path to the ini file list_databases() Return a list of database files in the dbpath directory search(database, ips, dump=false, filter=none, mode=none) Search a specific database file Parameters database The full path to a database file. ips a list of ip addresses to search for. dump if True dump the full netflow records, otherwise just the seen timeslots filter optional additional netflow search filter to be used when dump=true mode set to pipe to have nfdump list pipe delimited records search_all(ips, dump=false, filter=none, mode=none) Search all database files. Takes the same parameters as search() 4.2 Example The Searcher class can be used to search for records: >>> from netflowindexer import Searcher >>> s = Searcher("/spare/tmp/netflow/nfdump.ini") >>> print s.list_databases() [ /spare/tmp/netflow/ db ] >>> for record in s.search_all([ ]):... print record :00: :05: :10: :15: :20:

10 netflow-indexer Documentation, Release >>> for record in s.search_all([ ], dump=true):... print record :59: UDP : > : :59: UDP :53 -> : :59: UDP : > : :59: UDP :53 -> : :59: UDP : > : :59: UDP :53 -> : :59: UDP : > : :59: UDP : > : File metadata Search results are actually an object. str() will return simply the time of the matching flow records, but there are other fields available: >>> for record in s.search_all([ ]):... print repr(record) SearchResult(filename=/spare/tmp/netflow/nfcapd , time= :00:00, profile=tmp) SearchResult(filename=/spare/tmp/netflow/nfcapd , time= :05:00, profile=tmp) SearchResult(filename=/spare/tmp/netflow/nfcapd , time= :10:00, profile=tmp) >>> for record in s.search_all([ ]):... print record.time, record.profile :00:00 tmp :05:00 tmp :10:00 tmp These field extractions are done via the pathregex configuration option. 4.4 Searching with pynfdump pynfdump 1 is another module I have written. You can easily use netflow indexer with pynfdump: >>> from netflowindexer import Searcher >>> import pynfdump >>> d=pynfdump.dumper() >>> s = Searcher("/spare/tmp/netflow/nfdump.ini") >>> records = s.search_all([" "], dump=true, filter= dst port 53, mode= pipe ) >>> for rec in d.parse_search(records):... print rec[ dstip ], rec[ bytes ] File metadata 8

11 netflow-indexer Documentation, Release The above example used netflowindexer to find all flows to , then used nfdump to filter it by dst port 53, and finally handed it off to pynfdump for parsing Searching with pynfdump 9

12 CHAPTER FIVE INDICES AND TABLES genindex search 10

13 INDEX L list_databases() (netflowindexer.main.searcher method), 7 S search() (netflowindexer.main.searcher method), 7 search_all() (netflowindexer.main.searcher method), 7 Searcher (class in netflowindexer.main), 7 11