Transcription

1 Offline Flow Analysis Tool (OFAT) Version 2 Documentation, March 9, 2010 OFAT.pbs Year, month, day, router name, UVA_gap, UVA_min_flowlength, UVA_long, UVA_short, code directory, output directory yyyy mm dd rrrr sss sss ss s /###/ /###/ OFAT.sh Undecided_Flowlength.txt All_Long_Flow.txt Ftp.txt Ssh.txt Smtp.txt Http.txt Https.txt Nntp.txt Imap.txt Unidata.txt Rsync.txt Rtsp.txt Unassigned.txt Dynamic_private.txt MFDB_Long_Only.txt: contains flow identiers (subset of 5 tuple: srcaddr dstaddr srcport dstport prot) for which only long flows (no short flows) were seen MFDB_Long_Short.txt: contains flow identifiers for flows that occurred as both long flows and short flows rsync Flow-export FindLongFlow.R Concatenate.R Protocol.R Port.R LongShortMatch1.R LongShortMatch2.R AddFlowlength.R statistics.txt which contains the following numbers: statistics NumFlow NumLongFlow Bytes ByteslongFlow protocol statistics NumTcp NumIp NumAh NumEsp NumGre NumUdp port statistics NumFtp NumSsh NumSmtp NumHttp NumHttps NumNntp NumImap NumUnidata NumRsync NumRtsp NumUnassigned NumDynamic_private MFDB_Long_Only_Flowlength.txt MFDB_Long_Short_Flowlength.txt srcaddr dstaddr srcport dstport prot flowlength code parameters output file intermediate result

2 netflow.ascii FindLongFlow.R Entries1.txt Bytes1.txt LongFlow.txt FindlongFlow.R reads each 5 minute netflow file, calculates the flowlength of each flow entry, and extracts those flow entries whose flowlength is longer than or equal to UVA_long Entries1.txt LongFlow.txt sort.temp_06.txt Bytes1.txt Undecided_Flowlength.txt Concatenate.R sort.temp_06.txt Entries1.txt Bytes1.txt sort.temp_06.txt All_Long_Flow.txt statistics.txt Undecided_Flowlength.txt

3 Concatenate.R concatenates the long flows got from FindLongFlow.R using UVA_gap, if one long flow entry's firstunix minus another long flow's lastunix is less than or equal to UVA_gap, these two long flow entries are concatenated as one flow All_Long_Flow.txt Protocol.R statistics.txt Protocol.R reads the long flows generated by Concatenate.R, and seperates these long flows based on Protocol

4 All_Long_Flow.txt MFDB_Long_Only.txt Port.R statistics.txt MFDB_Long_Only.txt Ftp.txt Ssh.txt Smtp.txt Http.txt Https.txt Nntp.txt Imap.txt Unidata.txt Rsync.txt Rtsp.txt Unassigned.txt Dynamic_private.txt Port.R reads the long flows generated by Concatenate.R, and seperates these long flows based on Port number. MFDB_Long_Only.txt is also generated based on Port number

5 netflow.ascii Ftp.txt Ssh.txt Smtp.txt Http.txt Https.txt Nntp.txt Imap.txt Unidata.txt Rsync.txt Rtsp.txt Unassigned.txt Dynamic_private.txt LongShortMatch1.R LongFlow.txt sort.temp_06.txt All_Long_Flow.txt statistics.txt Ftp_1.txt Ssh_1.txt Smtp_1.txt Http _1.txt Https_1.txt Nntp_1.txt Imap_1.txt Unidata_1.txt Rsync_1.txt Rtsp_1.txt Unassigned_1.txt Dynamic_private_1.txt Ftp_2.txt Ssh_2.txt Smtp_2.txt Http _2.txt Https_2.txt Nntp_2.txt Imap_2.txt Unidata_2.txt Rsync_2.txt Rtsp_2.txt

6 LongShortMatch1.R finds the long flows that have matches in the short flows(generated using UVA_short), e.g. Ftp_1.txt contains the matches based on source port number, Ftp_2.txt contains the matches based on destination port number, For Unassigned and Dynamic_private, only source IP address and destination IP address are needed Ftp_1.txt Ssh_1.txt Smtp_1.txt Http _1.txt Https_1.txt Nntp_1.txt Imap_1.txt Unidata_1.txt Rsync_1.txt Rtsp_1.txt Unassigned_1.txt Dynamic_private_1.txt Ftp.txt Ssh.txt Smtp.txt Http.txt Https.txt Nntp.txt Imap.txt Unidata.txt Rsync.txt Rtsp.txt Unassigned.txt Dynamic_private.txt MFDB_Long_Short.txt LongShortMatch2.R MFDB_Long_Short.txt Ftp_1.txt Ssh_1.txt Smtp_1.txt Http _1.txt Https_1.txt Nntp_1.txt Imap_1.txt Unidata_1.txt Rsync_1.txt Rtsp_1.txt Unassigned_1.txt Dynamic_private_1.txt Ftp_2.txt Ssh_2.txt Smtp_2.txt Http _2.txt Https_2.txt Nntp_2.txt Imap_2.txt Unidata_2.txt Rsync_2.txt Rtsp_2.txt

7 LongShortMatch2.R finds the long flows that do not have matches in short flows from Concatenate.R All_Long_Flow.txt MFDB_Long_Only.txt MFDB_Long_Short.txt AddFlowlength.R MFDB_Long_Only_Flowlength.txt MFDB_Long_Short_Flowlength.txt AddFlowlength.R finds the longest flowlength of each 5 tuple and adds flowlength to MFDB_Long_Only.txt and MFDB_Long_Short.txt code input file output file deleted file

8 Note: The Internet2 data after exported has 24 columns: "unix_secs","unix_nsecs","sysuptime","exaddr","dpkts","doctets","first","last","engine_type", "engine_id","srcaddr","dstaddr","nexthop","input","output","srcport","dstport","prot","tos", "tcp_flags","src_mask","dst_mask","src_as","dst_as" Please make sure the ESnet data has the same number of columns with same positions as above since the R programs use this as the basis for parsing the flow-export ASCII output file. Modifications in OFAT.pbs necessary before execution: If a batch process is submitted to a Linux cluster via PBS, then use the PBS file, and submit using qsub OFAT.pbs ; Otherwise simply submit the OFAT.sh line with the required arguments. Before using this file, modify (a) address to which notification of the start and end of the job is sent by the job scheduler. The current address in the file is zy4d@virginia.edu. Change this to be your own. (b) Parameters There are 10 input parameters: Year, month, day, router name, UVA_gap, UVA_min_flowlength, UVA_long, UVA_short, code directory, output directory; The format is: yyyy mm dd rrrr sss sss ss s /###/ /###/, divided by space. e.g LOSA /home/zy4d/ofat /net/longtmp/zy4d The year, month, day represent the date for which the 288 Netflow files (one file for every 5- minutes) are copied and then analyzed. The router name is a four digit code used by Internet2. Change this accordingly for ESnet. See Modifications necessary for OFAT.sh for the rsync command used to download the Netflow files from the Internet2 netflow server. UVA_gap represents the accepted gap during flow concatenation by Concatenate.R. For example, if one flow record shows a last parameter of time x, and another flow record shows a first parameter of time first that is apart by less than UVA_gap, these two flow records will be assumed to be for the same flow. See the command that invokes this R program in OFAT.sh (argument 5). Set to 5 minutes in current example. UVA_min_flowlength: only flows longer than this duration are considered to be long flows. This is also used by Concatenate.R. See the command that invokes this R program in OFAT.sh (argument 6). Set to 10 minutes in current example. UVA_long: Internet2 Netflow has set is active-timeout-interval as 60seconds. Therefore, in each 5-minute Netflow file, a long persistent flow may see as many as five flow records, each of length >=59sec. UVA_long is thus typically set to 59, so that all flows reported in each 5-minute Netflow file that have flow lengths >=59 are reported out. Only these flows are concatenated in Concatenate.R. The UVA_long is an argument to FindLongFlow.R. See OFAT.sh which shows that argument 7 is invoked with this program call.

9 UVA_short is set to 5 sec. This is meant to capture all flows that are shorter than 5 sec. This is used by LongShortMatch1.R. A thought was to list only those flow identifiers that occur as long flows and not as short flows. But because a majority of flow identifiers of long flows also occur in short flows, a better idea is to have the packet header processing module of HYNES wait for some duration before initiating the request of a circuit. See in OFAT.sh that argument 8 is called with this program. Parameter 9, a directory, is used to indicate the folder in which all these R programs are located. Parameter 10, also a directory is used to indicate the folder to which the Netflow data files are copied and processed. All R programs are temporarily copied into the data folder to run the programs and then deleted. Before execution, make sure that there is no folder with the router s name under this directory, since as you run the programs, there will be a folder created with the router s name; and within this folder, there will be a folder created with the name the date you specified. Parameters 9 and 10 can be the same. All intermediate data files created by the R programs are also deleted upon completion of runs, including the original Netflow 5-minute files as they consume a significant amount of disk space. If there s no pbs system currently used in your clusters, then after log in to your Linux machine, type the last line in the OFAT.pbs file, which is:./ofat.sh LOSA /home/zy4d/ofat /net/longtmp/zy4d, modified with your directories and parameter settings. Modifications needed in OFAT.sh before execution: rsync related commands as required for ESnet. The command there has username mv@netflow.internet2.edu and requires a rsync.passwd in the directory. Output files: There are 5 output files, MFDB_Long_Only.txt, MFDB_Long_Short.txt, MFDB_Long_Only_Flowlength.txt, MFDB_Long_Short_Flowlength.txt and statistics.txt. MFDB_Long_Only.txt contains the flow identifiers for flows that only occur as long flows, as identified by the concatenation program. MFDB_Long_Short.txt contains flow identifiers for long flows, which also occur as short flows. The file format for both these files is as follows: srcaddr dstaddr srcport dstport prot, if any of the 5 tuples are not needed in the flow identifier, then the number -1 is assigned. MFDB_Long_Only_Flowlength.txt and MFDB_Long_Short_Flowlength.txt contain the flow identifiers plus flowlength. The file format is: "srcaddr","dstaddr","srcport","dstport","prot","flowlength". e.g

10 statistics.txt is the statistics file, please refer to the architecture for the format. All_Long_Flow.txt contains all the long flows after concatenation. Ftp.txt, Ssh.txt, etc, are the long flows after concatenation categorized based on port number. Please refer to the architecture for all the file names. The file format is: "unix_secs","dpkts","doctets","srcaddr","dstaddr","srcport","dstport","prot","firstunix", "lastunix", flowlength. e.g There are also several intermediate result files for temporary pattern analysis that will be deleted later. LongFlow.txt contains the flow entries whose flowlength is longer than or equal to UVA_long. The file format is: "unix_secs","unix_nsecs","dpkts","doctets","first","last","srcaddr","dstaddr","srcport", "dstport","prot","firstunix","lastunix","flowlength". e.g Undecided_Flowlength.txt contains the long flows after concatenation for which the difference between the unixtime at the end of day and the flow s lastunix is less than UVA_gap. The file format is: "unix_secs","dpkts","doctets","srcaddr","dstaddr","srcport","dstport","prot","firstunix", "lastunix","flowlength". e.g Circuit rate will be added later. Future work: Flows that cross midnight. Need to learn from ESNet data whether flows extend across multiple days. In Internet2, the max. length we found after concatenating flows from one 5-day period was 7 hours, and so for this dataset, all that is required is to find flows that cross midnight. But we will need to concatenate across multiple days to see if ESnet has multi-day flows.