Traffic matrix generation using NetFlow traces

This section explains how to use the toolbox to generate traffic matrices from Netflow data. The toolbox includes data from Abilene network to generate an accurate traffic matrix using topology, BGP and NetFlow information. We will first explain the required data formats and needed file/directory structures. We will then explain how to generate an inter-domain traffic matrix from NetFlow traces, and eventually how to produce the corresponding intra-domain traffic matrix.

Required data formats and file/directory structures

One example of BGP information and Netflow traces is available is the archive:
example/abilene/abilene_20050101_bgp_netflow.tar.gz

BGP information

RIB dumps

You will need a snapshot of the entire BGP RIB of each router. Typically, such information is obtained using an additional monitoring machine running Zebra bgpd which participates in the iBGP full-mesh. Resulting format is Zebra binary MRT dump, which can be converted to ASCII machine readable format using route_btoa from the MRTd package (http://www.mrtd.net). We have included the tool in the directory src/perl/bgp/.

Typically, the toolbox module expects the following directory structure. Each BGP RIB (converted in ASCII machine readable format using route_btoa rib_file -m > output) should be in a directory called BGPbaseDirectory/router_id/optionaldumpsubdir or BGPbaseDirectory/router_rid/optionaldumpsubdir where router identification information is found exclusively in the XML topology format. Please note that if a router has several ip addresses, these should be added in the XML topology format. For each node, the rid field should be filled with router main IP address. If the router has other addresses, you must add an interface for each of them (field interface and in particular its subfield ip). For a concrete example, see Abilene topology in examples/abilene/abilene.xml.

A script is provided to automatically convert, compress and optionally rename all compressed BGP RIBs found in a given directory. The script is located in the file src/perl/bgp/rib2ascii.pl.

Cluster file

C-BGP has some scalability problems when too much prefixes are passed to it. That's why we use clustering which allows to group prefixes announced with same BGP parameters, and to advertise only one of them for each group (cluster). The clustering is done by an ad-hoc perl script, we provide a perl script called bgpsum3.pl¹¹ in src/perl/bgp/. Here is the command to execute the script ./bgp-sum3.pl -ribs-dir=directory_holding_ribs > clusterFileName where the -ribs-dir parameter corresponds to a directory which contains the RIB converted to ASCII machine readable format (the files can be gzipped).

NetFlow traces

NetFlow traces for each router are also required. The toolbox expects an aggregated text format extracted from the NetFlow traces which is the following

src_prefix dst_prefix flow_size

Where the flow size is expressed in bytes.

A perl script to generate these aggregated files netflow2prefixes2.pl ¹²is supplied in src/perl/netflow/. First parameter to give to this script is the directory with the BGP ribs (in optionally gzipped ASCII machine readable format), second parameter is the directory with non aggregated traffic, third one is the directory where aggregated traffic should be written, and finally last parameter is the last directory in the path on which you want to work (example: 2005-01-01).

All required libraries can be found in src/perl/netflow/perl.tar.gz. We have also included the necessary flow tools in src/perl/netflow/flowtools.tar.gz. You need to uncompressed both archive to use the script.

All the aggregated files should be placed in directories following the same structure as BGP dumps.

Traffix matrix generation steps

You can find a typical example of source code doing this in
examples/abilene/AbileneExampleTM.java. To use it as is, you can move the file to src/java/be/ac/ulg/montefiore/run/totem/trafficMatrix/. Otherwise, you have also a scenario event for the traffic matrix creation in itself (see last section). The traffic matrix generation capabilities are now also integrated in the graphical interface (see 10.4.2).

Generating domain BGP information from BGP dump

Information about iBGP and eBGP sessions must be added to the XML topology format. eBGP sessions can typically be extracted from BGP dumps (structured as presented above). The class BgpFieldsCreation has two useful methods addiBGPFullMesh(String topologyName, String iBGPTopologyName) which will add an iBGP full mesh to a topology file specified by its name, and produce a topology file with added iBGP full-mesh. The second method addeBGPSessions(String topologyName, String eBGPTopologyName, String BGPbaseDirectory, String optionaldumpsubdir/filename) will create the eBGP sessions.

Note that the graphical interface allows you to directly use those functions.

Creating inter-domain traffic matrix from NetFlow

Starting from the aggregated NetFlow traces, we first generate an inter-domain traffic matrix. The class InterDomainTrafficMatrixGeneration has the method
generateXMLTrafficMatrixfromNetFlow(Domain domain,
String NETFLOWbaseDirectory, String optionalsubdir/filename, String[] suffixes, String interdomainTrafficMatrixFileName). The array suffixes can be used to specify potential suffixes for NetFlow file names.

Generating intra-domain traffic matrix from inter-domain traffic matrix

From this inter-domain traffic matrix, we can generate an intra-domain traffic matrix. The class POPPOPTrafficMatrixGeneration contains the method HashMap readCluster(Domain domain, String clusterFileName, CBGP cbgpinstance, String BGPbaseDirectory, String optionaldumpsubdir/filename) that will load BGP messages corresponding to ``cluster prefixes'' in the CBGP instance and return a hashmap which allows to find to which cluster a prefix belongs, and thus find the route for this prefix. From then on, the method TrafficMatrix generateTrafficMatrix(TrafficMatrix temporaryTM, HashMap clusters, Domain domain, String interdomainTrafficMatrixFileName) will generate the intra-domain traffic matrix. The first argument temporaryTM allows to add a traffic matrix to an existing one, for example to produce a traffic matrix for 20 minutes when NetFlow files are available for 5 minutes.

Scenario events

We have created one scenario event documentated in the scenario part of this manual generateIntraTM. An example of traffic matrix creation on Abilene using scenarios can be found in
examples/abilene/Scenario/generateIntraTM-20050101.xml. To test it, just uncompress the archive abilene_20050101_bgp_netflow.tar.gz found in the same directory, and run the scenario using totem.sh -s command line.