NETWORK SIMULATORS: A DEVELOPER S PERSPECTIVE

Transcription

1 NETWORK SIMULATORS: A DEVELOPER S PERSPECTIVE Marek Małowidzki Military Communication Institute 5-13 Zegrze, Poland [email protected] Keywords: network simulators, ns-, J-Sim, OPNET, FTP Abstract In the paper, we compare simulation modes and programming interfaces of three popular network simulators: ns-, J-Sim and OPNET. Our observations and conclusions are based on a realworld FTP application case study, and also on our earlier projects. I. INTRODUCTION Network simulation used to be an important tool for researchers and engineers that allows to analyze network s behavior and performance. Network simulator is usually the first place to verify a new idea (a protocol setting, a queuing scheme, etc.). The reasons are various, from lack of hardware (both network devices and appropriate transmission media), through difficulties to modify the equipment (which is often simply impossible), to other problems such as complex configuration of real-world equipment, dependable statistics collection and the requirement to separate simulated process from other activities. Network simulators are software tools that model networks using programming language level objects (since most currently used simulators use object-oriented approach, we could equally well omit the quotation marks; to be more precise, however, and keeping OPNET in mind, we leave them and comment that objects could mean data structures and correlated procedures). Most of them are open in the sense that they enable development of new models or modifications of existing ones. Additionally, they usually allow to modify the simulation process. Sometimes, these capabilities are limited to a predefined simulation scenario, but more often a user can affect the simulation execution programmatically. In the paper, we would like to describe and compare programming interfaces of three popular network simulators, ns- [1], OPNET [] and J-Sim (formerly known as JavaSim) []. We comment on both low-level interfaces, which are used for component development, and high-level interfaces, which usually provide a convenient means to combine the components in a simulation setup and then affect the simulation scenario. We draw our conclusions from an FTP case study, presented in the paper, and from our previous simulation projects in OPNET [] and ns- [7] environments. We believe that the paper could be an interesting lecture for readers who are buried in one tool and do not have the time to try another one. Additionally, we are not aware of many papers that take a similar topic, which is probably caused by the complexity of simulation tools accompanied by different modeling architectures and software technologies employed by these tools. The paper is organized as follows: First, we review related work. Then, we describe the three simulators, used in our comparison, and briefly mention other simulation tools. In the following section, we present a case study, which involves an implementation of a real-world FTP service in all the three simulators, and at the same time we have a closer look at simulation modes and programming interfaces that each of the simulators implements. Finally, we comment on simulation results and conclude with final remarks. II. RELATED WORK A detailed comparison of OPNET and OMNeT++ [13] may be found in earlier versions of OMNeT++ User Manual. Newer versions of the manual do not contain this section (although, some remarks on OPNET are still present). The authors of J-Sim also provide a short comparison [1] of their simulator to ns- and SSFNet-Java and find J-Sim to be most scalable (fastest for large-scale scenarios). D. Nicol provides a rebuttal in [1], showing that ns- is the fastest simulator in large-scale performance tests, closely followed by SSFNet-C++, with J-Sim being significantly slower. [] introduces a concept of a simulator utility, which collects various features in a single number, and finds SSFNet-C++ to exhibit the highest utility, with ns- being the runner-up. [3] discusses scalability of simulators written in Java. [5] contains a comparison of mobile ad-hoc network simulations in ns-, OPNET and GloMoSim [1] (something we would call a model comparison). Unfortunately, the results from all three simulators differed significantly (both values and behavior were different). As real equipment was not used for reference, one could not say which simulator was right. Similarly, [] compares output from ns- and OPNET with traces collected from a network testbed for FTP and CBR traffic and finds the simulators to be quite accurate. III. SIMULATORS OVERVIEW In this section, we shortly describe the three network simulators that we focus on: ns-, J-Sim and OPNET. ns- has been chosen due to its popularity in the research community. OPNET is a reference network simulator due to its high-fidelity models; from technical point of view, however, it is somewhat obsolete, as it uses C as its main programming language. Finally, J-Sim is the youngest one from these three, and is implemented in Java, with parallel, multithreaded execution in mind. In the next three sections, we present an overview of these simulators. ns- The ns- Network Simulator [1], [] has been developed under the VINT (Virtual InterNetwork Testbed) project, which is a joint effort by people from University of California at Berkeley, University of Southern California's Information Sciences Institute (USC/ISI), Lawrence Berkeley National Laboratory (LBNL) and Xerox Palo Alto Research Center (PARC). The main sponsors are the Defense Advanced Research Projects Agency (DARPA) and the National Science Foundation (NSF). ns- seems to be completely free for both educational and commercial purposes (although some older code, still in use, explicitly grants rights to educational type of use only, so the

2 actual status is a bit unclear for commercial projects). The simulator is available with a full source code, validation tests, a rich set of examples and a good manual. Moreover, additional support may be provided by the ns- users mailing list. ns- has been used in a great number of research projects. Our observations suggest that ns- is the most often used simulator in research projects related to IP networks. The main reasons behind this popularity are the fact that it is free and easy to use and that many well-recognized scientists have contributed significant amount of work to this project. Thus, probably most modern features related to IP networks are implemented in ns- and even if not merged with the distribution, they could be found somewhere in the Internet. The ns- version depicted in this work is.7 (the newest one as of the time of writing). The simulator s roots refer to the REAL network simulator. ns- is the second version of the network simulator and differs in a number of details from the earlier version. The ns- simulator is easy to use since it employs a brilliant idea of using two languages for programming purposes. Low-level simulation mechanisms are implemented in C++ while higher-level activities (configuration setup, parameters setting, steering of simulation process) could be easily achieved with a simple, scripting language, which is OTcl (Objective Tool Command Language, [1]). We have found this combination extremely useful: Even quite complex actions could be quickly achieved in OTcl. ns- maintains two object hierarchies, a compiled one, grouping C++ objects, and an interpreted one, consisting of OTcl objects. Important simulation compiled objects are mirrored in the interpreted hierarchy, with instance members closely synchronized (so that changes in one hierarchy are instantly visible in the other one). Moreover, method calls work across different hierarchies with a good level of transparency. In most, even quite advanced, cases it is sufficient to develop OTcl code (e.g., high-level simulation models may be implemented in OTcl); low-level packet processing (e.g., queue models in routers) usually requires C++ development, though. One more thing to be emphasized: The use of OTcl is important not only from the point of view of user-friendliness. It also greatly simplifies simulation validation. It seems that ns- has been designed with this issue in mind: It should be possible for other users to verify someone else s simulation results. In most cases, it is sufficient for ns- to publish OTcl scripts used in simulation to allow other users verify if all required circumstances were met to regard the results as reliable. Moreover, the validation tests suite, which should be run for every release after compilation, ensures that the simulator is valid and ready to work. As it was already stated above, many researchers contribute to ns-. Thus, the simulator is quite recent, what means that it is possible to find a reliable implementation for even quite new protocol options, queuing disciplines, etc. While network equipment models are quite simple (e.g., a router is just a node with at least two network interfaces), ns- puts stress on dependable simulation of network protocols and queuing models. The simulator allows for single-threaded, synchronous simulation. Although a parallel and distributed version is available [3], the main distribution is not multithreaded-safe (MT-safe). Fortunately, we have found ns- to be quite fast; this may be the result of simplified network equipment models. ns- s standard GUI, called nam, is quite simple; it is intended to visualize simulation results (it shows packet flows, queue lengths, packet drops, etc.). Thus, ns- is a batch-type simulator, based on scripts that start and control the simulation and finally convert interesting data traces into appropriate format. At least, this is the mode we have found most convenient. The lack of advanced GUI is a matter of choice rather than a disadvantage. OPNET OPNET [], [5] is the name of a set of simulation tools based on the same simulation engine, developed and shipped by OPNET Technologies, Inc. The flagship product is, from our point of view, OPNET Modeler, which is addressed for programmers rather than end-users and enables the development of building blocks, used in simulations (other OPNET family products are, for example, OPNET IT Guru and OPNET SP Guru). OPNET had been originally developed at the Massachusetts Institute of Technology (MIT) and since 197 has become commercial software. OPNET is a commercial product and its availability is strictly limited. The simulator is available with full source code for all simulation modules (although the code for the simulation engine is not available as is not the source code for some restricted protocols), examples and exhaustive documentation. Additional support, available through OPNET mailing list, requires additional maintenance license. Simply speaking, there is nothing free about OPNET there are a number of additional OPNET packages that require separate purchase. The Modeler version described in the paper is 1.5. The main programming language in OPNET is C (recent releases support C++ development). The initial configuration (topology setup, parameter setting) is usually achieved using Graphical User Interface (GUI), a set of XML files or through C library calls. Simulation scenarios (e.g., parameter change after some time, topology update, etc.) usually require writing C or C++ code; although in simpler cases one can use a special Failure Recovery object, which allows to specify failure/recovery times for links and nodes. We have found OPNET modules implementations to be complex and fragile in contrast to ns- s relatively well designed C++ objects, OPNET s implementations are monolithic and use a number of global variables (since some compilation tricks are employed, the term global may be not strictly correct here, but this is how programmers perceive it). Modifications of OPNET behavior are difficult, error-prone and require significant effort and amount of time. A distinguished OPNET feature is the detailed models of network equipment. (In fact, one could say that everything in OPNET happens in a way that closely follows real network behavior.) To some degree (strictly speaking, above the data layer), packet processing in OPNET closely mirrors the way real-world network equipment works. For example, a router in OPNET first receives a packet, then uncompresses it (if necessary), selects the appropriate outgoing interface (if the packet is not addressed to the router), transfers the packet to the outgoing interface s buffer through router s main memory, fragments the packet if necessary, etc. Packet (de)compression and processing times are taken into account. The simulator is shipped with a library of network equipment, which contains most popular box types. Usually, the same type of equipment differs in configuration and some parameters settings. For example, the underlying model for an IP

3 router is the same in all cases but usually various router types are equipped with different interface cards and additionally are attributed different packet processing speed values. OPNET s simulation is based on a discrete event queue. Before release 1., most of OPNET simulation was synchronous and single-threaded. A small portion of code (mainly, radio propagation models) could have been parallelized on multiple processors (although every subsequent processor requires an additional parallel license). Release 1. has brought a breakthrough, as currently OPNET enables multithreaded, parallel execution on multi-cpu machines (as usually, every processor requires a separate license). OPNET supports a parameter, called parallel execution window, which defines a time interval within which parallel execution of events from different module instances is allowed [] (a module s state machine is not reentrant; thus, event execution must be serialized at the module instance level). Usually, parallel simulation in OPNET will not be deterministic, as it depends on operating system s thread scheduling rules. Since OPNET s network equipment models are very detailed, and in fact the simulation process closely reflects the processing that happens in real-world equipment, OPNET s simulation is relatively slow and requires powerful workstations. OPNET is equipped with an advanced GUI interface. Although the simulator may be launched from a command line, working with GUI is the default mode. The lack of scripting language is limiting (from a developer s point of view). On the other hand, the GUI models network elements and protocols in a very nice way; this is really helpful for a user in understanding how the things work. J-SIM J-Sim (formerly called JavaSim, the former name conflicted with Sun s Java trademark) has been developed by a team at the Distributed Realtime Computing Laboratory (DRCL) of the Ohio State University. Additional third-party packages are also available. The project has been sponsored by the National Science Foundation (NSF), DARPA s Information Technology Office (DARPA/ITO, through the Quorum global distributed computing program and the Network Modeling and Simulation program), Air Force Office of Scientific Research s Multidisciplinary University Research Initiative (AFORS MURI), the Ohio State University and the University of Illinois at Urbana-Champaign. J-Sim is free and available with source code, examples, tutorials and white papers. Additionally, there is a publicly available mailing list for J-Sim users. The J-Sim version depicted in this paper is 1.3. J-Sim is implemented in Java and uses Tcl binding in the form of Jacl [19]. Java is used to create simulator s objects, called components, while Tcl enables topology setup and provides a limited means of simulation control. Similarly to ns-, J-Sim offers higher-level Tcl-based programming interface. In releases up to 1.3, unlike in ns-, one could not build model extensions in Tcl. We comment on OTcl integration in J-Sim further in the paper. J-Sim is based on a so-called Autonomous Component Architecture (ACA). The components are written in Java and expose some well-defined interfaces. The components communicate with each other through ports, and are bound to contracts. Contracts define component interactions at layer boundaries (e.g., how an FTP application creates TCP sockets); they allow to make use of lower layer s services but at the same time introduce tight coupling between layers (e.g., an FTP socket application must use TCP socket transport component). Components are assembled at runtime by wiring their ports. The components are organized in a tree-like structure (i.e., a parent component is a directory for its children). The Runtime Virtual System (RUV) is a set of Tcl commands, built atop Tcl/Jacl, which resembles file system commands in UNIX, used to manipulate the component tree (e.g., mkdir, cp, mv, rm, cd, pwd, ls). Also, hierarchical components names look similarly to file paths in UNIX (e.g., /net/router/port@group). Two simulation modes are available for J-Sim: the multithreaded, real-time process-based, and the classical, discrete event based. The multithreaded mode. The simulation within J-Sim is by definition multithreaded. The runtime manages a pool of worker threads to provide a new execution context for a component to handle and process incoming data. Because of this mode, and unlike in ns- and earlier OPNET releases, developers should pay special attention to proper synchronization of critical regions in a component s code. Note that due to possible various thread scheduling in different Java Virtual Machines, subsequent runs of the same simulation scenario (with no randomness) may yield different results. The simulation time is advanced proportionally to the wall time when at least one thread is active, otherwise it is advanced to the moment at which at least one thread may be woken (a jump in time is made). Thus, while the simulation is based on events (usually, data reception on a port), time management is completely opposite to discrete event simulation in ns- or OPNET, where events are scheduled at fixed time points (and the simulation time is frozen between two events). However, the larger value of a time scale, the closer the time management in J-Sim to discrete event simulation is (event processing times become negligible). Note that this so-called real-time process-based approach has some severe consequences: Helper actions, not directly related to simulation (e.g., log file formatting, component state validation, etc.) may affect the still elapsing simulation time. Additionally, if a component code needs to add some artificial delay (e.g., a slow router is simulated), then the delay cannot be guaranteed, due to thread scheduling within a JVM (and due to process scheduling within an operating system). For example, a component developer needs to use J-Sim-specific thread synchronization and delay primitives, to enable the kernel to make a jump in time when all threads are suspended for some timeout; otherwise the simulation may be frozen when a thread sleeps. Discrete event mode. In this mode, J-Sim behaves exactly as most other simulators. A single thread processes discrete events, with zero execution time, and the simulation is completely deterministic. Table 1 on page presents a short comparison of all the three simulators. OTHER SIMULATION TOOLS In the paper, we present three simulators of our choice. As one may guess, this is not the whole picture. While the number of general-purpose network simulators and auxiliary simulation tools is limited (due to complexity of modeling real-world protocols in a reliable manner), at least a few more tools should be mentioned.

4 QualNet [1] is a commercial network simulator from Scalable Network Technologies, Inc., written in C. SSFNet [11] is a set of standards for discrete-event simulation of large, complex systems, with implementations in C++ from Darthmouth College [1] and Java from Renesys Corporation. OMNeT++ [13] is a discrete event simulation engine, written in C++, with thirdparty protocol models for the Internet. Cnet [15] is a simple, free, C-based simulator that contains only the physical and application layers. Its interesting feature is that a node can only get information about other nodes through message exchange (and not the global memory unless a programmer intentionally cheats). PacketStorm [1] from PacketStorm Communications, Inc., marketed as the Internet in a box, is a network emulator, able to be plugged between real network equipment and emulate a network with configured structure, traffic conditions and errors. Finally, Brite [17] is a universal network topology generator from Boston University, able to generate complex network topologies that preserve crucial features of the Internet (C++ and Java implementations are available). An interesting observation is that some of these tools use Tcl (or its GUI library, Tk), for various purposes. These tools include OMNeT++ (Tk-based GUI) and PacketStorm (Tcl configuration scripts). IV. APPLICATION CASE STUDY: FTP In this case study, we evaluate the complexity of the implementation of the FTP service [] that would closely emulate real FTP. In most simulators, including the three ones depicted in this paper, FTP is just a file transfer over a TCP connection (OPNET s model is more complex, see below). In reality, FTP uses two separate TCP connections: the control connection, used to exchange commands (user name, password, transfer mode, file name, etc.), and the data connection, used to actually transfer the file. FTP example is complex enough not to be trivial and at the same time sufficiently simple for us to be able to implement it in three completely different tools in a reasonable time. Since some simulators offer a convenient scripting high-level interface, which is sufficient for the whole task, we additionally comment on lowerlevel development. ns- As it was already stated, ns- implements OTcl binding, and the high-level interface is rich enough that many applications could be implemented completely in OTcl. In fact, our RealFTP has been completely implemented in OTcl. To supplement the picture, however, we give a brief overview of how the binding works. All important C++ objects have their OTcl counterparts and may be affected from OTcl. For every new C++ class to be mirrored in OTcl, a dedicated class, deriving from TclClass, should be provided [] (which requires just a few lines of code). Since C++ does not have reflection capabilities (e.g., such as in Java), some modest programming effort is necessary to provide transparency. C++ variables must be explicitly bound to be available in OTcl. Providing a means for fully transparent virtual invocations up and down the inheritance chain and across the language boundary would be probably impossible; thus, C++ code is invoked using OTcl object s unknown instance procedure (instproc), which in turn invokes another instproc, cmd; finally, cmd calls a well-known C++ object s command() member function, which should process the operation and return the results to the interpreter (or delegate the task to its base class command() implementation). Providing an OTcl instproc with the same name as the C++ member function effectively hides the C++ implementation and the next mechanism (which calls the next procedure up the inheritance chain [1]) does not work any more. Calling an OTcl instproc from C++ requires an explicit call to interpreter; callbacks from C++ to OTcl are implemented in this way: C++ components invoke OTcl instprocs when an interesting event occurs. Despite the fact that ns- uses somewhat older C++ (without exceptions or templates), component development in ns- does not involve any significant problems in itself. As it was already stated, we have implemented our service in pure OTcl, which was quite simple. Two OTcl classes were implemented, RealFTPClient and RealFTPServer, as subclasses of ns- standard class Application. Both classes follow standard ns- conventions and provide a simple, userfriendly interface with only a few instprocs to configure, start and re-start file transfers. Most functionality is embedded in the client class; the functionality encompasses quite complex re-establishing of TCP connections (due to problems with reusing them in ns-). The server simply responds to control commands. Unlike in the real world, there is a one-to-one relationship between the client and server. A developer using the implementation has only to create objects for the client and server and then override the empty implementation of done instproc in client in order to repeat transfers. The critical functionality, which greatly simplified the implementation, was the send instproc from the standard ns- Application/TcpApp class; send executes the supplied script (an OTcl command) after the file transfer is complete. Finally, it is worth it to mention that the elegance and developer-friendliness of OTcl comes at some price, which includes processing overhead and increased memory consumption. J-Sim J-Sim is implemented in pure Java. The choice of Java has many virtues, which mainly stem from the programming language s features (portability, safety, automatic memory management, etc.). Software development in Java is much simpler when compared to C++ (not to mention pure C). However, in practice, simulation models must be rewritten from scratch while C/C++-based tools can borrow code from each other. (Regarding the hot C++ vs. Java performance debate, we will stay aside.) Components are implemented as Java classes that inherit from a set of J-Sim s standard classes. Components that implement the special ActiveComponent interface become active components and are given a dedicated thread. (This is why multithreaded mode has been implemented; multiple threads on a single CPU machine only slow the simulation, because simulation threads are in most typical situations always ready to run.) Passive components are, in contrast, event-triggered. Active components provide a more natural programming paradigm, as they imitate a natural execution of a program (and are most convenient for writing applicationlevel components). Of course, the number of active components should be limited; otherwise, simulation performance could suffer significantly. Additionally, the number of threads in use should exceed the number of active components that may be simultaneously running (or, a deadlock can occur). Implementations must take into account the multithreaded environment and apply locks, where necessary.

5 Due to Java s reflection capabilities, all public fields and methods are automatically available from Tcl through the Jacl binding, and no action is required from the developer. As in ns-, the Tcl binding adds performance overhead in terms of execution time and memory usage. In J-Sim releases 1.3 and older, Tcl was mainly used for simulation setup and control (through actions fired at specified time moments); due to low scalability of procedural language and lack of standard hooks back into Tcl, the development of more complex simulation logic in Tcl was difficult. During the project, OTcl binding was integrated with J-Sim, which now supports callbacks. Unlike in ns-, object hierarchies in OTcl and Java are completely separated. Additionally, access to the OTcl interpreter must be locked, as the Jacl implementation is currently not multithread-safe. We have ported our implementation of RealFTP to J-Sim. We modified J-Sim s standard ftp/ftpd components, which are designed to actually send files; the modified versions send arbitrary data of a specified size. When a file transfer is complete, ftpd sends a notification to a special notify@ port, at which a callback component of a J-Sim s TclClass is connected. On the notification reception, the callback component executes a configured OTcl script in a synchronized context. It is difficult to honestly estimate the programming effort but it seems that after the OTcl integration has been finished, development effort should be comparable to ns-. We would like to note that the multithreaded environment required much care to avoid deadlocks, which is true for both Java and OTcl code. OPNET As a commercial application, OPNET comes with ready models for most typical applications (FTP, HTTP, , database traffic, voice on-off traffic, etc.), which may be configured through a set of pre-defined attributes (e.g., for FTP, major parameters involve downloads ratio, inter-request time and file size). These models, as in other simulators, are simplified, although this time FTP is not just a TCP transfer, since a request-response pair is exchanged (which emulates a real FTP with a single control communication). However, OPNET also provides a configurable Custom Application model, which may be used to model even complex application behavior, e.g., a more complex communication scenario, involving message exchange between a few applications. We have used the Custom Application to define a task that simulated quite closely the behavior of a real-world FTP application, exchanging control commands and finally transferring a file (thus, the task was composed of five phases). The process was quite straightforward and involved a few GUI forms that needed to be used for the configuration. Another, a more complex approach would involve a C/C++ implementation with the existing FTP modules (the client FTP manager, gna_ftp_mgr, the FTP client, gna_ftp_cli, and the server front-end manager for all application types, gna_clsvr_mgr) taken as a starting point (OPNET s modules are written in C but custom C++ code is supported). A simple solution would be to modify the manager so that it used a separate FTP transfer for each control command (and, finally, for file transfer). A fuller implementation would have to modify both the client and the server; due to fact that the implementations are quite low-level (e.g., they need to handle and analyze packets, perform packet reassembly, etc.) and complex (especially the server side), we think it would be relatively difficult and time-consuming. Finally, we had to modify the profile manager, gna_profile_mgr, to finish simulation when all applications have finished. What we found severely limiting in OPNET is the lack of good simulation automation features. As it was already mentioned, network topology in OPNET is usually created from GUI, which is inconvenient for larger topologies, even with object copying (1 FTP pairs!). It is possible to build the topology programmatically, through OPNET libraries, but it would involve significant programming effort. Finally, we took the following approach: We saved a small configuration as an XML file, then wrote a dedicated script to augment this file to a topology of a required size ( = 1,,, 1), and imported manually resulting XML scripts back to OPNET, to produce binary project files (importing XML from command line is currently not possible). We used another pair of scripts to prepare simulation image (using OPNET s op_mksim utility) and run simulations (using op_runsim); the parameter was modified using OPNET s environment files. A similar approach to automate OPNET simulations, using Excel and custom C programs, is described in [7], although the topology does not change. Fig. 1: The topology in OPNET ( = ) Unlike in the two other simulators, equipment models in OPNET closely follow real appliances. For example, to connect 1 hosts to a router using LAN, it is necessary to employ hubs, and since the hubs have limited number of ports, a few hubs must be used. The hubs should be attached to switches (chaining is currently not supported). Similarly, links cannot have arbitrary bandwidth values: One has to choose one of standard link types (e.g., DS1, E1, etc.). This may look unnecessary but, in our opinion, this is a very nice feature, which allows to realize how the actual network looks. Fig. 1 shows the resulting topology in the OPNET simulator. Note finally that OPNET has aggregated LAN models, able to simulate a given number of workstations, but we did not want to use it to make the comparison more accurate.

6 Name/Version OPNET Modeler 1..A ns-.7 J-Sim (formerly JavaSim) 1.3 Availability Support Topology/ Scenario Extensions (components) Simulation mode Brief summary (with subjective assessment) Highly expensive, commercial software (no publicly available trial). Available source code for simulation modules (except restricted protocols). excellent manual mailing list (maintenance license required) source code and examples Open-source software, available with full source code, validation tests and examples. good manual publicly available mailing list source code and examples Open-source software, available with full source code and examples good manual publicly available mailing list source code and examples GUI, XML, imports (e.g., HP OV) OTcl scripts (or C++) Tcl scripts (or Java) (as of 1.3) Failure Recovery object and its parameters OTcl or Java (future releases) C/C++ C/C++ synchronous, single-threaded, discrete event queue based, with zero event processing time, fully deterministic multithreaded, discrete event queue based, with zero event processing time distributed simulation: HLA (High-Level Arch.) quite slow, heavyweight expensive commercial software ready, high-fidelity equipment and protocols models; a reference simulator unique (e.g., military) features; widely used in NATO projects Table 1: OPNET, ns- and J-Sim features OTcl (higher level) C++ (lower level) synchronous, single-threaded, discrete event queue based, with zero event processing time, fully deterministic parallel/distributed version available (Parallel /Distributed NS, PDNS) fast, quite modern, free OTcl binding simplified equipment models many recent TCP mechanisms implemented for ns- currently most popular in research projects Java (as of 1.3) also OTcl for higher level (future releases) synchronous, single-threaded, with zero event processing time, fully deterministic multithreaded, real-time process-based, with event processing times taken into account, non-deterministic scalable, modern, free Tcl/Jacl binding (OTcl/Jacl) simplified equipment models new simulation paradigm (active components) V. SIMULATION RESULTS In this section, we present the simulation results obtained for all three simulators. We used a simple, dumbbell topology, shown in Fig., with FTP client-server pairs repeating times FTP sessions that involved the exchange of four control commands (USER, PASS, TYPE, RETR) in a synchronous manner, followed by a 1MByte file transfer (with no delays involved). The link capacity grew linearly from 5kb/s (5b/s, = 1) to Mb/s (b/s, = 1) except that in OPNET we always selected a standard link type with a bandwidth value no smaller than desired (PPP_DS1 or PPP_E1). During the tests, we measured the simulated time, the simulation time (how long the simulation run) and the average goodput (number of file bytes received by the client, which did not include control commands). Traces and statistics were not collected. ftpc ftpcn 1Mb/s, 1ms delay DropTail gw1 5kb/s - Mb/s 1ms delay RED gw linterm: pkt; maxthresh: pkt; max_p: 1.; q_weight:. Fig. : The topology and settings 1Mb/s, 1ms delay DropTail ftps ftpsn We would like to emphasize that the results should not be viewed as a direct performance comparison, despite various detailed figures that we present for a reader s convenience. First, OPNET data were collected on a different platform and the topology was different. Second, we focused on interfaces and default simulation modes rather than performance for example, the multithreaded mode in J-Sim involves significantly higher cost than the discrete event one, as it requires multiple threads to be created and running. Thus, it is obvious that we cannot use thousands of ActiveComponents, as the number of simultaneously running threads could be too high. Finally, some implementation details differed as well. For example, the TCP implementation in J-Sim used in this project has both connection establishment and tear-off phases omitted (J-Sim has a better TCP model, but it is more difficult to use an associated application model from OTcl). The results for ns- and J-Sim are presented in Fig. 3 and Fig., respectively. ns- is very fast for smaller problems but the simulation time grows faster than simulated time (grows faster than linearly) and exhibits some fluctuations. As the test is quite static (i.e., it includes the same actions repeated multiple time, with no artificial randomness), a possible explanation could be memory fragmentation. The simulation setup time (the case) grows from.5 s. ( = 1) to 13.5 s. ( = 1). J-Sim is considerably slower for smaller tasks but also more scalable for larger ones (J-Sim performs both faster and more correct for larger problems) as the simulation time grows linearly in the simulated time function. The setup time grows from 9 ( = 1) to 1 s. ( = 1). From our other measurements we could observe that active components really add significant overhead (at least 5% for

7 small scenarios), so it is difficult to decide how J-Sim s discrete event mode would perform with respect to ns-. The average goodput exhibits an interesting problem: J- Sim usually needed a noticeably longer simulated time to complete the simulation (from about 5% to %), which caused goodput to be significantly lower for J-Sim (usually, 5%-% for greater values, see Fig. and Fig. 5). A possible explanation is as follows: The necessity to synchronize access to the OTcl interpreter limits concurrency; thus, when an OTcl code starts activity in Java, the activity occurs on a parallel thread and may be started immediately, while other OTcl callbacks are blocked and waiting for the interpreter. This may cause the bottleneck link to become underutilized for some periods. In fact, during the test we observed that various flows finish their activities with increased delays to the fastest one. A possible workaround would be to enable multithreaded OTcl execution (which may be technically infeasible), use the single threaded simulation mode or affect thread scheduling (e.g., to suspend the simulation engine while an OTcl callback is executed). As one could see, a multithreaded environment adds significant complexity to the problem, as the simulation results depend on how the threads are scheduled. Finally, it would be interesting to repeat these tests on a multi-cpu machine to check how much J-Sim (and OPNET) could gain from parallel execution. The results for OPNET are presented in Fig. 7. The results are incomplete as OPNET hang, for unknown reasons, for greater values of and ; thus, we do not draw any conclusions. VI. CONCLUSIONS Simulation modes and programming interfaces are probably not the most important criteria that determine a network simulator choice. The choice may depend on the nature of a problem we would like solve through simulation experiments. For fine-grained simulations that study protocol details, the availability of highquality models is usually most important; large-scale simulations require special procedures and high performance, while details are not that significant. In most typical problems, however, programmer friendliness is an important factor. Well designed and documented interfaces allow to hide much of underlying complexity and get the desired behavior in a reasonable time. We think that scripting front-ends that many simulator provide are greatly useful, especially if object-oriented programming is possible. This simplifies the development of application-level simulation logic, keeping performance overhead at a reasonable level, and provides a convenient means for simulation validation. Add the same time, however, adding support for a scripting language involves some technical challenges in a multithreaded simulator. Another common trend is the migration of simulation tools towards multithreaded, parallel execution on multiprocessor machines. Unlike distributed simulation, which has to face communication overhead, multiprocessor machines allow to overcome the processing limitations of a single processor. In this paper, we review three popular network simulators: ns-, J-Sim and OPNET. Every one of them represents a different epoch in network simulation. We describe their main features and present a case study that allows to estimate implementation complexity and also comment on simulation performance (although we are far from detailed comparison). All scripts and component extensions used in this project are available from []. ACKNOWLEDGMENT The author would like to acknowledge the financial support provided by the Foundation for Polish Science (FNP). We would also like to thank Hung-ying Tyan, the main author of J-Sim, for his help during the project. REFERENCES [1] The Network Simulator - ns- [] The ns Manual, available at [3] PDNS - Parallel/Distributed NS [] OPNET Modeler Home Page: [5] Opnet Technologies, OPNET Modeler Documentation, v. 1.. [] Opnet Technologies, Updating OPNET Model Code for Parallel Simulation, Opnet While Paper, Product Release 1. [7] D. McGuire, K. Grzemski, S. Bliesner, B. Deng, L. Shatos, S. Million, Eficiently Managing a Large Number of Simulation Runs, OPNETWORK 3, Washington, USA [] J-Sim Home Page: [9] Evaluation of J-Sim: [1] The QualNet Simulator Home Page: [11] Scalable Simulation Framework Home Page: [1] Darthmouth SSF (DaSSF) Home Page: [13] OMNeT++: Discrete Event Simulation System: [1] About GloMoSim [15] The cnet network simulator, [1] PacketStorm Communications, Inc., [17] BRITE: Boston University Representative Internet Topology Generator, [1] OTcl Tutorial (Version.9, September 95) [19] Jacl, Java Command Language, available at [] W. Stevens, TCP/IP Illustrated, Vol. I, Addison-Wesley 199. [1] D. M. Nicol, Scalablity of Network Simulators Revisited, SCS SNDS, 3, Orlando, USA [] D. M. Nicol, Utility Analysis of Network Simulators, International Journal of Simulation: Systems, Science, and Technology, 3 [3] D. M. Nicol, Scalability of Garbage Collection in Java-based Discrete-Event Simulators, UKSim 3 [] G. F. Lucio, M. Paredes-Farrera, E. Jammeh, M. Fleury, M. J. Reed, OPNET Modeler and Ns-: Comparing the Accuracy Of Network Simulators for Packet-Level Analysis using a Network Testbed, ICOSMO 3 [5] D. Cavin, Y. Sasson, A. Schiper, On the Accuracy of MANET Simulators, POMC', October. [] M. Małowidzki, K. Malinowski, Optimization Flow Control in IP Networks: Distributed Pricing Algorithms and Reality Oriented Simulation, SCS SPECTS, San Diego, USA [7] M. Małowidzki, Simulation-Based Study of ECN Performance in RED Networks, SCS SPECTS 3, Montreal, Canada [] Project Home Page:

8 'results-ns\elapsedtime.3d' 'results-ns\simulatedtime.3d' 'results-ns\goodput.3d' Simulated time [s] Fig. 3: ns- results: simulation time, simulated time and goodput (Intel Celeron GHz, 5MB RAM, Linux 7., swapping disabled) 'results-jsim\elapsedtime.3d' 'results-jsim\simulatedtime.3d' 'results-jsim\goodput.3d' Simulated time [s] Fig. : J-Sim results: simulation time, simulated time and goodput (Intel Celeron GHz, 5MB RAM, Linux 7., swapping disabled) 1 'results-ns-1\elapsedtime.out' 'results-ns-1\simulatedtime.out' 'results-ns-1\goodput.out' 'results-jsim-1\elapsedtime.out' 1 'results-jsim-1\simulatedtime.out' 5 'results-jsim-1\goodput.out' Simulated time [s] Fig. 5: ns- vs. J-Sim: simulation time, simulated time and goodput for = 1 (link bandwidth set to Mb/s) 35 'results-ns-1s\elapsedtime.out' 'results-ns-1s\simulatedtime.out' 'results-ns-1s\goodput.out' 'results-jsim-1s\elapsedtime.out' 'results-jsim-1s\simulatedtime.out' 5 'results-jsim-1s\goodput.out' Simulated time [s] Fig. : ns- vs. J-Sim: simulation time, simulated time and goodput for = 1 (link bandwidth grows linearly from 5kb/s to Mb/s) Simulation time for = 1 Simulation time for = Goodput for = 'timesrep1.txt' 5 'times.txt' 1 'Goodput.txt' Fig. 7: OPNET's results (Intel Pentium IV HT 3,GHz, 1GB RAM, Windows XP Professional, kernel optimized, sequential; link: PPP_DS1 (for < 1) or PPP_E1 (for = 1))