Traffic flow simulation and optimization using evolutionary strategies

Transcription

1 Aalto University School of Science Master s Programme in Machine Learning and Data Mining Alejandro López Vidal Traffic flow simulation and optimization using evolutionary strategies Master s Thesis Espoo, June 7, 2011 Supervisors: Instructor: Doc. Timo Honkela, Ph.D., Chief Research Scientist, Aalto University Professor Yago Sáez Achaerandio, Universidad Carlos III de Madrid Doc. Timo Honkela, Ph.D., Chief Research Scientist

2 Aalto University School of Science Master s Programme in Machine Learning and Data Mining Author: Alejandro López Vidal Title: Traffic flow simulation and optimization using evolutionary strategies ABSTRACT OF MASTER S THESIS Date: June 7, 2011 Pages: xii + 88 Professorship: Information and Computer Science Code: T-61 Supervisors: Instructor: Doc. Timo Honkela, Ph.D., Chief Research Scientist Professor Yago Sáez Achaerandio Doc. Timo Honkela, Ph.D., Chief Research Scientist Different studies, such as the survey that IBM yearly conducts about commuting to work, verify the importance of a well known problem: traffic congestion in big cities. Solving this problem has concerned professionals from many scientific and technological disciplines, including physics and artificial intelligence (AI). AI researchers have contributed to mitigate the problem by adapting their techniques to control traffic, obtaining remarkable results. An important tool in AI is the use of Artificial Neural Networks (ANN), whose numerous features make them a suitable technique in a wide range of problems. However, we found that the use of ANN to control city lights has never been fully seized by any of the past researches, in our opinion, as a consequence of the learning process adopted in their approaches, that was not adequate due to the nature of the problem. This thesis studies the effect of different neuroevolutionary methods in adapting ANNs to efficiently control traffic semaphores. These methods include biological, cultural and linguistic evolution. Furthermore, the performance of this methods is compared with previous approaches using a microscopic traffic simulator, which was enlarged in order to include different realistic scenarios in a square shaped city. The study has been implemented using a combination of Java language, Netlogo social simulation environment and Matlab. The results of this work illustrate the potential of the adaptation of neuroevolutionary concepts to control systems, which opens the door to further research in the topic and possible expansions to other research areas that includes control systems, such as decision support systems in air traffic control or harbor control. Keywords: Language: Traffic flow, Traffic light, Social simulation, Evolutionary algorithm, Neuroevolution, Biological Evolution, Cultural Evolution, Language emergence, Cellular Automata, Learning Algorithms, Artificial Neural Network English ii

3 Acknowledgments Writing a thesis is not an easy task, and I would not have been able to do it if it was not for the support of many people, not just during the process itself, but also during my whole education. First, I want to dedicate this work to my family, specially my parents Alejandro and Élida Josefa, and my sister Gloria, because without their support and guidance during all these years, I would not be the person I am today. I also want to express my gratitude to all my teachers, specially those who were excel in their work and passed me more than just knowledge but also values that make me who I am. I would like to express a special thanks to my supervisor Dr. Timo Honkela, who directed this work and always believed in me. I certainly do not forget my friends and colleagues, specially Alberto L., Alberto S., André, António, Isabel, Katariina, Óscar and Paloma, who not only have supported me and been with me both the good and the bad times, but also contributed to this work with their invaluable comments. As a final note, I would like to apologize to all of those who I inadvertently omitted in this acknowledgment, because I cannot possibly name everyone that contributed significantly to this work. Espoo, June 7, 2011 Alejandro López Vidal iii

4 Abbreviations and Acronyms ANN EA GSM MLP IL ITS SLD DLD CLO UML SSH API VNC SFTP GNU GPL JVM ECA Artificial Neural Network Evolutionary Algorithms Global System for Mobile Communications Multilayer Perceptron Inductive Loop Intelligent Transportation Systems Single Inductive Loop Detector Double inductive loop detector camera Linkerover Unified Modeling Language Secure Shell Network Protocol Application Programing Interface Virtual Network Computing SSH File Transfer Protocol GNU s Not Unix GNU General Public License Java Virtual Machine Elementary Cellular Automata iv

5 Nomenclature α i v ff β ρ i τ i a i f F b F f F g f n F p coefficient of drag of vehicle i, depending on vehicle s cross-section and its aero-dynamic shape. the free-flow speed dimensionless slope of the guideway, considering that sin (α) tan(α) being α is the angle of the slope the occupancy time of vehicle i reaction time of the driver from vehicle i the acceleration of vehicle i dimensionless coefficient of friction (about 0.3 for cars and buses) braking resistance force fluid resistance force guideway resistance force friction coefficient between the wheels and the ground, a suitable approximation of this coefficient is shown in Eq. 2.5 propulsive force g acceleration of gravity (about 9.8 m /s 2 ) g ri g si h si h ti the time gap of vehicle i the space gap of vehicle i the space headway of vehicle i the time headway of vehicle i v

6 K k k i k j l i m i N o ti P q cap q out R s R t,s R t t i T mp v i v r w x i h Km m space region of measurement density of the traffic flow power to weight ratio of the vehicle i the jam density length of the vehicle i mass of the vehicle i number of vehicles in the measurement region on-time. Time period the i vehicle is above the detector. Lenght of the population capacity flow the outflow from a jam to a queue discharge capacity space region of measurement K during an instant dt region of measurements corresponding with a region of the road K during a given time T mp region of measurements corresponding with a point in the road dx during a given time T mp time instant of the passing of vehicle i time of measurement velocity of the vehicle i vehicle speed relative to the fluid the characteristic/kinematic wave speed position of vehicle i hour Kilometer meter vi

7 pat s tic patch second ticks vii

8 Contents Abbreviations and Acronyms IV 1. Introduction Definition of the problem Scope of the thesis Layout of the thesis Traffic flow theory History Microscopic Car-following model Macroscopic Density Flow Mean speed Occupancy Fundamental relation of traffic flow theory Shock waves Fundamental diagrams and empirical measurements Space-mean speed versus density Flow versus density Space-mean speed versus flow Traffic flow regimes single-regime models multi-regime models states theory State of the art Marching Green wave Self organized algorithm (SOLA) Manual tuned system viii

9 3. Methods and tools Artificial Neural Networks: Multilayer Perceptron Model and topology Training Evolutionary Algorithms General process Neuroevolution Cultural Evolution Language Emergence Simulation Characteristics Software tools Java Neural Network Toolbox in Matlab NetLogo Other software Development Workbench: Overall architecture UML Class Diagram Example of an experiment Scenarios Simple city City with four directions City with four directions and turns Fitness Function Plan Results Simple city Biological evolution Cultural evolution Evolving communication Other scenarios City with four directions City with four directions and turns Discussion and Conclusions Discussion of the results Validation of the NetLogo simulator Future work ix

10 Try different fitness functions Extending the simulation Implement the solution Conclusions A. Possible Combination of Lights 81 B. Structure of a Network File 86 x

11 List of Tables 2.1. Mathematical formulation of the car-following model Greenberg s relation for the traffic flow Rules of the SOLA algorithm Back-propagation algorithm in pseudo-code General evolutionary algorithm in pseudo-code Principals measures in the simulator Rules of the cellular automaton that simulates the simple city Summary of the experiments realized with the simple city Summary of the experiments to obtain the best value for the variance in the cultural evolution Summary of the experiments realized with the city with four directions Summary of the experiments realized with the city with four directions and turns xi

12 List of Figures 2.1. Main properties of a car and the interactions with the leading vehicle Trajectories of vehicles i and i + 1 with constant speed v i Results of a simulation of 2 cars using a formulation of the car-following model Principal measures used in macroscopic models Fundamental diagrams of the Greenshields theory and scatter plots of real data q e (k) diagrams representing an inverted-lambda shape and a Kerner s three-phase traffic theory Principal measurement areas around a traffic light Multilayer perceptron Encoding a neural network on a Chromosome Imitation learning with 2 neural networks UML Diagram of classes in the main program UML Diagram of classes in the node Sequence diagram of an experiment Sequence diagram of a simulation Example of the 3 scenarios and cellular automata simulator Rules applied in an intersection regulated by a traffic light Plan for controlling the density of cars Results of the evolution in the simple city Results of the direct imitation Results of the cultural evolution in the simple city Results of the evolution in the city with four directions Results of the evolution in the city with four directions and turns xii

13 Chapter 1 Introduction Life is too short for traffic Dan Bellack [26] Traffic congestion is one of the most acute problems in big cities. Capitals and big cities all over the world have rates of over 64% of people commuting to work by car, e.g. Stockholm, Toronto, Johannesburg, Melbourne, New York and Los Angeles. The average time spent on going to work is 32 minutes, covering a distance of about 21 km. In addition, Beijing and Mexico City, despite their high bus rates, are the two cities with the most painful commutes according to the survey carried out by IBM [26]. In general, the participants of the survey felt that the traffic situation had become worse or a lot worse in the past 3 years, with 20% of them saying that it had not improved at all, and a modest 5% stating that the condition had improved substantially. Although the information provided in the previous claim might be sufficient to justify the investigation of new ways to improve traffic flow, and hence the overall efficiency of the commutes, the same survey released additional interesting information. Participants of the study reported that their stress and anger had been incremented by traffic when they were asked to tell how traffic had disturbed their lives. They also recognized that their productivity in school or at work was affected. Furthermore, 38% of the polled (8,192 people) declared to have canceled a planned trip due to anticipated traffic. [26] Nevertheless, these subjective opinions are not the only consequences of congested traffic. There are other issues that have to be taken into account, such as the increase in greenhouse gas emissions caused by the extra time that the car engine is working or the increase of accidents on account of bottlenecks. Fortunately, current developments in traffic control technology are obtaining promising results, helping people to arrive faster at their destina- 1

14 CHAPTER 1. INTRODUCTION 2 tion 1. There are several approaches on directing the flow of traffic using traffic lights. Some of them are simple, consisting in changing the light color every pre-established amount of time. Others are more complex and require synchronization among several signals, e.g. the green-wave method [51]. Alternative methods are self-organized [3, 10, 18] and require the presence of several sensors in the road. Moreover, Artificial Neural Networks (ANN) 2 have been applied to control traffic lights [45, 56]. ANN present several advantages, such as adaptability, flexibility and extensibility. Nevertheless, from our point of view, these advantages were never completely attained by previous approaches. The main problem is the learning process they follow to adjust the ANN. It is based on supervised methods, while we think it is more adequate to use optimization algorithms according to the nature of the task. The optimal coordination of traffic lights is intractable [39]. This implies that there is a high complexity in obtaining the data necessary to perform supervised learning. The purpose of this thesis is to use ANN evolved using Evolutionary Algorithms (EA) 3 to control traffic lights. The methodology followed during the research consists of two phases. First, the implementation of the discussed method as well as some of the approaches mentioned in section 2.6. And second, computer-based simulation of the different solutions in distinct scenarios to compare the results Definition of the problem Consider the following problem: In a given city there are several traffic lights. Each one of these traffic lights has sensors that detect the number of cars before and after it along different distances, i.e. using an inductive loop (IL) or a camera to detect vehicles [22]. Traffic lights have also a way to communicate with each other using radio signals, Wi-Fi, Ethernet, GSM 4 or any other communication method. Furthermore, signals can record historical data from their sensors. With all this information, a rational decision should be made to improve the current traffic situation. This decision may consist on maintaining the same state of the lights or changing to a different state. The decision can be made either by a local controller at the signal location 1 Some of these developments and their results are showed in Section For a further discussion of ANN see Section EA are explained in more detail in Section For a complete explanation of a real implementation of communication among different traffic entities, see [56]

15 CHAPTER 1. INTRODUCTION 3 or by a central controller. We decided to implement the latter in this thesis. In this case, all the data from the sensors is collected and sent to the central control unit. When the decision is made, the central controller sends the information back with the new state to the traffic light. Of course, when using this approach the communication among lights has to be simulated Scope of the thesis The goal of this thesis is to simulate and evaluate different traffic control approaches. In more detail, we simulated three different scenarios with different levels of realism. Additionally, we develop our own approach that consists of controlling the traffic lights with an ANN evolved using different EA: biological evolution, cultural evolution and the evolution of communication Layout of the thesis This thesis is divided in six chapters: introduction, traffic flow theory, methods and tools, development, results, and conclusions and discussions. In the traffic flow theory chapter, we provide the basic notions in this subject to understand the problem, and, in the state of the art section, an overview of related work, focusing on previous solutions. In the third chapter, we provide the theoretical framework that supports the development of our solution. We explain ANN and EA as well as Simulation from a theoretical point of view. Finally, the last section describes the software tools that support the development of our system. The development chapter explains the implementation of the necessary software to carry out different experiments along three scenarios. The results of these experiments are summarized in Chapter 5. Chapter 6 discusses about these results comparing them with those from other methods. Finally, we explain our final conclusions and consider future developments.

16 Chapter 2 Traffic flow theory Traffic flow theory 1 is fundamental for a correct comprehension of the underlying traffic mechanisms. It provides necessary calculations for an efficient transportation plan and also measures of performance of these plans. For a better understanding of this theory, we firstly present a brief history of it. Secondly, we introduce some important concepts about the principal perspectives of this theory, namely microscopic and macroscopic. Finally, the principal traffic flow regimes are identified and described in the last subsection. Additionally, in the State of the art section we present several approaches that deal with the problem presented in Section History In the beginning there was the Ford [15] Traffic flow theory started with the contribution of scientists from different disciplines, such as mathematics and physics. Early attempts adopt either the microscopic or the macroscopic theory [15], depending on the knowledge background of the scientists who study the phenomenon. Microscopic models examine every vehicle individually considering its characteristics, including length, speed, position and acceleration [33]. One of these original models was developed by Reuschel and Pipes [41]. This model is based on the concept that the speed of the following car is a linear function of the distance between the lead car and the following car. Unfortunately, this concept has never been proven empirically [15]. However, in 1958 Herman and Montrol [9] created the car-following model, which continues being 1 Also knows as intelligent transportation systems (ITS) by the industry 4

17 CHAPTER 2. TRAFFIC FLOW THEORY 5 where, M n a n (t T ) = λv(n 1)/n (t) M n : Mass of n th vehicle a n (t T ): Acceleration of n th vehicle after reaction v(n 1)/n (t): Relative speed of (n 1) th to the n th vehicle in time t T : Response delay of a driver. T = 1.5s λ: Sensitivity Term. λ /M = 0.37s 1 Table 2.1: Mathematical formulation of the car-following model and its principal variables. useful at present and is the basis of many current theories. The car-following model is similar to the Reuschel and Pipes one, but with a slightly different concept; that is, the acceleration of the following car is proportional to the relative speed between the lead and the following car, with a time-lag. The mathematical formulation of this model is presented in Table 2.1 On the other hand, the macroscopic approach describes the traffic flow as a whole. In this case, rather than individual characteristics, it is measured the mean speed, the flow rate, or traffic density among other variables [27]. Lighthill and Whitham [31] derived in 1955 a model using this point of view. It is based upon the fluid mechanics and represents the traffic as a continuum similar to a fluid. This model is useful to describe some intrinsic peculiarities of traffic, such as the propagation of shock waves [43]. However, this model produces errors in a wide range of scenarios and should not be use in practical applications to model the movement of traffic [15]. In 1959, the apparent rivalry situation between these approaches changed when Gazis et al [17] showed that a macroscopic relationships can be obtained from microscopic models. They used the car-following model to derive the main relation of the Greenberg theory [20]. This relation, showed in Table 2.2, is based in the equation of continuity of a compressible fluid. During the following years, the two-fluid model [24] was the only remarkable contribution of the scientific community to this field. This model presents the traffic in cities as if it was composed of vehicles that are moving and vehicles that are stopped. This helps to describe any city with only two parameters, the percentage of cars that are not running and the average speed of those that are moving. Those parameters can be extracted from a

18 CHAPTER 2. TRAFFIC FLOW THEORY 6 where, q = ρu = cρ ln ( ρ j /ρ) u: Velocity ρ: Density c: Optimal speed, depends on road and vehicles characteristics ρ j : Density of traffic on traffic jam, also depends on external conditions Table 2.2: Greenberg s relation for the traffic flow and its principal variables. concrete town and empirical evidences show that the model is very accurate and reliable. In the past two decades, an increasing number of studies have been published in this area. Currently, the number of specialties involved in ITS are increasing, involving aspects from sociology, psychology, economy or environmental science among others. In addition, there are theories that cover almost the entire set of possible traffic situations. In our opinion, the only problem is the lack of an unified theory that embraces all the aspects of the rest of theories. However, solving this problem is out of the scope of this thesis Microscopic 2 When talking about microscopic characteristics, we mean those that are related with individual vehicles and interactions among them as well as with the road and other infrastructure. This characteristics start with the condition of the driver. This condition can be separated into several factors such as the age, stress levels, visual perception, fatigue and medical conditions. These factors lead to different variables, among which, the reaction time of the driver to diverse situations τ i is the most important one affecting the outcome. Due to its stochastic nature, the usual implementation of this behavior is along with a computer simulation model [27]. Although all these factors and variables may be considered to build an accurate model of traffic, it creates a more complex model which in many cases will be discouraged as 2 The name of the variables and symbols from this point agrees with the notation presented in [33], used names are described in the nomenclature section.

19 CHAPTER 2. TRAFFIC FLOW THEORY 7 it increments unpredictable situations [32]. More commonly, microscopic models contemplate aspects from physical properties of vehicles and roads. The most common ones are the following [11]: Propulsive force: This force applies in the direction of the vehicle. It is the result of the engine power, the coefficient of friction and the gravity force. An approximation of the formula is: { F p m = a p g min f n, k } (2.1) v This approximation assumes that the power supplied by the engine is constant, which can be assumed for electric motors but is more inaccurately for internal combustion engines. However, this formula is useful as it shows that the maximum acceleration is inversely proportional to the speed, which implies the existence of a theoretical maximum speed regardless of the power of the engine. Fluid resistance: This force represents the importance of the wind and the aero-dynamism of the vehicle. A general purpose formula of this force is: F f m = αv2 r (2.2) From this formula we can extract that the force opposes the motion. Nevertheless, a pair of considerations must be done. First if the wind blows in the direction of travel, the sign of the formula should be reversed, implying that the force favor the movement; and secondly, the speed of the wind should be considerate 0 in case of cross-winds; that is, the component of the wind in the vehicle s direction is 0. Rolling resistance: When the tire rolls on the ground, it generates a force which has a component that always opposes the movement. This force, called rolling resistance, is the main cause of the sound and heat produced by the wheels. It can be approximated by a linear function of speed. Nonetheless, despite its importance at low speed is not very noticeable at high speeds when the fluid resistance is more important due to the square exponentiation of the speed in the formula of the latter. Braking resistance: This force is proportional to the intensity with which the brake pedal is depressed. Generally, when the propulsive force is greater than zero, this force is equal to zero, and vice versa, due to the

20 CHAPTER 2. TRAFFIC FLOW THEORY 8 fact that the throttle and the brake pedal are rarely depressed at the same time. The bounds of this force are: 0 F b m gf n (2.3) Guideway resistance: The last force to be consider here is the guideway resistance. This force is only present when the trajectory described by the vehicle is not linear and/or not horizontal, i.e., there is a positive or negative slope; and/or there are curves in the road. For convenience, we can divide this force into two components. When the path is not horizontal exists an acceleration that opposes the movement, in case of a positive slope, and helps the movement if the slope is negative. Road inclination is crucial to understand some cases of congestion near bridges and tunnels [33]. A suitable approximation of the function of this forces is: F g m = gβ (2.4) In this equation, we assumed that the slope of the guideway β is small enough to consider that the sine and the tangent of the angle are approximately the same. Finally, the effect of the curves is important as it affects the coefficient of friction used in Eq. 2.1 and Eq The updated version of the friction coefficient is: where c: d2 y(x) dx 2 f n f [ 1 + v 2 c/g ] (2.5) Finally, all the microscopic theories share the use of properties from the vehicle and basic interactions between that vehicle and the one ahead. The most important of these properties are presented in Fig In that figure we can see the position x i and length l, of a given car, as well as the space gap g si and space headway h si of that car with respect to the next one. By general consent, the reference point of the vehicle is its rear bump. There are two perspectives to understand the space headway h si, it can be seen as the difference between the vehicles position x i+1 x i and the sum of the length of the vehicle and the space gap l + g si.

21 CHAPTER 2. TRAFFIC FLOW THEORY 9 Figure 2.1: Main properties of a car and the interactions with the leading vehicle.the car i is in position x i and the leader i + 1 in position x i+1. The position is typically measured from the rear bumper of the vehicle. The car has a length l, which added to the space ahead g si gives the space headway h si. The next figure (Fig. 2.2) shows the trajectories of two vehicles in a spacetime diagram. Both trajectories correspond to vehicles driving at constant speed v i, which is the tangent of the trajectory. In that figure we can see time and spatial characteristics. The main time characteristic is the time position t i, which represents the time passed since a initial instant t 0. All the other attributes can be derived from this one. It is also of importance the time headway h ti which, as well as the space headway, can be seen as the difference between the vehicles time position when passing a concrete point t i t i+1 or the sum of the occupancy ρ i and the time gap g ti. The spatial characteristics are the same described for the previous figure Car-following model The car-following model is one of the fundamental microscopic models of the traffic flow. The original formulation of this theory is presented in Table 2.1 for historical reasons. It was first formulated by Chandler et al in 1958 [9], but it is still currently valid with slight modifications. An example of one of the current modification of this theory is: a i (t i τ i ) = Sens v i+1 (t) v i (t) (x i+1 x i ) 2 (2.6) Sens: Sensitivity of the driver, tipically 5000 m2 /s The main difference between the two approaches is that in the latter case the distance between the vehicles is decisive. In the modern formulation, when the vehicles are separated enough, the influence is almost canceled,

22 CHAPTER 2. TRAFFIC FLOW THEORY 10 Figure 2.2: Trajectories of vehicles i and i + 1 with constant speed v i. The x axle represents the time and the y axle represents the distance. The headway time h ti can be seen as difference of times when the cars pass the same point t i t i+1. This value can be decomposed between time gap g ti and occupancy ρ i. The space measures are the same than in Fig Diagram based on [27].

23 CHAPTER 2. TRAFFIC FLOW THEORY 11 while in the original, the influence is constant. In typical circumstances, the former theory behaves the same as the latter with a constant speed of approximately 116 m. In the next figure (Fig. 2.3) two cars are simulated using Eq The acceleration of the following car is calculated using a reaction time τ i of 1 s. It is clear that, as expected, the speed of the first car is replicated by the second one with a certain delay around 1 s. Another point to notice is that during the stationary time, from t = 20 s to t = 35 s the speed difference is decreasing as well as the acceleration of car 2. However, the prior formula doesn t behaves as expected when the leader car stops. In that case, the speed of the following car becomes slightly negative even though the cars are separated. Moreover, when the initial distance is smaller (e.g., 50 m), the behavior is more erratic, with minimum accelerations of m/s 2 which is physically impossible. Thus, in order to agree with Eq. 2.1 and Eq. 2.3 we can change Eq. 2.6 as follows: ( ( a i (t i τ i ) = min max Sens v ) ) i+1 (t) v i (t) (x i+1 x i ) 2, -9.8 m /s 2, 9.8 m /s 2 (2.7) In this equation we assume that the friction coefficient is 1. Additionally, we force the speed to remain positive regardless of the acceleration. Although this changes made the simulation operate between acceptable bounds, the acceleration and speed of the second car do not result realistic. Furthermore, a new simulation, with a initial distance of 20 m, shows that the second car overtake the leader. In conclusion, the car-following model of Eq. 2.6 is useful to simulate a free flow traffic in a highway, but is inadequate to describe the traffic in a city with cars stopped at traffic lights and similar situations. Additionally to this analysis, the fundamental diagrams derived using this model are shown in subsection Macroscopic A different, but related approach to traffic flow theory is the microscopic point of view. From this perspective, traffic is seen as a whole and individual properties are hidden for the benefit of collective measurements. The most important of those measurements are explained in the next subsections. In order to obtain these characteristics, any region shape can be used; nevertheless, for convenience only rectangular regions are used here. Thus, Fig. 2.4 presents a summary of the most characteristic macroscopic measurement regions. In this figure there are three regions well delimited. R s denote a spatial

24 CHAPTER 2. TRAFFIC FLOW THEORY 12 Figure 2.3: Results of a simulation of 2 cars using the formulation of the car-following model described in Eq.2.6. Both cars start in a resting position separated by 100 m. The leading car has a initial acceleration of 1 m /s 2 during 20 s, stays with the same speed during another 15 s and brakes with a rate of -1 m /s 2 for 20 s; finishing at rest position for another 5 s. a) Represents the acceleration of both cars; b) the distance from the reference point; c) the speed difference, which is used in the formula to calculate the acceleration of the following car; d) shows the speed of both cars.

25 CHAPTER 2. TRAFFIC FLOW THEORY 13 Figure 2.4: Principal measures used in macroscopic models. The time-space diagram represents the trajectories of several cars when passing through different measurement regions. R t represents a point in the road for a time of measure T mp ; R s is the space region K in a concrete time and R t,s is the space region K during the measurement time T mp. region in a concrete instant of time which can be obtained, for example, from an aerial photography. The temporal region R t corresponds with recorded data of vehicles occupancy in a point of the road obtained, for example, by single inductive loop detector (SLD). Finally, the region R t,s represents a general area of measurement with K length during T mp time whose data can be obtained for example from a succession of photographs from a aerial video camera. Note that we not explicitly describe the multi-lane counterparts of the equations presented in the following subsections for space economy and because the derivation is in most cases straightforward. For a more detailed derivation refer to [33] Density Following the notation described in [33], density is represented by the symbol k with vehicles per kilometer as its unit. The most intuitive approach to calculate this variable is to count the number of vehicles in the region R s of Fig. 2.4 and divide it by the length of the area K. This process leads to the following equation: k = N K (2.8) Density can be also seen as the total time spent by all the cars in a region divided by the area of the region[27]. This point of view is useful when it is not possible to measure the density in a spatial area. The previous definition leads to the following equation in the temporal region R t :

26 CHAPTER 2. TRAFFIC FLOW THEORY 14 N i=1 k = T i T mp dx = 1 N dx = 1 T mp dx v i T mp i=1 N i=1 1 v i (2.9) According to this equation in order to calculate the density in R t it is necessary to know the instant speed of the cars passing through the measurement point. One step further is to calculate the density in any measurement interval, for example R t.s. For this case, we can use Eq.2.9 also, but we need to know the total travel time, which is not always possible. However, we can calculate the density for any given time using Eq.2.8 and then, add all those partial measures as follows: k = 1 T mp k (t) (2.10) T mp This equation represents the discrete version, where, in case of a continuous function, the summation can be substituted by an integral. Finally, it is important to highlight the link between microscopic and macroscopic characteristics. In this case, density is equivalent to the reciprocal of the average space headway h 1 as shows the following equation: Flow k = N K = s t=1 N N i=1 h s i = 1 N 1 N i=1 h = 1 hs (2.11) s i Flow is the temporal counterpart of density. It can be described as the number of vehicles passing through a point in the road per time unit. This description leads to this equation: q = N T mp (2.12) This calculations are easily performed in the temporal region R t but are more difficult to perform in R s. Fortunately, flow can also be described as the total distance traveled by all the vehicles in the measurement region, divided by the area of this region[11]. This formulation is equivalent to the next equation: q = N i=1 X i Kdt = 1 K dt N v i dt = 1 K i=1 N v i (2.13) i=1

27 CHAPTER 2. TRAFFIC FLOW THEORY 15 Continuing the analogy with density, we can add consecutive measurements of the flow in order to obtain the flow in any arbitrary measurement region R t,s : q = 1 T mp q (t) (2.14) T mp There is also a microscopic equivalent to the flow, the reciprocal of the average time headway h 1 t. This equivalence can be seen in the following equation: q = N T mp = Mean speed t=1 N N i=1 h t i = 1 N 1 N i=1 h = 1 ht (2.15) t i The space-mean speed or simply mean speed corresponds with the total distance covered by the vehicles divided by the measurement time. In R s consist in the arithmetic mean of the speed of the vehicles present in that area while in R t is the harmonic mean: N N i=1 v s = X i=1 v i dt = 1 N i N dt N i=1 N i=1 T = v i (region R s ) N dx (2.16) i N i=1 dx 1 N (region R i=1 1 s ) v i 1 = v N i It is important to note that in the temporal area, the mean speed is calculated using the harmonic mean instead of the arithmetic mean and in the spatial area exactly the opposite. In case of using the other formula, the obtained variable is called time-mean speed v t in contrast to the spacemean speed v s, the latter being the most used and important. v t is generally greater than or equal to v s, according to the generalization of the inequality of arithmetic and geometric means. This last inequality can be understood easily if we consider that, when the time-mean speed is assessed, faster cars are consider over a much longer road section than slower cars [27]. As it is shown by Wardrop [55] v t = v s + σ2 s v s, where it is easy to see that if the sample variance σs 2 is close to zero, both, v t and v s are the same. On the other hand, sometimes it is necessary to calculate v s when having v t instead. To solve this problem we can use v t v s + σ2 t v s, which leads to the following approximation derived in [7]: v s v t 2 + v 2 t 4 σ2 t v t 2σ 2 t (2.17)

28 CHAPTER 2. TRAFFIC FLOW THEORY Occupancy Occupancy is the last macroscopic variable to be explained in this section. In theoretical analysis, it is not as significant as the other three ones, but it is remarkably more important during empirical measurements. This importance appears when it is not possible to obtain vehicle s instant speed, for example, when it is not possible to install a DLD (double inductive loop detector) and is only present a SLD. That kind of detectors can only report the occupancy, which corresponds to the fraction of time the measurement location was occupied by a vehicle [33]: ρ = 1 T mp N o ti (2.18) The vehicle on-time corresponds with the effective length of the car seen by the sensor divided by the speed of the vehicle. This effective length corresponds with the length of the car added to the sensor s own detection zone.o ti = l i+k ld v i. Finally, the importance of the occupancy appears with the following relation that holds for stationary traffic [11]. This relation allows us to obtain the density, when knowing the occupancy measured by the detector and the average length of the vehicles. i=i ρ = lk = k = ρ l (2.19) Fundamental relation of traffic flow theory Density, flow and time-mean speed are related through the fundamental relation of traffic-flow theory [55], which is very useful when knowing two of the variables, because it allows obtaining the third one. q = k v s (2.20) This relation can be used whenever q, k, and v s are continuous variables or smooth approximation of them, and the traffic consists of substreams composed by homogeneous and stationary traffic. Homogeneous traffic stands that every traffic substream has a homogeneous composition, i.e., the same type of vehicles. Finally, stationary traffic holds when all vehicle s trajectories are parallel and equidistant. This last condition is difficult to appreciate during a small window of time. However, it can be relaxed because it corresponds to a macroscopic scale. This relaxation allows to determine that there is stationary traffic when the total distance traveled by the vehicles and

29 CHAPTER 2. TRAFFIC FLOW THEORY 17 the time needed to cover that distance is the similar regardless of the size of that measurement area [11] Shock waves Shock waves [43] represent the apparent movement of disturbances through the traffic stream. A good example of a shock wave occurs when the distance headway is small due to high density (k > k c ). In this situation, a car in the stream can brake for any reason and rapidly accelerate again, to compensate the deceleration and continue with the initial speed. This event generates a chain reaction in the form of a wave, where the following cars mimic the behavior of the leading car. Shock waves are of two types, upstream and downstream. The example above illustrates an upstream shock wave, which are the most noticeable ones as they happen continuously during traffic jams. However, despite the apparent predominance of this type of shock waves, downstream ones also occur during the free-flow phase of the traffic. Along that regime, when the density is higher upstream than downstream, can happen that the acceleration of the leading vehicle generates a downstream shock wave as is shown in computer simulation as [19]. Shock waves are a decisive factor in the generation of traffic congestion. Some experiments regarding traffic waves suggest that traffic jams can be diminished or even vanished due to a driver special care on avoiding stop and go waves [5] Fundamental diagrams and empirical measurements The fundamental diagrams represent a visual tool for understanding different traffic flow theories and their implications. It is important to mention that the existence of a direct causal relation has never been shown between any of the variables displayed in the following diagrams. Therefore, the fundamental diagrams represents models that try to fit the experimental data with their curves. As a consequence, each theory draws a different diagram. For example, the car-following model presented in Eq. 2.6, can be derived in the following expression when there is homogeneous and stationary traffic [27]:

30 CHAPTER 2. TRAFFIC FLOW THEORY 18 Figure 2.5: Fundamental diagrams of the Greenshields theory [21] and scatter plots from data collected by video camera CLO3 located at the E17 three-lane motorway near Linkeroever, Belgium [1]. a) Is the space-mean speed versus density or (k, v t ) diagram, b) is the flow versus density or (k, q)diagram, and c) space-mean speed versus flow or (q, v t ) v s = v ff k j k + v ff (2.21) and the relation between qand v s can be derived as follows: v s = v ff ( 1 k k j ) 1 v s = k v s k v ff k j k j q(fund. rel.) = v s v2 s v ff q = k j ( v s v2 s v ff ) (2.22) This expression, when depicted, represents the fundamental diagram of Greenshields [21] which can be seen in Fig. 2.5.

31 CHAPTER 2. TRAFFIC FLOW THEORY Space-mean speed versus density This first fundamental diagram represents the equilibrium relation between space-mean speed and density v se (k). Fig. 2.5 a) depicts this fundamental diagram according to Greenshields theory, as well as a scatter plot of real data. This visual aid helps us to verify the first intuitive feature: as the density increases, the average speed decreases monotonically. There are a number of other features we can also extract from this diagram. First, that the density is always delimited between zero and the the jam density k j. Second, the mean speed is restricted between zero and a maximum, which corresponds with the free-flow speed v ff. Also, there is a small range of low densities, in which the space-mean speed remains constant corresponding with the free-flow speed more or less. Although, this third feature is only present in the scatter plot, in the original Greenshields theory there is a an artificial flattering of its upper-left part, which is not present in the figure because the function was created according with Eq The final feature is that the flow can be obtained as the area of a rectangle delimited by the origin and a point of the fundamental diagram. This is represented in the picture by the area of the gray rectangle Flow versus density The most extended fundamental diagram is the one representing the equilibrium relation between flow and density q e (k). In Fig. 2.5 the diagram b) shows one of this possible relations. Like in the previous case, this diagram is the one corresponding with Greenshields theory. In that sub-figure is also plotted real data which shows the discrepancy between that theory and the reality. It also shows that no simple relation can be made between flow and density because the points are extended along a big cloud. Nevertheless there are several features that can be extracted from this diagram: From the origin to near the critical density k c, the flow increases in a linear manner. Usually, this region is called the free-flow branch of the fundamental diagram. Close to the critical density point, there is a bending of the fundamental diagram. This circumstance is a consequence of the blockade produced by slow cars impeding the fastest. At the critical density point k c, the flow is maximum. This point is known as the capacity flow q cap.

32 CHAPTER 2. TRAFFIC FLOW THEORY 20 After the critical density point is reached, the traffic state changes to the congested regime. During this regime, the state of the traffic deteriorates as the density increases. When the jam intensity k j is reached, the flow becomes zero and the vehicles are stopped. As in the case of the previous diagram, the other variable that appears in the fundamental relation of the traffic flow theory can be extracted for any point. In this case, the mean-speed can be calculated as the slope of the line through that point and the origin. Finally, it is important to mention that from this diagram can be extracted the kinematic wave speed w. The value of this variable corresponds with the slope of the tangent in any point of the diagram. According with this description, the shock wave travels downstream during the free-flow regimen, disappears when the critical point k c is reached and travels backwards or downstream in the congested phase Space-mean speed versus flow This third type of diagram differs from the previous ones in that it does not represents an injective function. In Fig. 2.5 c) can be seen that, for almost any flow values, there are two values of the mean speed. To be more precise, the inflection point is the only value of the flow to which corresponds an unique mean speed. This inflection point matches with the capacity flow q cap, which separates the free-flow regime and the congested regime, i.e., upper branch and lower branch respectively. The mean speed versus flow diagram v se (q) is easily understandable for some researchers, such as economists, depending on their background. Nonetheless, is not as intuitive as the v se (k) diagram and therefore it is not used very frequently [33] Traffic flow regimes There are many different traffic flow theories. One of the first ones is the Greenshields theory [21] which was first proposed in 1935 and whose fundamental diagrams are depicted in Fig One important characteristic of this theory is that the diagrams are explained using a single equation. On the other hand, newer models use different equations depending on the regime in which there is the traffic flow. As a consequence, different models can be classified depending on the use of one or multiple equations to obtain its fundamental diagrams.

33 CHAPTER 2. TRAFFIC FLOW THEORY 21 Figure 2.6: q e (k) diagrams representing an inverted-lambda shape (a) and a Kerner s three-phase traffic theory (b) single-regime models Models that use only one equation are usually named single-regime models, for example, Greenberg and Underwood models [20, 53]. The aim of these models is to simplify traffic phenomena at the same time that tries to fit the observed data into its curves. However, the existence of a single equation to cover the entire range of circumstances, does not mean that it exhibits a homogeneous state, but quite the opposite. As we claim in the previous subsection, the fundamental diagrams of the Greenshields theory depicts two different traffic flow states, namely free-flow traffic and congested traffic. Furthermore, in this theory, those regimens are separated by the global maximum k c of the function in the q e (k) diagram multi-regime models Likewise, models that use more than one equation are named multi-regime models. Multi-regime models were first introduce by Edie in 1961 [13] and other examples are the theories form Smulders [47] and Newell [36]. These models evaluate the original conditions of the traffic flow and then determine in which regime the flow is placed and thus applying the correct equation. A good representation of this type of models is represented in Fig. 2.6 a). This fundamental diagram presents two straight lines overlapping in what is called inverted-lambda shape. During the free-flow period, the flow augments with the density until it reaches k c, which coincides with q cap (1). At that point the flow drops dramatically as it enters in the congestion regime (2). During that regime the flow decreases with the increase of the density.

34 CHAPTER 2. TRAFFIC FLOW THEORY 22 However, the same process does not apply when the model is in the congestion regime and the density starts to decrease. In that situation, when the density reaches k c nothing anomalous happens and the flow continues increasing with the decrease of the density. This condition continues until the two lines intersect at the outflow from a jam q out, point when it shifts back to the free-flow regime. The hysteresis behavior presented by this model can be observed in real situations as Treiter and Meyers shows in their work [52] states theory The previous models assume the existence of an underlying causal relation between two of the variables involved in Eq On the other hand, Kerner is more skeptical and he refuses this idea, which is substituted by his fundamental hypothesis of three-phase traffic flow theory [29]. This theory is based in the assumption that the traffic flow can be separated into three phases, namely free flow (F), synchronized flow (S) and wide-moving jam (J). The q e (k) diagram corresponding with this theory can be found in Fig. 2.6(b). Observing that figure is obvious that the F and the J phases corresponds with the concept of equilibrium relation between density and flow, or in other words, a fundamental diagram. Moreover, those phases are similar to the inverted-lambda diagram discussed in the previous point. However, the synchronized flow phase contrasts with this notion as it covers a two-dimensional region. Due to this crucial factor, this theory differs from the other multi-regime models. Kerner uphold that the direct transition from F to J can be possible but not likely to happen. For him, the natural transition is F->S and S->J because it is not necessary a critical perturbation to change the states. An important issue regarding this model is that it gives no explanation for some transitions and simply describes them. As a consequence, Kerner himself accepts that his theory is merely qualitative. However, several microscopic models show an agreement between their results and Kerner s theory. Some of these models are [30] and [28], using both cellular automaton in their construction. The popularity of this theory is incrementing due to the support of those microscopic models along with the fact that it fits nicely with the observed data. At present, thanks to this popularity, this theory is a trend topic that many researches continue investigating and expanding [33].

35 CHAPTER 2. TRAFFIC FLOW THEORY State of the art An important issue in controlling the traffic in cities are the intersections. In most cases, when the quantity of the traffic is high, it is necessary to use traffic lights to control the vehicles and the pedestrians equitably and with security. Due to its importance, traffic engineers strive to increment the efficiency of the controlling systems. Their effort includes the development of new technologies to synchronize the lights and the addition of sensors to detect the state of the traffic. Our intention is to provide a method that uses all these information and infrastructure to support the decision process in controlling these signals. In this section, we review some of the most typical techniques to control the lights. Furthermore, we discuss the integration of some of them in our model to validate the obtained results of our solution Marching This technique consists in a manual or automatic synchronization of the traffic lights, which permits them to change colors at the same time, every fixed period. In Fig. 4.5 the three scenarios adapted for our experiments show a snapshot of the city controlled with this technique. In the first and second scenarios (a and c), all the lights in the north-south direction are green, meanwhile in the west-east direction is the opposite and all the signals are red. When the period is about to finish, all the lights that are green change at the same time to amber. Finally, when the period expires, lights that were red change to green, and those that were amber change to red, thus completing the period. The second phase is carried out in a similar manner, which allows a equitable distribution of the preference in the intersection for the 2 directions. In the third city (d) the process is similar, but the cycle is divided into four periods instead of two. In any of these periods, one of the four directions has the priority to pass and the other three wait in red light Green wave This technique is very similar to the previous one: it also needs a synchronization among the signals, and it lacks of any response to the actual situation due to the absence of sensors to control the lights, meaning that they are passive, or likewise, that their their decision on changing the lights is only based on their internal state.to understand its generalization for several streets, first we will focus on a single street. Imagine a straight street with only one direction that has a number of intersections directed by traffic lights. At any given time, when a signal changes from red to green, the next

36 CHAPTER 2. TRAFFIC FLOW THEORY 24 one remains in red for a bit less than the travel time between the lights, and then also, changes to green. If this synchronization continues for all the lights, and the traffic is free-flowing (i.e., there is not a long queue waiting in the semaphore) any car can virtually travel through the entire length of the street stopping zero or one times at most. In case of the city from Fig. 4.5a, the generalization is straightforward and both directions take advantage of the green wave solution in all their streets [51]. The case is different for the second city; due to geometrical constraints, the green wave can only benefit two directions, while cars in the other directions traverse opposite to the green wave, and hence, they stop in several lights, leading to a worse situation than with the marching technique. Finally, no beneficial implementation of this technique exists in the diagram that represents the third city, as a consequence of the turns and the and its layout. When the geography of the area permits the use and full benefits of this technique, it represents an affordable solution that increments the performance of the traffic. Moreover, due to its advantages, it is used in many cities such as San Francisco, Amsterdam, Copenhagen, Dresden (Germany) and Keynsham (UK) [4, 37, 49]. Furthermore, it can be improved by the addition of sensors, as in the GLIDE system [38] Self organized algorithm (SOLA) As stated before, the previously described methods are passive. On the other hand, self organized methods are active, acquiring multiple advantages due to the reaction to the current state. These factors allow them to adapt their behavior automatically to, for example, prevent further problems in a situation of strong traffic or to make changes more dynamically when the traffic is more sparse. Here we explain briefly the self-organized method based on rules that Gershenson and Rosenblueth [19] presented in their article and that expanded the idea from a series of articles by Ball [3], Gershenson [18] and Cools et al. [10]. We call it SOLA after the name they use to reference it in their implementation. The diagram with the schematic of an intersection showing the principal areas needed for the rules is in Fig. 2.7.The rules that controls the light are enumerated in Table 2.3. In a normal traffic situation, rules 1 and 2 tend to create an improved marching like scheme; that is, if the demand is similar in both directions, these rules distribute equally the time the light is green, but if it is not the case, they benefit the direction with more traffic. The 3th rule permits the emerging of a green-wave, as it allows few cars (that should take part

37 CHAPTER 2. TRAFFIC FLOW THEORY 25 Figure 2.7: Principal measurement areas around a traffic light. 1. On every tick, add to a counter the number of vehicles approaching or waiting at a red light within distance d. When this counter exceeds a threshold n, switch the light. (Whenever the light switches, reset the counter to 0.) 2. Lights must remain green for a minimum time u. 3. If a few vehicles (m or fewer, but more than 0) are left to cross a green light at a short distance r, do not switch the light. 4. If no vehicle is approaching a green light within a distance d, and at least one vehicle is approaching the red light within a distance d, then switch the light. 5. If there is a vehicle stopped on the road a short distance e beyond a green traffic light, then switch the light. 6. If there are vehicles stopped on both directions at a short distance e beyond the intersection, then switch both lights to red. Once one of the directions is free, restore the green light in that direction. Table 2.3: Rules of the SOLA algorithm.

38 CHAPTER 2. TRAFFIC FLOW THEORY 26 of a green-wave platoon) to pass through the intersection without stopping. Furthermore, rule 4 is useful in a situation of light traffic (e.g., during the night), making the changes very dynamic, which directly increases the performance in that situation. Finally, rules 5 and 6 appear when the traffic is very intense, and help avoiding dead-locks (whose presence quickly degrade the traffic performance), thus allowing a quick recovery when the situation is better Manual tuned system The Minnesota department of transportation maintains an updated manual on Traffic Signal Timing and Coordination, whose last revision is from March 2009 [34]. This manual explains the complete process on placing a traffic light, since the emergence of the need, to the posterior plan of revisions. The most important points of the planning are the following: A complete research, with several sources, is carried out to gather the information needed to successfully complete the plan. Data and information obtained during this phase include: geometric conditions of the intersection, volume of the traffic, travel time and delay data. Study of the area to: ˆ place the sensors in a convenient position ˆ design and install the signal Creation of the timing control plan, taking care of factors such as pedestrians, cycle lengths and interval of lights. Integration of the system in a simulator. Due to the nature of the task, the computer simulation is of a great importance in the process. It allows to check the validity of the solution prior to the real implementation, saving money and improving the performance without any inconvenience for the driver 3. In this manual, the suggested simulator is Synchro and SimTraffic. Use the simulation to fine-tune the model, which may lead to obtain more efficient results. Implementation of the model in the real situation, with the aid of the results obtained during the previous steps. 3 A further explanation of the simulations and their characteristics and advantages can be found in Section 3.3

39 CHAPTER 2. TRAFFIC FLOW THEORY 27 Test of the implementation. This step is necessary to analyze the congruency between the model and the simulation and to check if the results are as expected. In this point, if the expectations are not fulfilled, the solution is reevaluated to find the origin of the problems. The final point is to create the future plan of revisions. Normally, the correctness of the timing plan is verified at least once a year and a complete analysis should be made every three or five year. Nevertheless, this values can change due to special circumstances; for example, in case of an important intersection or a point where it is expected to have large fluctuations in the future.

40 Chapter 3 Methods and tools This thesis is grounded in concepts from different disciplines, such as computer simulation, artificial intelligence and transportation engineering. These areas are necessary during different phases of the development. Concepts from transportation engineering are necessary to understand the problem and to evaluate the solution. Computer simulation is essential to construct a realistic model of the traffic, which is used both to obtain our solution and to compare different solutions. Finally, artificial intelligence is crucial to construct the adaptable and robust solution that represents the core of this thesis. This chapter presents a brief description of concepts from AI and simulation as well as the software used to implement them Artificial Neural Networks: Multilayer Perceptron 2 Artificial Neural Networks (ANN) is an important branch in the field of artificial intelligence. It has become one of the most extended and known tools to solve a wide range of problems, due to its multiple qualities such a great versatility, adaptability, and self-regulation. An ANN is a computational model that mimics a biological neural system. It represents a model that, given an input or stimulus react producing an output. To be more concrete, it is composed by one or more neurons connected through synaptic links with each other. Thus, the output of one neuron can inhibit or excite 1 Concepts from transportation engineering are explained in the traffic flow theory chapter. 2 This section is based in the Ham s book [23], for an extended discussion about this topic we refer the reader to that book. 28

41 CHAPTER 3. METHODS AND TOOLS 29 other neurons, propagating the stimulus into the entire network until the final output. Some problems that are usually tackled with ANN include supervised, unsupervised and reinforcement learning; regression, classification, pattern recognition and data processing. Additionally, the continued investigation in the field derives into the development of new tools dedicated to each type of problem. There is a large variety of ANN, whose principal differences are the topology, the transition function and the learning algorithm. Thus, this section is mostly focused in the multilayer perceptron (MLP), which is the model used in our experiments Model and topology The MLP presents a feed-forward, multi-layered and fully connected architecture. Therefore, the MLP layers are ordered and every neuron is connected with all the neurons of the next layer, but only with them; that is, no feedback connections are allowed in the model. This structure can be seen in Fig. 3.1 which shows an schematic MLP with a single hidden layer, p inputs, m outputs and L neurons in the hidden layer. This diagram shows the three different types of layers, whose key features are the following: Input Layer: Its inputs correspond with the external inputs. Thus, the number of neurons in this layer correspond with the size of the input vector. These neurons do not need to make any transformation to the data and its only function is to connect each variable of the input vector to each neuron in the first hidden layer. Nevertheless, this layer frequently normalizes the input vector, that is, it transforms the original data in such manner that it can fall into the range between 0 and 1. Hidden Layer: These layers are responsible for carrying out the computation. Each hidden layer can have any number of neurons and there can be any number of hidden layers, although in the representative figure there is only layer. The number of layers and neurons in each layer is determined by the complexity of the problem, but relatively small networks can perform acceptably well. Each input of each neuron has assigned an importance or weight for that neuron. That weight is represented in Fig. 3.1 as w h ij. Additionally, all neurons possess a weight attached to a virtual input that is always 1. This weight is typically called bias and represents the activation threshold of the neuron. Every

42 CHAPTER 3. METHODS AND TOOLS 30 Figure 3.1: Multilayer perceptron weight is multiplied by its corresponding input value and then added together with the bias to obtain the value u j. Finally, a non-linear function σ is applied to the value u j to obtain the final output h j. Frequently this function is a variant of the sigmoid to keep the output within a small range. Output Layer: The final layer is similar to a hidden layer with few differences. First, the number of neurons is not arbitrary and correspond with the length of the output vector. Hence, the output of the neuron corresponds to one variable in the output vector and it is not attached to any other neuron. Finally, the typical function used in this layer is the identity function to avoid restrictions in the output range. The ANNs, and therefore the MLPs, can be seen from a purely mathematical point of view, although its first purpose was to emulate the biological neural networks. From this point of view the weights and biases of the first layer can be arranged in a matrix in the following form: W h = w11 h w12 h w1i h w1p h w21 h w22 h w2i h w2p h wj1 h wj2 h wji h wjp h B h = b h 1 b h 2. b h j. (3.1) w h L1 w h L2 w h Li w h Lp b h L Now, it is straightforward to obtain the vector U, which is simply a matrix multiplication between the weights matrix W and input vector plus

43 CHAPTER 3. METHODS AND TOOLS 31 the addition of the biases vector U = W h X + B h. To continue the calculation of the output it is necessary to apply the function σ to this vector to obtain the vector H which is the final output of the first hidden layer in this example. H = σ h (U) = σ h ( W h X + B h) (3.2) Finally, to obtain the output vector, the vector H is used as the input of the output layer and it is applied the same methodology, thus: Y = σ y (V ) = σ y (W y H + B y ) = σ ( y W y σ ( h W h X + B h) + B y) (3.3) When there are more hidden layers, the same process is repeated for every hidden layer. This mathematical treatment allows us abstract the MLP, thus simplifying it to matrices and matrix operations Training Training an ANN consists in the adaptation of the network to the current problem. This process may radically differ depending on the type of ANN and the problem. In the MLP s case, the procedure consists of two steps: select the topology and adjust the weights and biases (or simply weights) of the layers. The topology selection is usually arbitrary, due to the fact that, relatively large changes in the structure, causes a minor impact in the final result. This selection is frequently part of the designer s choice and it is adjusted using trial and error. Nonetheless, depending on the algorithm, this selection can be automatic or even adapt itself as part of the learning process [48]. On the other hand, the adjustment of the weights is very important. An appropriate adaptation derives in a suitable network, which accurately represents the desired function; while a poor adaptation leads to an undesirable solution. The training method used in this thesis is crucial to understand its results. Our method is based on genetic algorithms and is discussed in the Section 3.2. Nevertheless, the back-propagation algorithm is briefly explained here for practical reasons as it provides a good example of one of the most known training methods. Back-propagation The back-propagation is a supervised learning algorithm that iteratively minimizes the training error. The principal aim of a learning algorithm is

44 CHAPTER 3. METHODS AND TOOLS 32 initialize the network do for each example done calculate the output obtain error update the weights while the stopping criteria is not reached Table 3.1: Back-propagation algorithm in pseudo-code. to teach the network how it should behave. In the case of the supervised learning, it is a set of examples which makes it possible. An example (in this domain) is a pair of vectors with a possible input X and its desirable output D. These examples act as teachers that enhance the understanding of the problem. Hence, as a rule of thumb, the training set should be sufficient large to give a overall view of the problem. This algorithm consists in three parts: network initialization, learning process and stopping criterion. These parts and their more important substeps are presented in Table 3.1. The initialization consists in the selection of the original weights. This selection typically uses random small numbers, although there are other variants. The learning process is divided into several sub-steps, the most important of them is to update the weights. This change in the weights is proportional to the mean-square error. Given an example, the error is calculated as the difference between the output that the network generates Y and the desired one D. This error is propagated backwards, that is the reason which gives name to the algorithm. Thus, the output layer is updated in first place using this equation: where, w y ji (k + 1) = wy ji (k) + µδy j h i (3.4) δ y j = E q v j = (d j y j ) { 1}{σ (v j )} = d j y j (3.5)

45 CHAPTER 3. METHODS AND TOOLS 33 The µ > 0 is the learning parameter. This number indicates how much the weights should change in every iteration. The update formula is slightly more complex in the case of the hidden layers, because it requires the error calculated in the previous layers: w h ji (k + 1) = w h ji (k) + µδ h j x i (3.6) where, L h+1 δj h = δ h+1 k w h+1 kj k=1 σ ( ) u h j (3.7) When h + 1 exceeds the number of hidden layers, δ h+1 k = δ y j and wh+1 kj = w y kj Ṫhis formula is applied to every hidden layer from the last to the first one. When all the weights has been updated, the second example is used to repeat the process. Finally, when all the examples have been used it is called an epoch. Every time an epoch finish, the stopping criterion is evaluated. If the criterion is not accomplished, the algorithm runs for another epoch. The process is repeated until the criterion is reached, moment when the method stops and thus, finishing the learning process. Stopping criteria There is a number of stopping criteria for using in this algorithm. First, the number of epochs can be determined previously, and the process stops when that number is reached. Second, the algorithm stops when it reaches a certain error level. A third criterion is that the system stops when the error is the same or very similar during a certain number of epochs. Finally, any combination of these or other criteria can be used to determine when the algorithm terminates Evolutionary Algorithms This thesis addresses an optimization problem, which was already presented in Section 3. The main characteristic of this type of problems is the impracticality to find the best solution, if any. This problems are usually

46 CHAPTER 3. METHODS AND TOOLS 34 NP-hard, which means that, for moderate size problems, obtaining the optimal solution with the actual technology may require computational time in the order of thousands of years. Fortunately, in most problems, obtaining the optimal solution is not crucial, which is very important as it allows us to search in the space of solutions to find an adequate one. Evolutionary algorithms (EA) represent a branch of artificial intelligence that searches in the solution space using heuristics grounded in the natural evolution of life, which was first presented in Holland s article in 1975 [25]. The evolution of life may consist in a society of similar individuals that compete for the resources and the right to reproduce. This reproduction typically involves two individuals, which grants genetic diversity, the reinforcement of common features, and the appearance of new ones due to the combination of different genes. Moreover, the genes of the offspring randomly mutate to also reinforce the diversity, allowing the appearance of new characteristics that were not present in neither of the parents. The combination of these genetic operators leads to a process that perpetuates good features and penalizes bad ones, resulting in an offspring that is better adapted to the medium in average. Two important concepts in this field are the genotype and the phenotype. These concepts are usually confused, but the correct understanding of their differences is important for an accurate comprehension of the topic. The genotype corresponds with the codification of the individual. On the other hand, the phenotype is the behavior that an individual presents, based on its genome. The most important point is that the correspondence between a gene and a feature is not direct. As a consequence, the whole genome is necessary in order to know the features that the individual is going to present; or similarly, that it is not possible to predict the final behavior based only on its genotype, it is necessary to test the individual. As stated above, EA mimics this natural evolution, which means that all the concepts of the natural evolution has a counterpart in the evolutionary algorithms. First, the natural individual is replaced by a solution of the universe of solutions. Accordingly, the genome is the encoding of this solution, and may vary depending on the problem and the algorithm selected. Second, the phenotype corresponds with the behavior of the solution when it is tested. Third, the adaptation to the medium and thus the reproductive chances are measured using a fitness function. Finally, the crossover consists in the mixture of the parent genes, and the mutation in the random change of the new genes.

47 CHAPTER 3. METHODS AND TOOLS General process Solving a problem with an evolutionary algorithm needs 3 steps: Codification of the solution, selection of the fitness function, and finally, run the algorithm itself. Genes codification First, it is necessary to determine a codification of the solution that fits the problem. During the codification, the main priority for the designer is to maintain it as simple as possible, avoiding redundancies. In addition, the codification might be smooth, which means that small changes in the genotype code should not trigger large changes in the phenotype. An example of an advantageous codification is the binary code. This codification is only possible when the universe of solutions is finite. If it is the case, with a cardinality of N, the number of bits that are necessary to code the solutions is L = log 2 N. In this situation, the redundancy diminishes as N approaches 2 L, disappearing when N = L 2. Additionally, when the solutions can be sorted according to their similarities, the gray code is the optimal encoding. Fitness function The success of the algorithm is conditioned by appropriate selection of this function, because it determines the probability of an individual to participate or not in the following steps. Basically, this function gives a grade to the individuals according with their aptitude in solving the problem. It is important that the function is uniformly distributed over the universe of solutions, in order not to penalize excessively an inadequate solution which might possess a great potential. Finally, the fitness function also needs to be smooth; that is, small changes in the phenotype should not generate large differences in their fitness. The fitness function is typically the most intensive in computational time. This means that an effort needs to be made during the function encoding, to minimize the impact of its running time in the overall algorithm. Due to its importance, the fitness function used in this thesis is further discussed in Section General algorithm The general algorithm to solve a problem using evolutionary algorithms is the presented in Table 3.2. The first decision is to determine the number of

48 CHAPTER 3. METHODS AND TOOLS 36 random initialization of the population do for each example calculate the fitness function done for each new individual select parents according with its fitness combination of the genes from the parents mutate new genes done update population while the stopping criteria is not reached Table 3.2: General evolutionary algorithm in pseudo-code. individuals in the population P, e.g., 50. Afterwards, their chromosome are initialized randomly, which assures an uniform distribution of the population along the universe of solutions. Following, the individuals in the population are ranked using the fitness function. Next, the generation of a new individual starts, firstly, by selecting its parents according with their fitness value. To create the new genome, genes from the father and from the mother are selected randomly, although it is also possible the agamic reproduction; that is, there is only one individual involved in the reproduction. Finally, the creation of the new individual finish with few genes that change or mutate. A number of new individuals (less than or equal top ) create the offspring, who totally or partially replace the older population. This simple cycle, called generation, is repeated until an stopping criterion is reached, in analogy with the back-propagation algorithm. For this reason, refer to Subsection on page 33 for some examples of stopping criteria Training MLPs using genetic algorithms: Neuroevolution During the development of this thesis, we use an adaptation of the algorithm to train feed-forward ANN using genetic algorithms. This algorithm was first presented in the article of Montana and Davis [35], whose main characteristic are related below:

49 CHAPTER 3. METHODS AND TOOLS 37 The encoding of an MLP on a chromosome is presented in Fig This encoding translates all the weights and biases of the network into an array of real numbers. This algorithm was developed to use in supervised learning. Therefore, it is necessarily a set of examples to calculate the fitness function. This value is equivalent to the sum of the square error of the examples when they are presented to an individual. As a consequence, this is consider a minimization problem. The initialization of the network is not arbitrary, and, instead of the usual uniform distribution between [ 1, 1], it uses a symmetric exponential distribution with a µ = 1. The parents are randomly selected using a distribution of probabilities D. Their probability is set according with their position f, when they are ordered by their fitness value. In the process of calculating the probability, first, a value called parent-scalar (s) is selected between the range [0.86, 0.92]. Another value µ is calculated using s and P to make the sum of all the probabilities equal to 1. Finally, their probability to be selected is: D (s, f, µ) = µs (f 1), P f=1 µs(f 1) = 1 µ (s, P ) = 1/ P f=1 s(f 1). There are 3 crossover operators: ˆ Crossover-weights: This operator selects randomly, with equal probability, genes from one parent or the other, one at a time. ˆ Crossover-nodes: Instead of individual weights, this operand selects all the weights related with a node from one of the parents, and they are copied to the new genome as a block. The nodes are selected in the same fashion as the genes from the previous operator. ˆ Crossover-features: This operator is applied in the same way than the previous one. The only difference is that, before the selection, the nodes from one of the parents are rearranged. This aims to place in the same position nodes that plays a similar role in both of the parents. This strategy is more useful during the first generations, previous to the point when members of the population start to show a resemblance among them. And, there are 4 mutation operators:

50 CHAPTER 3. METHODS AND TOOLS 38 Figure 3.2: Encoding a neural network on a Chromosome ˆ Unbiased-mutated-weights: With a probability of 0.1, any of the new genes can be substituted for a new value from the initialization distribution. ˆ Biased-mutate-weights: Similarly, this operator also mutates a gene with a probability of 0.1. However, instead of substituting it with the new value, the new gene is the result of the sum of the old value and the new one. ˆ Mutate-nodes: This operator selects 2 non-input nodes from the new network and then mutates all their incoming weights. The new value is calculated in the same way as in the case of the biased-mutate-weights. ˆ Mutate-weakest-nodes: In this case, instead of 2 random nodes, the one selected to mutate is the weakest. The concept of weakness that is used is not disclosed in the paper. The only characteristic that is mentioned is that this number can be positive or negative. In the former case, the mutation of the incoming weights follows a biased procedure; while it is unbiased in the latter case. Finally, this paper introduces a list of parameters called operatorprobabilities. They represent the probability of selecting an operator. Furthermore, they are automatically updated during the course of a run in the algorithm. This property releases the designer from manually tuning these parameters.

51 CHAPTER 3. METHODS AND TOOLS Cultural Evolution The cultural evolution algorithms are intimately related with the cultural transmission in human communities. This concept is based in the notion that some aspects of our behavior are culturally transmitted rather than genetically inherited. The cultural transmission allows the community to acquire experiences from their ancestors. These experiences can be further expanded with the addition of their new experiences, which, afterwards, they can transmit to their descendants in a continuous cycle. Abundant research has been done in cultural evolution applied to neural networks, which can be verified by the large number of articles on this subject [2, 6, 14, 42, 40]. In this subsection, focus is on the work of Denaro and Parisi [12], whose article on this subject presents the algorithm that we adapt in our solution. Their algorithm is based on the imitation learning concept, whose basic diagram can be seen in the Fig There are two actors in this process: the teacher, who possess a certain knowledge, and the learner, whom we want to imitate the teacher s behavior. To achieve this goal, we present the input vector of an example to both ANN. Yet, because both networks possess different weights, their outputs are different. This allows us to modify the weights of the learner using the backpropagation algorithm 3. We use the output of the teacher as the desired output, thus, making the learner a bit more similar to the teacher. If this process is repeated with a sufficient large set of examples and during an adequate number of epochs (i.e., until conversion, or any other stopping criterion), the learner will present a similar behavior to that of the teacher. A key element is that the learner is similar rather than equal to the teacher, due to some factors such as the learning process or the random nature of the initialization process. This feature is important when, as in our case, we pursuit to obtain diversity, which is basic to evolutionary algorithms. Using this process, the complete algorithm has the following form: The first population is randomly generated and they can follow or not a process of backpropagation to learn a sub-optimal solution, whose examples can be constructed, for example, using a heuristic. Selection of the teachers (rather than parents) from those with the best fitness value. This process was agamic in the Denaro and Parisi paper, but it is straightforward to adapt it to use two teachers. 3 This algorithm was explained in Section on page 31

52 CHAPTER 3. METHODS AND TOOLS 40 Figure 3.3: Imitation learning with 2 neural networks: The left one is the teacher and the one in the right is the learner. The mutation is developed with the addition to random noise to the teacher s output. This noise corresponds with a normal distribution with mean of 0 and a range [ 0.5/+0.5] Language Emergence The appearance of the language is an essential element in the cultural evolution of the humanity. The communication makes the connected individuals extremely more powerful than the isolated ones, helping the community to share their knowledge about avoiding problems and taking advantage of opportunities. Additionally, some researchers indicate that the cultural evolution of communication is possible in a population of neural networks [46]. Therefore, during this thesis, we carry out some experiments on evolving communication. The aim was to provide a useful tool to the neural networks that represent our individuals, so they can share information and their intentions; e.g., if they are going to change the light or continue with the same color. Cangelosi and Parisi explain in their paper an algorithm for emerging language in an evolving population of neural networks [8]. We adapted this algorithm to suit our requirements. A brief description of their algorithm is depicted below: The number of inputs and outputs increments by, for example, three units. The extra outputs of one ANN are connected to the extra inputs of others ANNs. These extra connections allow the creation of words of three bits, or equivalent, a language of eight words.

53 CHAPTER 3. METHODS AND TOOLS 41 The initial weights follows an uniform distribution between [ 1, 1] 20% of the parents are selected for reproduction. Each of the eligible parents generates 5 new individuals which are identical copies prior to the mutation. The mutation consist in changing 10% of the weights with the addition of a new value from the initial distribution. The evolution of language cover two different but related processes, namely, the emission and the comprehension of the message. An individual needs both abilities to attain successful results. This circumstance culminates in a theoretical drawback for this approach, due to the fact that the evolutionary pressure can only justify an evolution in the understanding of the language, but not in the emission of the message, at least in a direct manner. Nevertheless, the evolution in the production of messages can be seen as a necessary condition, resulting in an indirect pressure that provokes its evolution in accordance with the understanding of the same language. This argument solves partially the theoretical problem and is supported by the experiments realized by Cangelosi and Parisi Simulation Simulation is the process of studying the behavior of a system by observing the behavior of a model that represents the system. From this definition, it is possible to extract the two major parts in any simulation: the real system and the one that imitates it. There are several situations in which is advantageous to simulate a real system, some examples are the following. First, when it does not exist a mathematical formulation, or any analytical method of resolution, or they are too complex. Second, when the real system is prior to construction and it is necessary to experiment to see its results. A third situation is when the conditions we want to simulate occur rarely in the real system, or we want to test extreme situations in order to find the boundaries where the real system can work safely. Finally, in any situation in which it is not possible to experiment with the real system but it is interesting to control some variables related with it. Furthermore, due to its nature, the simulation can frequently run faster than the real system, and even several simulations can be executed in parallel, thus, it allows to carry out a large number of trials in the same time than a single test in the real system.

54 CHAPTER 3. METHODS AND TOOLS 42 On the other hand, there are also some drawbacks regarding simulation. Although desirable, it is not feasible to imitate all the variables present in the real system, due to some inherent limitations and because the presence of unimportant details can lead to undesirable noise. As a consequence, the imitation can never be exactly equal than the real system. In addition, it is frequent that the system is intrinsically stochastic, forcing the simulation to represent some variables using probabilistic distributions. These distributions use random numbers, which necessarily conducts to differences between the output of the simulation and the real results. In a conclusion, simulations permit to study a system in many aspects with small cost and in situations when it is not possible to use the real system, but it is important to know its limitations, and assume its inaccuracies and errors Characteristics State of the system The state of the system is the value of all the variables regarding the system and the objects that participate in the simulation, in any given time. It is important because it completely defines the system, allowing to evaluate the system and continue/start the simulation at any given point or situation. Classification Continuous: The continuous time simulations adjust better to reality. However, they are usually constructed using differential equations, which makes them useful only for small scale systems. They are frequently used in physics and chemistry, when, for example, it is required to monitor changes in temperature or in pressure over time. Discrete: This kind of simulations describe the system every time step, assuming that no major changes occur during that time lapse. ˆ Fixed increments of time: In this subtype, the increments of time, or ticks, are invariably of the same length. This increments are usually fine grained in order to avoid losing any important event between two ticks, which might lead to inaccuracies in the results. On the other hand, an increment in resolution leads to a decrement of performance, or increment of running time. Consequently, it is important to find a compromise solution between the two extremes. This subtype of simulators is the most popular

55 CHAPTER 3. METHODS AND TOOLS 43 and Netlogo, the simulation engine we use in this thesis, is a good example of it. ˆ Events related increments: The simulation is updated every time the simulator calculates that a new event happens. This makes the simulation more efficient than in the previous case, but it is more complex, which makes it impractical when the number of agents is very high and they are constantly generating events. ˆ Client related increments: In this simulation, the simulator keeps track of every agent, and models their logic sequence of events. It is not very efficient, but it is easily implemented in a spreadsheet, which makes it ideal for small applications. It is used mostly in queue simulation or to calculate the ideal ciphers in overbooking applications, where every agent corresponds with a client Software tools for modeling and simulation To carried out the experiments in the present work, we have developed a computer program that implements concepts described in the previous sections. This computer program or simply system consists on several parts linked together by means of a common programming language. This section outlines the main features of the third party tools used (Matlab and NetLogo), the programming language used to develop the system (Java 5.0), and other software used to synchronize all the parts. The system operation is described in the next chapter Programing language: Java 5.0 Java is an object-oriented high-level programming language. Its multiple advantages influence our criterion to use it to implement our system. A number of its features are the following: Due to its present relevance, there are a wide range of tools developed using the same technology, which grants their direct interoperability. A good example of these tools are those used in our system and that are explained in the next subsections. It is compiled to bytecode, which increases the performance with respect to the interpreted code.

56 CHAPTER 3. METHODS AND TOOLS 44 The compiled program runs on the Java Virtual Machine (JVM), which allows the program to be independent of the computer architecture. This is very useful for our system, because the same program runs on the server (Windows 7) and on the triton nodes (Unix), which minimizes the development time. On the other hand, the JVM is adapted to the architecture to take advantage of its characteristics, with the consequent rising in performance. It is concurrent, which means that several threads can run at the same time, and thus seizing the presence of muti-core and hyper-threading technology. This feature is very important due to the nature of the task, which creates the need of running several simulations at the same time, in order to minimize the execution time. It is also object oriented and class based. This feature helps in the integration of abstract concepts in the system on account of the direct correspondence between these concepts and the objects in the computer program Neural Networks simulation and training: Neural Network Toolbox in Matlab Matlab is a computing environment dedicated to numerical computation with its own programming language [50]. It is specially designed to work with vectors and matrices, creating a complete and efficient environment for mathematical purposes. In addition, it possess several toolbox that help in constructing robust solutions very easily. One of its toolboxes is the Neural Network Toolbox, which we use to model multilayer perceptron. This toolbox needs a set of examples and the number and sizes of the hidden layers. It uses this information to create the network and its connection and to extract the maximum and minimum values of each input and output attribute. This data is necessary to normalize and denormalize the input and the output values respectively, being also useful to eliminate the attributes with only one value. Nevertheless, it creates the need of a set of examples that is not always present and the presence of outliers generate problems regarding the range of the attributes. Following the creation, the initialization is performed, which consists on the assignation of random numbers to the weights and biases. However, the initialization function can be changed and customize, which was used to perform some of the experiments. After that, the network can be trained using a great number of different training algorithms, including the basic backpropagation algorithm, which is the one

57 CHAPTER 3. METHODS AND TOOLS 45 we used when necessary. Finally, the network can be simulated, which means that a matrix of input vectors is presented to the network and the network calculates the output. The transfer function used between the hidden layers is the tangent sigmoid, presented in Eq Despite the presence of this tool, the complete process of network simulation was implemented in the main program to both, increase the performance (avoiding unnecessary calls to Matlab) and to run the simulation in the nodes without the necessity of using Matlab. 2 f (x) = 1 (3.8) 1 + e ( 2x) Additionally, the evolution process is also carried out in Matlab: given the fitness values calculated with the simulation, it applies the evolutionary operators to the population of ANN to create the offspring Social simulator: NetLogo NetLogo is a multi-agent programming language and integrated modeling environment [57]. It is programed in Java, allowing a native interoperability and a complete control from a Java program. It comes with a large number of models in its model library, allowing a quick start and possible expansions of those models. It is also tick based, which makes it suitable to model complex systems with multiple (thousands) agents. A simulation in NetLogo is divided into two 4 parts: the interface and the procedures. The interface is divided into the 2D/3D view of the world, the command center, which allows to interact with the system directly using a command line-like environment, and the controls, that are visual input and output element, such as buttons, switches and monitors. The 2D/3D view permits to follow the simulation at the same time that the command center and the controls allows to communicate with the system. The procedures are created using a special language similar to Logo and that is fully programmable, simple and oriented to the use with agents. Additionally, it has a large vocabulary of built-in primitives, which favors the rapid creation of the simulations Other software Apart from the three main tools already explained, there are other pieces of software that were used during the development of the system. Their 4 three if the information about the model is counted

58 CHAPTER 3. METHODS AND TOOLS 46 characteristics are briefly explained here, as well as the role they play in the solution. Ssh connection: SSHTools SSHTools is a suite of Java SSH applications providing a Java SSH API, SSH Terminal, SSH secured VNC client, SFTP client and SSH Daemon. Is distributed under GNU General Public License (GPL) which means that it can be used freely if its distributed under the same license. We use the SSH protocol to connect with the triton server, which is the responsible to distribute the workload among nodes that carry out the simulations. Additionally, two SFTP connections are necessary: one to send the files with the encoding of the neural networks and the other to receive the results of the simulations. JMatLink JMatLink connects Java and MATLAB using native methods, which make possible and efficiently the use of Matlabs s computational engine inside the java application.

59 Chapter 4 Development 4.1. Workbench: Overall architecture The system is divide into two main blocks: the evolutionary part and the simulation part. The evolutionary part evolves a population of neural networks using the evolutionary strategies described in Section 3.2. On the other hand, the simulation tests the performance of a neural network in controlling the city lights. The result of the simulation is then used in the evolutionary part as the fitness value that determines the evolution pressure. This division is the result of the high demand of resources by the simulation, making it impractical to be executed in a single computer. When testing the individuals, the simulations are independent, which makes viable the parallelization of the process. This parallelization is very convenient when the population is large (e.g. 50) as in our case. To parallelize the simulations, we used a cluster from the Aalto University called triton. Therefore, these simulations run in different nodes of the cluster, taking advantage of the multiple cores that these machines possess as well as the memory and network resources. The communication with the nodes is made through the front host which then uses Slurm to equitably distribute the work load between the nodes UML Class Diagram The class diagram of the system is summarized in Fig. 4.1 (evolutionary part) and Fig. 4.2 (simulation). The essential characteristics of these classes are listed as follows: Main: Is the class that executes the principal program. It maintains two variables: Population, that has the state of the evolution, and the 47

60 CHAPTER 4. DEVELOPMENT 48 fitness, that has the results of the individual along the evolution. Population: It maintains all the variables to carry out the evolution and communicates with the cluster to calculate the fitness and with Matlab to both, generate the offspring and to create the files with the encoding of the networks. JMatLink: It is the class that communicates with Matlab using native methods. It offers several primitives to run commands and to exchange data with Matlab. SshClient: It is the main class to communicate with the cluster, and implements the primitives to connect and authenticate with the front host. SessionChannelClient: It is used to send the commands to the cluster in order to perform the simulations. SftpClient: This class is used to send the encoded networks to the cluster and to receive the fitness values afterwards. ExternalSimulation: It is the class that executes the program in each node. First, it reads the file of the network to pass it to the City, so it can perform the simulation. When the simulation finishes, this class also writes the results in a file. City: This class and its variables represent the simulated city. It is an abstract class, and therefore, different classes extend it, regarding the simulated city. The CityNetLogo is used for the simple and the four directions city, the CityNetLogoFull represents the four directions and turns one, and the CellSim is used with the cellular automaton simulator. HeadlessWorkspace: This class is the link with NetLogo. In the same manner as the JMatLink, it implements different functions to execute commands and to retrieve information from the simulator. CellSim: This class implements a simulator based on a cellular automaton. It is used during the first half of the generations due to its great performance, although is not as accurate as the simulation performed with NetLogo, which is used in the last part of the simulation to fine-tune the evolution.

61 CHAPTER 4. DEVELOPMENT 49 NeuralNetwork: This class mimics the behavior of the multilayer perceptron from the Matlab neural network toolbox. Consequently, it calculates the response of the network to the current inputs, which are the sensors, and if there is communication among traffic lights, the output from other networks. Plan: This class stores the desired variations in the traffic flow along the simulation. It is used to change the state of the traffic and thus test the solution in different situations. Each variation, encoded in a PlanSteep, contains the mean time of appearance of a new car from an incoming street, and the time left before the next change in this value. TrafficLight: This class represents the traffic lights. It stores all the variables necessary to describe the state of the light. Due to the clear differences between the scenarios, this class is an abstraction of TrafficLightSimple and the TrafficLightFull class, which, respectively, are used in the simple and four directions scenarios, and in the full directions one. Sensors: This class stores the value of the sensors, which measure, for example, the presence of stopped cars after the light or the number of cars approaching the green light. In the same fashion as the previous one, this class is also implemented by the SensorsSimple and the SensorsFull classes Example of an experiment All the experiments share the same basic structure in their execution. The sequence diagram describing its operation is presented in Fig. 4.3 and Fig. 4.4, separated again into the same parts. The evolution begins by creating the initial population. Matlab, through the JMatLink engine in the diagram, initializes this population in a different manner depending on the experiment. In the next step, the cluster nodes calculate the fitness value of a neural network by running a simulation where that neural network controls the traffic lights of the city. Matlab encodes the networks into files 1 that are then sent to the nodes using SFTP. When the simulation is finished, the fitness value is written in another file, which is retrieved using the SFTP again. Finally, Matlab creates the offspring using the fitness values and the evolutionary strategies, which are 1 An example of this files and the explanation of its structure can be shown in Appendix 4.2.1

62 CHAPTER 4. DEVELOPMENT 50 Figure 4.1: UML Diagram of classes in the main program.

63 CHAPTER 4. DEVELOPMENT 51 Figure 4.2: UML Diagram of classes in the node

64 CHAPTER 4. DEVELOPMENT 52 Figure 4.3: Sequence diagram of an experiment, details of the sequence diagram of the simulation are presented in Fig. 4.4 subject to the experiment. The generation of new offspring continues until the new generations do not improve their performance with respect to their predecessor, e.g., 50 times. The first step in the simulation corresponds with the decoding of the file containing the neural network, which is used to create the network object that controls the semaphores. Next, the city stores that network and initializes its variables, to be prepared for the simulation itself. The simulation consists on a set of actions that are repeated during a number of iterations or ticks, e.g times. On every tick, the plan is verified and, if necessary, the value of the mean is updated in the exponential distribution that controls the appearance of a new cars in the city. Following, NetLogo (through the worspace object) collects the values from the sensors in the simulated city. These values refresh the state of the lights, who generate the input matrix

65 CHAPTER 4. DEVELOPMENT 53 of the network. The lights use the calculated output of the network to make their decision about the new state of the traffic lights. Each individual decision is compiled together and sent to the simulator. Finally, the simulator recreates the consequences of these decisions in the simulated scenario, which is explained more carefully in the next section. When the last tick finishes its execution, NetLogo calculates the fitness value, which is written in its file Scenarios We have developed three scenarios to test the flexibility of the techniques used in this work. The three scenarios were recreated in NetLogo, with the exception of the simple city that was also implemented using a cellular automata simulator, to execute the experiments in a shorten time. These scenarios adopt essentially the same communication protocol with the rest of the system and follow a similar procedure to simulate the city. When the initialization command is received, the simulator sets its variables, creates the roads, prepares the intersections and the traffic lights, and finally, places the initial cars in their positions. Next, in every step of the simulation, the simulator counts the number of cars in different regions to simulate the sensors placed near the traffic signals, and sends the data to the main process, so it can make the decision. When the decision is received, it is forwarded to the lights and, or either they change to red, green or amber, or either they continue with the same color, depending on their actual state. After the city lights update their colors, the simulator kills the cars that have reach their destination at the end of the street, and also creates new cars in the incoming streets. The appearance of new cars follows an exponential distribution, whose mean is updated by the main process in order to control the density of the cars in the city, and thus creating a wide range of situations. Finally, the cars move forward according with their speed, which is revised to see if they should accelerate or decelerate depending on the presence or absence of cars in front of them, and the color of the traffic light if they are near to one. The process followed to calculate the acceleration/deceleration of the cars differs in each scenario, and hence it is explained in more detail in the following sections. The presence of pedestrians was intentionally avoided in the simulation. We consider that they can be integrated in the traffic using the Barnes dance, which has been implemented in many cities [54]. It consists on changing every light controlling traffic into red at the same time, so the pedestrians can cross the street at the same time, going in every direction, even in diagonal. Therefore, the reason it is avoided is that, if this approach is adopted, it

66 CHAPTER 4. DEVELOPMENT 54 Figure 4.4: Sequence diagram of a simulation

67 CHAPTER 4. DEVELOPMENT 55 Measures Amber time a Car length Max. Speed Accel. b Decel. c Model 3 tic 1 pat 3.47 pat/tic 0.46 pat/tic 2 1 pat/tic 2 Real 3 s 4 m 49.9 Km/h 1.85 m/s 2 4 m/s 2 a Security time between the green and the red light states. b Acceleration c Deceleration Table 4.1: Principals measures in the simulator and their equivalence in real life affects any solution in the same manner, and consequently, it should not influence in the fitness value. However, the presence of sensors to detect pedestrian waiting and other restrictions, such as the preference to some special vehicles, such as buses or taxis, are discussed in the next chapter. These scenarios were developed by expanding a previous model included in the NetLogo models library called Traffic Grid. Numerous improvements were performed on this model, including incrementation of the maximum velocity, inclusion of acceleration/deceleration, creation/disappearance of cars, enlargement of the streets and introduction of the amber phase. The amber light was included to avoid security problems (accidents) in our model in the same manner than in real traffic, we calculated than it needed to last for at least 3 s to solve the problem of the amber light [16]. Table 4.1 shows some of the common measures for all the scenarios and their equivalence in real units Simple city The design of this model intended to be simple, because it was used to determine the feasibility of our experiments. Consequently, the cars in this city only move in two directions: from the upper part to the bottom (North- South direction) and from the left to the right (West-East direction). There are five streets in each direction, flowing through the entire length of the model. Every time two perpendicular streets intersect there is a traffic light, so consequently there are twenty five semaphores in the city. The height and width are of the same size in the scenario, and the streets are equidistant from each other, as a result, the length of the streets between two intersections is the same, being more concretely 21 patch, or equivalently, 84 m. In Fig. 4.5a there is a sample of this city. The technique used to calculate the speed of the cars loosely resembles the Eq. 2.6 from the car-following theory, but with the limitations of a dis-

68 CHAPTER 4. DEVELOPMENT 56 (a) Simple city (b) Simple city implemented using cellular automata. (c) City with four directions (d) City with four directions and turns. Figure 4.5: Example of the 3 scenarios and cellular automata simulator while been controlled using the marching technique.

69 CHAPTER 4. DEVELOPMENT 57 crete time simulation with a resolution of 1 s. First, the stopping distance is calculated in case the car decided to accelerate as the result of this process. The reason for this is that, it corresponds with the maximum distance the car would travel if it were decided to decelerate in the next tick. Second, if there is a red light in that distance, or an amber light that is close to change to red, the car reduces its velocity. In any other case the next step is to check if there are cars in that distance. If there is a car, and it is in a different direction, the decision is to decelerate to avoid an accident. If there is a car, but it is in the same direction, the speed is accommodated to follow it. Finally, if there is no other car in that distance, the decision is to accelerate. Input and output vectors The seven principal inputs are based on Fig. 2.7: Number of cars approaching green light in region d, number of cars approaching green light in region r, number of cars stopped in region e after green light, number of cars approaching red light in region r, number of cars stopped in region e after red light, cumulative number of cars approaching red light in region d since the last exchange of lights, and the number seconds since that exchange. When there is only one output, the decision of the network is interpreted as continue with the same state if the output is 0, and exchange the colors of the traffic light if it is 1. In the latter case, first there is an immediate change from green to amber light in the green side, and finally, after the amber period, both lights change to their new state. In some experiments, the number of inputs increments up to ten. There are two differences between this vector and the seven inputs one: the addition of the Number of cars approaching red light in region d, and the rearrange of the values to make the vector insensitive to changes in the color of the light. This means that the streets are numbered and the variables in the input vector correspond with measurements from the same street, with independence of the color of the light controlling that street. This extended vector allows the appearance of a new state: both lights in red. Consequently, the output also changes from one variable to two: 00 means no change, 01 and 10 first light green and second light green respectively and 11 both lights in red. Additionally, the number of inputs and outputs also rise in the experiments of language development, following the process described in Cellular automaton simulator Running the simulation to test the solution is the most dilatory process during the experiments, despite the parallelization of its execution, which was

70 CHAPTER 4. DEVELOPMENT 58 explained in the previous section. Therefore, a faster simulator was needed, due to the number of experiments performed in this city, and the long time needed to complete each experiment, e.g. seven hours. This simulator was constructed using Elementary Cellular Automata (ECA), and following the model described in [19]. An ECA consist of an array of binary cells; that is, they can be either on (1, true) or off (0, false). The function that determines the state of the cell is called rule (not to be confused with the rules of the SOLA algorithm), and uses the current state of the cell and the two adjacent ones to calculate the new state. Consequently, the new state (0, 1) is calculated from 3 binary inputs, which leads to 2 3 = 8 possible inputs, resulting in a maximum of 2 8 = 256 possible rules. Therefore, these rules are typically named with a number that goes from 0 to 255. To understand the naming criterion, imagine the rule is decoded in binary into , and similarly, the eight first numbers in in binary are These eight numbers correspond with the current state of the three cells, and hence, the new state is calculated by obtaining the bit in the same position when decoding the number of the rule. As an example, if the left neighbor state is 1, the cell state is 0 and the right neighbor state is 0 (100 2 = 5 10 ) the new state is 1 because there is a 1 in the fifth position in the rule 184 binary code. In this simulator, every street is a cellular automaton, and every cell represents the presence (on) or absence (off ) of a car in that position. When there is no intersection (or the green light is on), the rule applied in that street is the 184, which represents the west-east movement of the cars: in each step the cars move one position to the right if that position is free. In order to represent the intersections, and therefore, the traffic lights, two more rules are needed: number 252 and 136. The diagram of these rules, applied to an intersection, is shown in Fig Rule 252 represents the same movement as the184, but it does not allow the car to move to the right position even if it is free, thus simulating the red light. Similarly, the rule 136 also represents the same movement, but it does not consider the car in the left (i.e., 0XY and 1XY states obtain the same new state), because it considers that it belongs to the other street. The Table 2.3 lists the transition tables the three rules used in this simulator. The final result of the simulator is shown in Fig. 4.5b. This simulator exhibits some advantages, being the most important its great performance due to the simple calculations needed to simulate the cars. On the other hand, as a consequence of its lack of realism, the simulator reveals several disadvantages on using it, such as the absence of deceleration/acceleration (i.e., the car velocity is either 0 or 1 patch per tick), the constant space occupied by the cars and the inconsideration of the amber light. Consequently, we decided to combine the use of this simulator with

71 CHAPTER 4. DEVELOPMENT 59 t 1 t 184 t 252 t Table 4.2: Rules of the cellular automaton that simulates the simple city the one from NetLogo. Hence, the twenty five first generations evolve using the cellular automaton simulator to calculate their fitness and the rest of generations use the NetLogo one to obtain the most suitable solution City with four directions This city was developed to test if it was possible that the candidate technique 2 adapts to a different environment. We consider that it was necessary, because the greenwave has a good performance in a city with only two directions, but has some difficulties if there are streets in four directions. However, the presence of 2 more directions is not the only difference with the simple city. Derived from this circumstance, there are 4 different types of semaphores, one for each combination of the two horizontal streets with the two vertical ones. Moreover, the speed of the car is also calculated in the same fashion. Finally, this city does not implement the cellular automata simulator, because it was not necessary, as there is only one experiment developed for this city. Fig 4.5c presents a portion of this city City with four directions and turns The aim of this city was to create an scenario more realistic, where the cars can turn left and right in the intersections. As in the case of the previous scenario, it was constructed to test the performance of the candidate technique in a more realistic environment, with intersections that permit the convergence of four directions. In order to represent the four directions and the possibility of turning in each intersection, it was necessary to use twelve lights per intersection. However, this creates an important problem: Not all 2 The candidate technique is explained in the next chapter

72 CHAPTER 4. DEVELOPMENT 60 Figure 4.6: Rules applied in an intersection regulated by a traffic light. the combination of green are red lights are compatible, as it will create traffic security problems. The complete list of this combinations is presented in Appendix A. We use this list to control the lights: the number of outputs of the network is 12 (plus the addition of 3 more to emerge a language) binary values, each of them representing a light (0 green light, 1 red light). If the combination of lights is not in the list, the algorithm search in the list that one that is more similar, and automatically it becomes the decision for the new state. To simulate the turns, when a car is created, an output street is assign to it. In that manner, when a car reaches an intersection, it decides its turn to approximate to that street. As a result, the speed of the cars is calculated similarly than in the previous cities, but with a small difference: once the distance to stop is calculated, if the car is going to turn, not only the street where the car stay is checked, but also the new street. As a final note, for this city we cannot use the SOLA algorithm, so we developed a similar self organizing algorithm, also based in rules, but adapted to this type of intersection Fitness Function The selected fitness function corresponds with a measure of performance of the city as a transportation system. For the main fitness function, we

73 CHAPTER 4. DEVELOPMENT 61 decided to use the average of the average waiting time of a car in every tick through the entire simulation. The equation of this function is the following: F = T j=0 Cj i=0 w i,j C j (4.1) T Where T is the number of ticks that the simulation last, C j is the number of cars during the time instant j, and w ij is the time that has passed since the last time the car i moved, in the time instant j. This fitness function rewards the mobility of the cars, even if the car stops again soon, but penalizes dramatically when a car stops for a long time; for example, if a car stops during 30 seconds, its w value is going to be w = 1, 2, 3,..., 30, making the value of the fitness increase rapidly (following the Fibonacci s function), until the car moves again. The other function that we used in some experiments was the average of the average of the speeds of the car in every tick through the entire simulation. We use it to check if the evolutionary strategies could improve the performance of the system with a different function, which also changes the direction of the improvement: with the previous function, the best solution is the one with smaller fitness value, while in this case the best is the one with higher value. The function is the following, where all the variables have the same meaning as in the previous one, except for the v ij that means the velocity of the car i in the time instant j. F = T j=0 Cj i=0 v i,j C j T (4.2) Plan The simulation of the city lasts 2 h or 7200 s. During that period, the density of the cars in the city is controlled changing the mean of the exponential distribution that administers the appearance of new cars. Fig. 4.7 shows the diagram representing this plan, with the inverse of this value: the average of appearance of a new car. When we design this plan, we intended to represent a typical morning with a rush hour and another smaller peak one hour later, thus testing the solutions on intending to solve the first jam before the second peak.

74 CHAPTER 4. DEVELOPMENT 62 Figure 4.7: Representation of the plan that controls the density of cars in the city. The average of new cars is the inverse of the mean time of a new car. The reference curve is the density or cars when there is no streets crossing at the same level, and thus, cars can move freely without the impediment of traffic lights. The Genetic curve represents the density of cars during the simulation, when the lights are controlled by the best individual obtained using the biological evolution. Similarly, the Sola curve is obtained when the lights are controlled by the self organizing algorithm, and the Marching when using the marching system to control the lights.

75 Chapter 5 Results In this section we explain the experiments we carried out in this thesis, their results and the results of other techniques from Section 2.6: state of the art. Additionally, we use the implementation of the SOLA algorithm to generate the examples that Matlab needs to create the networks and to train them in some experiments, including those from Cultural Evolution Simple city We use the simple city scenario to test the feasibility of the experiments. We perform multiple experiments using it to find the candidate technique: The evolutionary strategy that more likely generates the individual with better performance in solving the problem described in Section 1.1. Afterwards, we use the candidate technique to perform the experiments in the other scenarios Biological evolution This experiments follow the algorithm described in Subsection3.2.2 with some modifications: Our problem cannot be solved with supervised training, which means that the fitness function can not be obtained from the error of a set of examples. Instead, we use the simulation to obtain the fitness value. The selection of the parents is identical in all the experiments to make easier the comparison between techniques. The selection of the parents is deterministic: 20% of the parents are selected to reproduction: those with better fitness value. Each combination of these parents forms a 63

76 CHAPTER 5. RESULTS 64 Experiment Structure Peculiarities Result ConceptProof Very sort plan ConceptProof Very sort plan NormalInitialization Sorter plan SpecialInitialization Sorter plan MediumNetwork Sorter plan LargeNetwork Sorter plan SmallNetwork Sorter plan Candidate Best result Candidate AlternativeFitness Alternative Fitness Function Marching - Lights change every 15 s GreenWave - Lights change every 15 s SOLA Table 5.1: Summary of the experiments realized with the simple city. couple that mates to create one or two new individuals to complete the new generation. We do not implement the operator-probabilities. Thus we use always the same operators: the Crossover-weights for the crossover, and the Biased-mutate-weights for the mutation. Table 5.1 presents a summary of the principal experiments performed with the simple city using this technique. ConceptProof1 and ConceptProof2 were brief experiments carried out to check if this algorithm was adequate to solve this problem. NormalInitialization and SpecialInitialization differ in the initialization function: the NormalInitialization uses uniform distribution for the initial weights, and SpecialInitialization uses a symmetrical exponential distribution. MediumNetwork, LargeNetwork and SmallNetwork were experiments created to check the effect of changing the size of the networks. Finally, Candidate1 and Candidate2 were the last two experiments, again, with different size of networks, to obtain the final results of this technique. The AlternativeFitness was developed using the alternative fitness function from Eq The layers have the same number of neurons than in the experiments of next subsection to compare their results (7 3 1).

77 CHAPTER 5. RESULTS 65 (a) Best, average and worst individual in(b) Zoom in to appreciate the details of each generation. the evolution of the best individual and the results of the others approaches. Figure 5.1: Results of the evolution in the simple city. The upper line is the greenwave result, the middle line is the marching result and the lower line is the SOLA result. The curves represent the worst, the mean and the best individual in each generation. The minimum result of the evolution was obtained in the 48 generation. b represents a zoom to see the details of the evolution of the best individual Cultural evolution In this subsection we present the results of the experiments regarding cultural evolution. These experiments follow the same process explained in the work of Denaro and Parisi [12], in order to check, step by step, the validity of the algorithm. All the experiments use the same network structure (7 3 1), and uses the alternative fitness function to measure its performance. Please, note that, using this function, the algorithm maximizes, rather than minimizes, the fitness value. Direct imitation In this experiment we trained the first generation with the examples set obtained from the execution of the SOLA algorithm. After that, during 50 generations, every individual of the population taught its behavior to a new individual, using imitating learning. The best fitness value obtained applying this algorithm is: Fig. 5.2a presents the results of this algorithm through the generations. An important detail to notice is that this value is obtained in the generation

78 CHAPTER 5. RESULTS 66 (a) The curves represent the best, the(b) Scatter plot with the error with the mean and the worst individual in eachsola algorithm examples and the Av- speed of cars generation. The maximum result of theerage evolution was obtained in the 18 generation. Figure 5.2: Results of the direct imitation algorithm from the cultural evolution tested in the simple city. number 18, despite the continuous decrement of the performs with the secession of generations. We think that this situation is a consequence of the randomness introduces by the learning algorithm, which can generate behaviors that were not present in the original population. Fig. 5.2b validates this hypothesis, due to the presence of points over the black line. In this figure, each point represents an individual, the x position its percentage of the error in the SOLA algorithm examples set, and the y position is the fitness value obtained by the individual. Consequently, if we consider the SOLA algorithm a sub-optimal solution, to obtain good results the degree of similarity with it should be high, i.e., low error. However, the points above the line represents individuals that do not follow this assumption, because they obtain better values of fitness than expected due to their similarity with SOLA. Selection In this experiment the firs generation is also trained to imitate the SOLA algorithm. However, instead of using all the individuals to train the new generation, we select the parents using the method described for the biological evolution. Additionally, to use two parents in the learning process, we simulate half of the examples with one parent and the other half with the other parent. The result of this experiment is: