Interpretable Fuzzy Modeling using Multi-Objective Immune- Inspired Optimization Algorithms

Transcription

1 Interpretable Fuzzy Modeling using Multi-Objetive Immune- Inspired Optimization Algorithms Jun Chen, Mahdi Mahfouf Abstrat In this paper, an immune inspired multi-objetive fuzzy modeling (IMOFM) mehanism is proposed speifially for high-dimensional regression problems. For suh problems, high preditive auray is often the paramount requirement. With suh a requirement in mind, however, one should also put onsiderable efforts in making the eliited model as interpretable as possible, whih leads to a diffiult optimization problem. The proposed modeling approah adopts a multistage modeling proedure and a variable length oding sheme to aount for the enlarged searh spae due to the simultaneous optimization of the rule-base struture and its assoiated parameters. IMOFM an aount for both Singleton and Mamdani Fuzzy Rule-Based Systems (FRBS) due to the arefully hosen output membership funtions, the inferene and the defuzzifiation methods. The proposed algorithm has been ompared with other representatives using a simple benhmark problem, and has also been applied to a highdimensional problem whih models mehanial properties of hot rolled steels. Results onfirm that IMOFM an eliit aurate and yet transparent FRBSs from quantitative data. T I. INTRODUCTION RADITIONALLY, modeling tasks involve the building of mathematial equations whih an best desribe the underlying proess. Suh a modeling pratie normally requires a deep understanding of the systems under investigation, hene the reason why it is often referred to as knowledge-driven Modeling. On the ontrary, Data-Driven Modeling (DDM), inspired prinipally from artifiial intelligene tehniques, is based on limited knowledge of the modeling proess and relies on the data desribing the input and output mapping. DDM is able to make abstration and generalizations of the proess and plays often a omplementary role to knowledge-based models. For omplex systems, the linear regression may not be suffiient, whih leads to the need of the non-linear regression tehniques. Among many of these tehniques, Artifiial Neural Networks (ANN), fuzzy rule-based systems (FRBS) and Neural-Fuzzy Systems (NFS) have been reeiving more attention during the last two deades due to the fats of not only being able to approximate pratially any given funtion to an arbitrary auray [1], but also being able to Manusript reeived January 1, 010. This work was supported in part by the EPSRC under Grant EP/F0464/1. Jun Chen is with the Department of Automati Control and Systems Engineering, University of Sheffield, Sheffield, S1 JD, UK (phone: +44 (0) ; [email protected]). Mahdi Mahfouf is with the Department of Automati Control and Systems Engineering, University of Sheffield, Sheffield, S1 JD, UK (phone: +44 (0) ; fax: +44 (0) ; [email protected]). generalize reasonably well for any previously unseen situations. The prevalene of these nonlinear regression tehniques is largely attributed to the breakthrough in the nonlinear optimization tehniques, suh as the Bak-Error Propagation (BEP) algorithm [, p. 46-5] and the bioinspired optimization []-[4]. Sine the first introdution of fuzzy logi, FRBSs have been widely used in systems and ontrol engineering []. However, the predominant approah in the traditional design of FRBS highly relies on human experts. Although learning omponents an be further inorporated into the proedures of oarse model induement [5] and its further refinement [, p. 46-5], it may suffer from two serious problems, viz. the deterioration of the model s interpretability and the over-fitting to the training patterns. Taking this into aount, one an find that bio-inspired optimization, in partiular Generi Algorithms (GAs), has a long history of being inorporated into fuzzy logi [6] and demonstrate a possible route to the remedy for the mentioned two problems. The main aim of this paper is to present a systemati immune-inspired multi-objetive fuzzy modeling approah whih an simultaneously aount for the interpretability of the rule-base and its preditive auray for regression problems. The paper is organized as follows: Setion II disusses the formation of the multi-objetive fuzzy modeling problems and the FRBSs used in this work; Setion III shortly reviews the existing evolutionary based approahes for improving FRBS s interpretability; IMOFM whih represents an alternative tati to solve interpretability issues will be introdued in Setion IV; in Setion V, in order to evaluate the performanes of IMOFM, the algorithm is tested using a simple benhmark problem and is applied to the predition of the mehanial properties of alloy steels; finally, onlusions are given in Setion VI. II. FUZZY RULE BASED SYSTEMS AND THE FORMATION OF THE MULTI-OBJECTIVE FUZZY MODELING PROBLEM A. Fuzzy Rule Based Systems (FRBS) Two fuzzy modeling paradigms, viz. Singleton [7] and Mamdani [8] FRBSs, are employed in this work due to their abilities of expressing linguisti meanings in both of their anteedents and onsequents. Generally speaking, a FRBS an be formulated as follows: where, is the ith linguisti value (fuzzy set) for the jth

2 linguisti variable defined over the universe of disourse ; the funtion assoiated with that maps to [ ] is the orresponding membership funtion; R i represents the ith rule in the rule base, and is the output of the ith rule. Typially, an be the funtion of the inputs or the linguisti value of the output, whih differentiate FRBS into Singleton (the former) and Mamdani (the latter) FRBS. It is worth noting that Singleton FRBS is a speial ase of TSK FRBS [7] when is the zero order funtion of the inputs. In some sense, Singleton FRBS shares the basi feature of Mamdani FRBS if one onsiders singleton onsequents as a speial type of fuzzy sets. B. Formation of the Multi-objetive Fuzzy Modeling Problem As Casillas et al. pointed out in [9], modeling is the task that simplifies a real system or omplex reality with the aim of easing its understanding. Hene, the development of reliable and omprehensible models must be the main theme of any modeling tasks. By reliable it is meant the model s apability of faithfully representing the real systems, in other words the model auray. By omprehensible it is meant the model s apability of expressing the behavior of the real systems in a omprehensible way, in other words the model interpretability. However, as Zadeh onjetured in his Priniple of Inompatibility [10], it is very likely that auray and interpretability may well be exlusive requirements in a modeling proess. The refletion of these in a fuzzy modeling senario represents a dilemma of designing FRBS. The auray vs. interpretability issue in a fuzzy modeling ontext an also be formulated as a multi-objetive optimization problem. Fig. 1 shows the front in a biobjetive fuzzy modeling senario where two ompeting objetives, viz. the preditive error (auray) and the rule base omplexity (interpretability), are minimized simultaneously. The aim is to find a set of approximate FRBSs as lose to the true front as possible. Fig. 1. front in a bi-objetive fuzzy modeling ase. By finding a set of solutions, human an understand the underlying problem in a muh greater depth, and finally a single optimal solution to a speifi senario is finally seleted and applied. In the ase shown in Fig. 1, if one requires ertain interpretability (transpareny) of the FRBS along with its good preditive auray the middle irle ould be the one that fulfils the user s need. As already stated in [11], this should result in a minimal human intervention during the modeling proess. III. LITERATURE REVIEW OF PREVIOUS WORKS Originated from Karr s work [6], the GA approah in fuzzy systems was initially utilized to adjust the parameters of membership funtions, whih leads to no signifiant differene when ompared to other learning paradigms. The real signifiane of employing evolutionary algorithms (EAs) for optimizing FRBSs omes from EAs flexibility in terms of being able to enode and evolve almost every omponent of the FRBS [1]. Suh a flexibility offers a solution so that one an take into aount the interpretability (struture) and the preditive performane of the FRBS in a more oherent way. Broadly speaking, there urrently exist two different EA-based streams to takle the interpretability issues: 1) The first stream is mainly onerned with the linguisti modeling using a Mamdani type of FRBSs, in whih a set of pre-speified fuzzy partitions (linguisti terms) are given a priori by experts or users (grid partition). These linguisti terms are fixed during the ourse of the evolution [1]-[15] so that their physial meanings are retained. Only the fuzzy rules are subjet to the seletion via GAs so that a ompat rule-base an be evolved from a large number of andidate rules, whih should lead to a more interpretable FRBS. Sine the seletion proess removes irrelevant and inonsistent rules, the auray is also improved. Further relevant researhes inlude those in [16]-[17], apart from the rule seletion, these works also tuned the linguisti terms by a modified GA. However, suh tuning is only operated in a restrained spae in order to maintain their original semantis. ) the seond stream generally uses an approximate fuzzy model as the starting point (in suh a ase, fuzzy partitions are extrated via some automati learning proedures; hene, there is not a global data-base a priori); the task is then to improve the model s explanatory ability, whih may have been lost during the automati learning proess, through a set of similarity-driven simplifiation and parameter adjusting operations [18]-[1]. Under this stream, a similarity measure is taken so that similar fuzzy sets an be merged. Consequently, similar rules are merged as well. Hene, the distinguishability of membership funtions and the ompatness of the rule-base are improved. Comparing the two streams leads to the following: 1) in the linguisti modeling stream, the target problems are normally assoiated with lassifiations and lowdimensional regression problems; this is beause that, for suh problems, the effet of the urse of dimensionality due to the grid partition and the need for the parameter tuning due to the preditive auray requirement are not serious issues; only very reently, suh a linguisti modeling framework has been adopted for high-dimensional

3 regression problems [17]; ) for high-dimensional regression problems, an approximate FRBS may represent a better hoie to start with due to the auray and ompatness requirement. However, to the best of our knowledge, majority of the works within the seond modeling stream were using TSK FRBS, whih breah the original intention of eliiting an interpretable FRBS. It is rather triky to deide whih modeling stream is more suitable. Both modeling streams have their limitations: 1) although linguisti modeling often leads to well distributed membership funtions more rules are required to ahieve similar preditive performanes as those provided by the seond modeling stream with fewer rules, this being due to the restrition imposed on the membership funtion searh spae; ) although the seond modeling stream often leads to a ompat rule-base and higher preditive auray, the membership funtions are not well distributed even after interpretability improvement; furthermore, if TSK FRBS is employed the transpareny in the onsequents will be lost. In the light of the above onsiderations, the proposed algorithm sits in the middle of the two modeling streams by using a ompat FRBS with ertain interpretability for highdimensional regression problems. Although a Singleton/Mamdani FRBS is used in this work, unlike those whih use similar types of FRBS within the first modeling stream, the membership funtions of the proposed method an move freely within the variable intervals. Hene, it is still within the seond modeling stream. However, it greatly improves the interpretability of the eliited FRBS, and an be viewed as a omplement to [17] due to the fat that more ompat and higher aurate FRBSs an be eliited. IV. AN IMMUNE INSPIRED MULTI-OBJECTIVE FUZZY MODELING (IMOFM) MECHANISM An immune inspired multi-objetive fuzzy modeling (IMOFM) proedure onsists of three stages whih follow priniples of vaination and the seondary response of the immune systems. Furthermore, the third modeling stage utilizes a Population Adaptive Immune Algorithm (PAIA) [], [] as the searh engine in searh of the optimal struture and parameters. In the following spae, Artifiial Immune Systems (AIS), in partiular PAIA, are first introdued followed by the desription of IMOFM. A. AIS for Multi-objetive Optimization The basi idea of using AIS for optimization is emanated from the Clonal Seletion Priniple and Network Hypothesis [4]. The Clonal Seletion Priniple desribes the basi features of an immune response to an antigeni stimulus, and establishes the idea that only those antibodies that reognize the antigen are seleted to proliferate. The analogies of this in AIS are omposed of the followings: 1) Ativation alulates the affinity (fitness) for eah antibody (solution) so that an adaptive number of lones an be seleted and produed; ) Affinity Maturation mutates the seleted good lones so that more searh spae an be explored; ) Reseletion selets good andidate solutions from the ombined parents and lones to provide seletion pressure to effetively drive the andidate solutions towards the front. The Network Hypothesis states that antibodies an be stimulated by reognizing other antibodies, and for the same reason an be suppressed by being reognized. Suh a suppression mehanism allows the regulation of the overstimulated antibodies to maintain a stable memory. The refletion of this in AIS is the so-alled Network Suppression whih is used to regulate the population so that it is adaptive to the searh proess. By synergizing all the above omponents, PAIA is proposed in [], []. In [4], the authors further proposed a multi-stage optimization proedure by inorporating the onept of vaination. The idea is to first use a single objetive optimization stage, ating as the vaination proess, to effiiently find any one of the solutions resting on the front. Then, PAIA an at as a post-proessing proedure to expand suh a solution along the front. For fuzzy modeling, small modifiations of the Ativation step are needed and will be disussed aordingly in Part B. Details regarding PAIA and multi-stage optimization are referred to [] and [4, h. ]. B. IMOFM for Obtaining a Set of FRBSs IMOFM is a three-stage modeling proedure. The first two stages are equivalent to the vaination proess in the first stage of the multi-stage optimization proedure. The aim is to first extrat an initial approximate FRBS and then to refine it in terms of its preditive auray. By doing so, an initial vaine model with the over-estimated number of rules an effiiently be eliited. Another reason of inluding the first two modeling stages, espeially the seond one (refinement), is that by doing so the most omplex rule-base an survive under the pressure of seletion. Without inluding the refining step, the rule-base with a omplex struture may be regarded inferior to the less omplex rulebase in a sense even if both them are inaurate in the early evolutionary stages. Hene, one may lose the hane of evolving the most aurate FRBS, whih normally omes with a omplex struture. The vaine model is then used in the third stage to seed the initial population of PAIA in order to obtain a set of fuzzy models with improved interpretability. To takle the problem of simultaneously optimizing the rule-base struture and parameters, a variable length oding sheme is adopted, and a new distane index is proposed to ope with the variablelength individuals, whih should improve the effiieny of the searh. For model struture optimization, a Model Simplifiation module is added after the Affinity Maturation in a bid to find transparent FRBSs. Fig. represents a shemati diagram of suh a modeling framework. 1) Eliitation of the Initial Singleton/Mamdani FRBSs Firstly, an evolutionary based K-means lustering algorithm [5] is used to group the available data into a predefined number of lusters. In order to onvert the obtained lusters into FRBSs, a ertain mehanism has to be

4 Yes First Stage: Extrating The Initial FRBS Seond Stage: Refining The Initial FRBS The Initial FRBS An Immune Algorithm Based Fuzzy Preditive Modeling Mehanism For Singleton FRBS, is equal to. If Centroid of Area (COA) defuzzifiation method is employed, the risp output of the initial Singleton FRBS an be omputed as below: (4) where, ( Third Stage: Multi-objetive Fuzzy Modelling A Set of Fuzzy Models No Initial Population Pool Ativation Clone Variable Length Coding Affinity Maturation Clone ( ) ), and ( ) is the parameter vetor whih is subjet to further tuning in the seond modeling stage. For Mamdani FRBS, the bell-shape membership funtions are used for : ( ) (5) established so that and the orresponding output (refer to Setion II) an be linked with the extrated lusters. Gaussian membership funtions are used for the inputs of FRBSs. In suh a ase, the ith identified luster entre in the input spae orresponds diretly to the entroids of the Gaussian membership funtions responsible for the ith rule. The spreads of the orresponding Gaussian funtions are obtained by first alulating the matrix as follows: ( ) (1) where, are k luster enters in the input spae, is the Eulidean distane, and speifies the degree of data point belonging to the ith luster. Spread is thus dedued as follows: ( ( ) ) ( ) ( ) Model Simplifiation Reseletion Network Suppression [ ]( ) where, j indiates the dimension of the spread in the input spae for the ith luster, N is the total number of data points. The maximum value of is piked to ensure a ertain degree of overlap between different lusters. This also ensures a smooth transition of the preditions over different regions. is used to adjust the degree of overlap, and is set to 0.95 in this work without any loss of generality. Hene, the Gaussian membership funtion on eah dimension an be speified using (): Stop Fig.. The proposed IMOFM framework. () ( ) ( ( ) ) () where, is obtained by using (1) and () but in the output spae. Unlike traditional Mamdani FRBS where defuzzifiation is normally applied on the overall implied fuzzy set [, p. 64], IMOFM employs the enter of gravity (COG) defuzzifiaiton on the implied fuzzy set as below: (6) Instead of using minimum and maximum, IMOFM hooses to use produt and plus for the T-norm and S-norm respetively. All these modifiations are done to ensure omputational effiieny, and more importantly, to ensure that an analytial solution desribed in (7) an be deduted. (7) where, is the enter of area of the membership funtion and is the peak ( ) if is symmetri; is the final defuzzified output of the FRBS. ( ) is the parameter vetor whih is subjet to further fine-tuning in a bid to improve the model s preditive performane. denotes the area under over the output interval [ ] and is alulated using (8). [ ( ) ( )] (8) Hene, after the first stage, a Singleton/Mamdani FRBS with the pre-speified number of rules is extrated from the numerial data, whih is analytial and an be refined further using the seond modeling stage. ) Refinement of Initial FRBSs In the seond modeling stage, a BEP with momentum terms algorithm [, p. 46-5] is developed to first improve the auray of the initial FRBS by adjusting the parameters in so that the rule-base with the over-estimated number of rules an also survive under the pressure of

5 seletion. By taking the partial derivatives of (4) and (7) with respet to eah parameter inluded in, one an obtain a set of parameter updating laws (due to the spae, interested readers are referred to [4]). One problem assoiated with the BEP updating formulas is that they inlude no onstraints with respet to the update mehanism of these parameters. Hene, during the ourse of the optimization, the enters of the membership funtions are likely to be plaed outside the boundaries. Hene, in this work a simple onstraint handling sheme is added, whih heks the boundary violation for enters during eah iteration step and drives any violated enters bak to the boundaries. ) Multi-objetive Fuzzy Modeling a) Forming Objetive funtions Two onfliting objetive funtions are formulated with the first fousing on the predition auray and the seond on the struture simplifiation as desribed in (9); and are th predited and real outputs; Nrule is the number of fuzzy rules in FRBS; Nset is the total number of fuzzy sets; RL is the summation of the rule length of eah rule ( Don t are is not inluded in the rule length). (a) (b) FRBS1 Rule1 q: ( 1 FRBS Rule1 Rule 1 1 ; ; b 1 ; Rule Singleton FRBS: q: ( 1 Mamdani FRBS: Rule ; b Rule 1 1 ; ; b 1 ; original vaine FRBS extrated from the first two modelling stages. and are the entre and the spread of the ith rule s onsequent. is a random number within [0, 1]. defines the minimum interval between the entre and its orresponding upper and lower limits of the input (or the output) variable, whihever is smaller. and are the user speified parameters whih define how muh different the newly generated FRBSs are from the original vaine one, and are set to 0. and 0.1 respetively. Rule4 ; b y y Rule5 Rule6 Fig.. Variable length oding sheme for: (a) a three-rule Singleton/Mamdani FRBS; (b) a six-rule Singleton/Mamdani FRBS. ) ) ( ) (10) b) Variable Length Coding Sheme and Initial Population As far as the multi-objetive fuzzy modeling is onerned, different enoding shemes have been proposed and an be broadly divided into two ategories: 1) enoding based on the global data-base; ) enoding based on the effetive rule parameters. The former is mainly found in the linguisti modeling stream [1]-[15], in whih a string or a rule matrix is formed as the hromosome in order to selet the effetive rules and linguisti terms from the andidate set. [17] represents the variant of the first enoding sheme, in whih the enoding omprised the struture oding and the parameter (data-base) oding. Key to the first enoding sheme is that the global data-base is usually kept unhanged or only varied in a onstrained searh spae. The latter is mainly found in the approximate modeling stream due to the lak of global data-base. Sine only the effetive rule parameters are inluded in the oding, a variable length oding sheme is inevitable and is used in this work. Fig. gives an example of how to enode FRBSs. As shown in Fig., the inrease of the ode length is only linear to the variable s dimension, whih effetively takles the effiieny of the searh and the urse of dimensionality. The initial population is obtained with all individuals generated around the vaine model using (10) and (11). Where, and are the entre and spread of the ith rule and the jth input membership funtion in the (11) Suh a forming approah only aquires the knowledge about the maximum allowable number of rules (i.e. the prespeified number of lusters) and the data so that emphasis of the third modeling stage is plaed on the automati eliitation of a set of FRBSs in the sense. The size of the initial population in this work is set to 7. ) Variation Operators and a New Distane Index The variation operator used in PAIA is Affinity Maturation whih mutates the seleted good solutions aording to their affinity values ( as desribed in (1). Suh a variation operator is used in IMOFM to optimize the assoiated parameters enoded in q (see Fig. ). (1) where N(0, 1) is a Gaussian random variable with zero mean and standard deviation 1. is a dimension index within the length of that has been hosen to mutate. One problem in using (1) is assoiated with whih is originally alulated based on the distane between two fixed-length individuals. Given the variable length oding sheme and the unonstrained optimization used in this work, a onomitant effet of the so-alled unordered sets of rules [6] may our. As pointed in [6], variations

6 over unordered individuals are equivalent to ombing the mother s gen for good vision and father s gen for urly hair, whih does not make muh sense. To takle this problem, a new distane index is proposed for the alulation of. The basi idea is to align the losest rules from different individuals in order to have a meaningful variation. ( ) ( ) ( ) (1) where, and are two FRBSs with and rules; is the length of the rule; ( represents the losest rule in ) with respet to the rule in ; is the absolute value of. d) Model Simplifiation A model simplifiation step is added as shown in Fig.. The aim is to remove the redundany both in the rules and in the fuzzy sets. On the rule level: 1) one of the insignifiant rules (rules that ontribute the least to any predition error inrease when not inlude these rules) is removed unless the rule base reahes the fewest rules designated by the user; ) one of singleton rules [1] (rules whose omprising fuzzy sets are similar to singleton set) is removed; ) the most similar pair of rules based on the Similarity of Rule Premise (SRP) [0] are merged. On the fuzzy sets level: 1) one fuzzy set that is the most similar to the universal fuzzy set is labeled as Don t are ; ) two most similar fuzzy sets from the inputs and output dimensions are merged to form a single fuzzy set based on the similarity measure [0]. It is worth mentioning that the above operations are exeuted for eah loned FRBS at eah iteration step. A set of thresholds whih ontrol the various similarity measures are speified by the users. Interested readers are referred to [4, h.5] for more details. During the experiments, it is found that the thresholds are not the ritial parameters due to the following two reasons: 1) only one fuzzy rule or two fuzzy sets are removed or merged at eah iteration step; ) elitism is adopted to reord any non-dominated solution found at eah iteration step. modeling stage using (10) and (11). The initial population size is set to 7. The number of iterations in the third stage is set to 100. Table I summarizes suh omparative results fousing on their preditive performanes and the number of rules. The results in Table I inlude the average values of 0 runs. Fig. 4 shows the membership funtion distribution of input (x ) through the three modeling stages. TABLE I COMPARISONS OF THE PREDICTIVE PERFORMANCE OF THE DIFFERENT MODELING METHODS FOR THE BENCHMARK PROBLEM Modeling Methods (Ref.) No. of rules No. of fuzzy sets & The type of FRBS Performane (RMSE training) [5] 6 1 Mamdani * [7] 5 10 Singleton * IMOFM_S (IMOFM for Singleton FRBS): Initial FRBS 5 10 Singleton * FRBS1(0 times) 5 9 Singleton # FRBS(0 times) 5 8 Singleton # FRBS(9 times) 4 8 Singleton # FRBS4(9 times) 6 Singleton # FRBS5(5 times) (5 T ) 4 Singleton # IMOFM_M (IMOFM for Mamdani FRBS): Initial FRBS 5 15 Mamdani * FRBS1(5 times) 5 14 Mamdani # FRBS( times) 5 1 Mamdani # FRBS(6 times) 4 11 Mamdani # FRBS4(8 times) 9 Mamdani # FRBS5(8 times) (5 T ) 5 Mamdani # &: For IMOFM_S, it is the number of fuzzy sets in its inputs; for IMOFM_M, it is the number of fuzzy sets in its inputs and output. *: Initial model extrated diretly from data using lustering algorithms or grid partition Refined model or the onsequents are omputed through the estimation methods. #: Simplified model after model simplifiation and parameter fine tuning. T: Total number of rule length. : Stardard deviation of the results obtained from 0 runs. Input (x) of a 5-rule initial Mamdani FRBS Input (x) of a 5-rule refined Mamdani FRBS Input (x) of a 4-rule simplified Mamdani FRBS A. A Benhmark Problem V. EXPERIMENTS The benhmark example used in this setion is a nonlinear stati system with two inputs and one output, whih has been studied in [7]. The system is defined as follows: (14) Although this problem is a simple low-dimensional problem, it is a very good example in terms of demonstrating how IMOFM works. The same 50 input-output data pairs as those used in [5] and [7] are olleted. The number of lusters is set to 5 in the first modeling stage. The refined 5-rule initial FRBS is used to seed the initial population in the third Input (x) First Modeling Stage Seond Modeling Stage Third Modeling Stage Fig. 4. The membership funtion distribution of input. Sine different runs will lead to slightly different FRBS onfigurations, Table I also reords the number of eah FRBS onfiguration found within the 0 runs. Most onfigurations are found more than 0 times within 0 runs, whih suggests that IMOFM is robust and onsistent. The Comparing to other Singleton and Mamdani modeling approahes, IMOFM was found to represent the most aurate results with simpler rule-base struture.

7 In order to test the influenes of the eah modeling stages, two variants of the proposed IMOFM are investigated: 1) the ombination of the first stage and the third stage; ) only the third stage. In the latter ase, the initial 5-rule FRBS is randomly generated within the variable domains. Table II summarizes the results of the two variants. It is worth mentioning that for the two variants, the thresholds for the model simplifiation are set at higher values so that the merging operation only happens when the fuzzy sets or rules are very similar. This is to ensure that the FRBSs with more rules are given a better hane of surviving in the early stages of the evolution. In [19], a nihe onept is used to maintain a set of FRBSs with the same number of rules. A substitution only happens within eah nihe so that one an evolve a set of FRBSs with the different number of rules without the worry of losing individuals with more rules. The proposed three-stage proedure does not need the aforementioned nihe onept if all the stages work as a unified proedure. In suh a ase, the most aurate FRBS is always the one with the number of rules lose to the maximum value. More importantly, this aurate FRBS will diret the searh from the most omplex struture (the more aurate one) to the simplest ones (the less aurate ones). This ensures the oexistene of FRBSs with various omplexities during the seletion. TABLE II COMPARISONS OF THE PREDICTIVE PERFORMANCE OF THE DIFFERENT MODELING STAGES FOR THE BENCHMARK PROBLEM Modeling Methods (Ref.) No. of rules No. of fuzzy sets & The type of FRBS Performane (RMSE training) IMOFM (the first stage and the third stage); numbeer of iterations: 000 Initial FRBS 5 10 Singleton * FRBS1 5 6 Singleton # FRBS 4 6 Singleton # FRBS 5 Singleton # FRBS4 4 Singleton # FRBS5 (4 T ) Singleton 0.75 # IMOFM (only the third stage); number of iterations: 4000 Initial FRBS 5 10 Singleton 1.06 * FRBS1 4 7 Singleton # FRBS 4 6 Singleton 0.1 # FRBS 5 Singleton # FRBS4 (8 T ) 4 Singleton # FRBS5 (7 T ) 4 Singleton 0.11 # *: Initial model extrated diretly from data using lustering algorithms or grid partition methods; T: Total number of rule length; #: Simplified model after model simplifiation and parameter fine tuning.. As shown in Table II, more iterations are needed for the two variants to ahieve a similar preditive performane as that obtained using the three-stage modeling proedure (refer to Table I), and only a few FRBSs are obtained. The most omplex struture whih is supposed to evolve to the most aurate FRBS is disarded during the optimization for the reasons already desribed. All these justified the inlusion of the first two stages. B. Real World Appliations In this setion, the problem of prediting the Ultimate Tensile Strength (UTS) of heat-treated steel is studied, whih features a high dimensional, nonlinear and sparse data spae. The UTS data set onsists of 760 data samples and inludes 15 inputs and 1 output whih is the UTS with the values between 516. and 184. In order to ompare with the work in [8], the data set is randomly divided into two parts: 75% of the data are used for training and the remaining data are used for testing. Another 1 unseen examples are also inluded. All the parameter settings related to IMOFM are the same as those used for the benhmark problem exept that the initial number of rules is set to 1. The results presented in Table III inlude the average values of 10 independent runs and only a few FRBS are presented due to the onstraints on spae. TABLE III COMPARISONS OF THE PREDICTIVE PERFORMANCE FOR THE DIFFERENT MODELING METHODS USING THE UTS DATA Modeling First Stage (lustering algorithm) Seond Stage (single objetive refining) Methods Training Testing Training Testing Validation [8] IMOFM_S IMOFM_M Third Stage (multi-objetive fuzzy modeling) Modeling Methods [8] FRBS1 FRBS IMOFM_S FRBS1 FRBS FRBS IMOFM_M FRBS1 FRBS FRBS No. of rules No. of Fuzzy sets in inputs Inputs: [ ] Output: 10 Inputs: [ ], Output: 9 Inputs: [ ], Outputs: 10 Modeling performane Training Testing/ Validation / / /41.01 Inputs: [ ], Output: /1.54 Inputs: [ ], Output: 7 10 Inputs: [ ], Output: Inputs: [ ], Output: 5 Inputs: [ ], Output: / / / /49.87 As shown in Table III, the problem of over-fitting speifially related to the seond modeling stage (vaine FRBS) under unseen situations is revealed in Table III. Suh over-fitting is mainly attributed to the omplex strutures involved in the first two modeling stages. However, the simplified fuzzy models an predit well even under unknown senarios. Fig. 5 shows the snapshot of the obtained approximate fronts at different iterations. The evolution starts from the most aurate FRBS and expands the front during the ourse of the optimization. Table IV summarizes the results of the UTS modeling problem using IMOFM with and without the

8 variable length oding and the new distane index. Muh bigger improvements have been registered for the FRBS with fewer rules sine they are more prone to suffering from the problem of unordered set of rules. Fig. 5. The FRBSs at different iterations. TABLE IV THE COMPARISON OF THE MODELING APPROACHES WITH AND WITHOUT VARIABLE LENGTH CODING SCHEME FRBS Configurations No. of rules IMOFM_S (without VLC) (Training RMSE) VI. CONCLUSION The main novelty of IMOFM is onsidered to be as follows: 1) the initial number of rules in the rule-base is not an important fator anymore sine in the third stage a set of FRBS with different struture are eliited; only the maximum allowable number of rules is required a priori; ) due to the vaination proess, the effiieny and preditive auray of the modelling are improved; ) by using the variable length oding sheme, the problem of the unordered set of rules is resolved, whih leads to a more effiient parameter optimisation; 4)the proposed method represents one of the first attempts whih uses an approximate Mamdani FRBS for high-dimensional regression problems, and an be viewed as a omplement to the linguisti Mamdani fuzzy modeling approah. REFERENCES IMOFM_S (with VLC) (Training RMSE) Improvement (%) FRBS % FRBS % FRBS % FRBS % [1] B. Kosko, Fuzzy Systems as Universal Approximators, IEEE Transations on Computers, vol. 4(11), pp. 19-1, [] K. M. Passino, S. Yurkovih, Fuzzy Control, MA: Addison-Wesley, 1998, pp [] J. H. Holland, Adaptation in Natural and Artifiial Systems, MI: The University of Mihigan Press, [4] J. D. Farmer, N. H. Pakard, The Immune System, Adaptation, and Mahine Learning, Physia, vol. D, pp , [5] M. Delgado, F. G mez-skarmeta Antonio, F. Martin, A Fuzzy Clustering-Based Rapid Prototyping for Fuzzy Rule-Based Modelling, IEEE Transations on Fuzzy Systems, vol. 5(), pp. -, [6] C. L. Karr, Geneti Algorithms for Fuzzy Controllers, AI Expert, vol. 6 (), pp. 6-, 1991 [7] T. Takagi, M. Sugeno, Fuzzy Identifiation of Systems and Its Appliations to Modelling and Control, IEEE Transations on Systems, Man, and Cybernetis, vol. 15, pp , [8] E. H. Mamdani, Appliations of Fuzzy Algorithm for Control a Simple Dynami Plant, Pro. Inst. Eletr. Eng., vol. 11 (1), pp , [9] J. Casillas, O. Cordon, J. Del Jesus Mara, F. Herrera, Geneti Tuning of Fuzzy Rule Deep Strutures for Linguisti Modelling, IEEE Transations on Fuzzy Systems, vol. 1, pp. 1-9, 001. [10] L. Zadeh, Outline of A New Approah to the Analysis of Complex Systems and Deision Proesses, IEEE Transations on Systems, Man, and Cybernetis, vol., pp. 8-44, 197. [11] F. Jiménez, A. F. Gómez-Skarmeta, H. Roubos, R. Babuška, Aurate, Transparent, and Compat Fuzzy Models for Funtion Approximation and Dynami Modeling through Multi-Objetive Evolutionary Optimisation, in E. Zitzler et al. (Eds.): EMO 001, LNCS 199, pp [1] F. Herrera, Geneti Fuzzy Systems: Taxonomy, Current Researh Treads and Prospets, Evol. Intel., vol. 1 (1), pp. 7-46, 008. [1] H. Ishibuhi, K. Nozaki, N. Yamamoto, H. Tanaka, Seleting Fuzzy If-Then Rules for Classifiation Problems Using Geneti Algorithms, IEEE Transations on Fuzzy Systems, vol. (), pp.60-70, [14] H. Ishibuhi, T. Nakashima, T. Murata, Three-objetive Genetisbased Mahine Learning for Linguisti Rule Extration, Information Sienes, vol. 16, pp , 001. [15] H. Ishibuhi, T. Yamamoto, Fuzzy Rule Seletion by Multi- Objetive Geneti Loal Searh Algorithms and Rule Evaluation Measures in Data Mining, Fuzzy Sets and Systems, vol. 141, pp , 004. [16] M. J. Gato, R. Alalά, F. Herrera, Adaptation and Appliation of Multi-objetive Evolutionary Algorithms for Rule Redution and Parameter Tuning of Fuzzy Rule-based Systems, Soft Computing, vol. 1 (5), pp , 009. [17] R. Alalά, P. Duange, F. Herrera, B. Lazzerini, A Multiobjetive Evolutionary Approah to Conurrently Learn Rule and Data Bases of Linguisti Fuzzy-Rule-Based Systems, IEEE Transations on Fuzzy Systems, vol. 17(5), pp , 009. [18] M. Setnes, R. Babuška, U. Kaymak, H. Lemke, Similarity Measures in Fuzzy Rule Base Simplifiation, IEEE Transations on Systems, Man, and Cybernetis-Part B, vol. 8 (), pp , [19] F. Jiménez, G. Sánhez, A. F. Gómez-Skarmeta, H. Roubos, R. Babuška, Fuzzy Modeling with Multi-Objetive Neuro-Evolutionary Algorithms, IEEE International Conferene on Systems, Man, and Cybernetis, vol., 00. [0] Y. Jin, W. Von Seelen, B. Sendhoff, On Generating FC Fuzzy Rule Systems From Data Using Evolution Strategies, IEEE Transations on Systems, Man, and Cybernetis, vol. 9 (6), pp , [1] H. L. Wang, S. Kwong, Y. C. Jin, W. Wei, K. F. Man, Multiobjetive Hierarhial Geneti Algorithm for Interpretable Fuzzy Rule-based Knowledge Extration, Fuzzy Sets and Systems, Vol. 149 (1), pp , 005. [] J. Chen, M. Mahfouf, A Population Adaptive Based Immune Algorithm for Solving Multi-objetive Optimisation Problems, in H. Bersini & J. Carneiro (Eds.): ICARIS 006, LNCS 416, pp. 80-9, 006. [] J. Chen, M. Mahfouf, Artifiial Immune Systems as a Bio-inspired Optimisation Tehnique and Its Engineering Appliations, in H. W. Mo (Eds.): Artifiial Immune Systems and Natural Computing: Applying Complex Adaptive Tehnologies, pp. -48, 008. [4] J. Chen, Biologially Inspired Optimisation Algorithms for Transparent Knowledge Extration Allied to Engineering Materials Proessing, The University of Sheffield, Ph.D. Thesis, 009, h.. [5] J. Chen, M. Mahfouf, An Immune Algorithm Based Fuzzy Preditive Modeling Mehanism using Variable Length Coding and Multiobjetive Optimisation Allied to Engineering Materials Proessing, in proeedings of the 008 IEEE International Conferene on Granule Computation (GrC008), pp.6-8, 008. [6] M. G. Cooper, J. J. Vidal, Geneti Design of Fuzzy Controllers: The Cart and Jointed-Pole Problem, in Proeedings of the Third IEEE Conferene on Fuzzy Systems, vol., pp. 1-17, [7] M. Sugeno, T. Yasukawa, A Fuzzy-Logi-Based Approah to Qualitative Modeling, IEEE Transations on Fuzzy Systems, vol. 1 (1), pp. 7-1, 199. [8] Q. Zhang, Nature-Inspired Multi-Objetive Optimisation and Transparent Knowledge Disovery via Hierarhial Fuzzy Modelling, Ph.D. Thesis, The University of Sheffield, U.K, 009, h.6.