Towards Model Evaluation using Self-Organizing Maps

Transcription

1 Towards Model Evaluation using Self-Organizing Maps M. Herbst M.C. Casper Dept. of Physical Geography University of Trier, Germany

2 Model evaluation and model comparison Can we process the information contained in model time series more efficiently? Standard performance measures (RMSE, R², ) and model evaluation: assumptions about model errors violated loss of Information 30 discharge [m³/s] Approaches to model evaluation using Self-Organizing Maps /01/95 23/02/95 04/04/95 14/05/95 23/06/95 02/08/95 11/09/95 21/10/95 dd/mm/yy

3 The Self-Organizing Map Some facts about Self-Organizing Maps (SOM): an unsupervised learning neural network model since 1982 developed by T. Kohonen (Helsinki University of Technology) biologically inspired by the organization of the cerebral cortex A Self-Organizing Map projects high-dimensional data on a low-dimensional map is topology-preserving: similar data is mapped to nearby locations common applications in pattern recognition, clustering etc.

4 Self-Organizing Map of model time series N=4000 x(t)={x 1,x 2, x } carry out a Monte Carlo Simulation with hydrological model arrange and cluster resulting discharge time series pattern by similarity similarity measure: Euclidean distance

5 The Self-Organizing Map architecture prototype vectors m i Input x(t)= {x 1, x 2, x 3 } Step 1: Initialization constitutes of neurons ( nodes ) located on a regular map grid data samples x(t) are considered as vectors with n (here 3) components each neuron has an associated prototype vector m ii

6 The SOM training (example) Euclidean distances di = x m i Input x(t)= {3, 4, 6} Step 2: randomly pick an input vector from the training set calculate Euclidean distances d ii find the neuron with the smallest d ii for the given x (the so called best-matching unit, BMU)

7 The SOM training (example) Updating of reference vectors: m i ( t + 1) = m ( t) + α( t) h ( t) [ x( t) m ( t) ] i ci i Input x(t)= {3, {5, 4, 9, 6} 1} (3;3.2; 5) Step 3: update reference vectors within the neighbourhood around the BMU ( Gaussian kernel ) pick another data item and repeat step cycle through the whole input data (2.2;4. 8;5.2) (3;4.5; 6)

8 Adaptation to multidimensional distributions data item x(t) is projected onto a neuron on the map the neighbouring neurons around the BMU adjust their weight vectors m i by Δm i BMU the degree of adjustment decreases with distance from the BMU x Δm i m i SOM V model space modified after Ritter et. al. (1990)

9 Method N=4000 x(t)={x 1,x 2, x } Monte-Carlo simulations (7 free parameters): 4000 data items (vectors) à à elements Normalize each time step using x = ( x x ) σ x Each node now represents some model realizations with similar time series pattern Display means of the 7 parameter values on each node Display means of performance measures on each node SOM-Training (15x22 = 330 nodes) Project the measured data onto the SOM (= find BMU)

10 Monte-Carlo Simulation: model parameters Name Function Range RetBasis storage coefficient baseflow [h] RetInf storage coefficient interflow [h] RetOf storage coefficient surface runoff [h] StFFRet storage coefficient runoff in impervious surfaces (urban areas) [h] hl horizontal hydraulic conductivity factor maxinf maximum infiltration factor vl vertical hydraulic conductivity factor

11 Method N=4000 x(t)={x 1,x 2, x } Monte-Carlo simulations (7 free parameters): 4000 data items (vectors) à à elements Normalize each time step using x = ( x x ) σ x Each node now represents some model realizations with similar time series pattern Display means of the 7 parameter values on each node Display means of performance measures on each node x Δmi m i BMU SOM SOM-Training (15x22 = 330 nodes) Project the measured data onto the SOM (= find BMU) V

12 Parameter mean values on each node storage coefficient baseflow (RetBasis) storage coefficient interflow (RetInf) storage coeff. surface (RetOf) storage coefficient urban (StFFRet) partially sensitive parameters? sensitive parameters insensitive/interacting(?) parameters horizontal hydraulic conduct. factor (hl) max. infiltration factor (maxinf) vertical hydraulic conduct. factor (vl)

13 Method N=4000 x(t)={x 1,x 2, x } Monte-Carlo simulations (7 free parameters): 4000 data items (vectors) à à elements Normalize each time step using x = ( x x ) σ x Each node now represents some model realizations with similar time series pattern Display means of the 7 parameter values on each node Display means of performance measures on each node x Δmi m i BMU SOM SOM-Training (15x22 = 330 nodes) Project the measured data onto the SOM (= find BMU) V

14 Statistical performance measures used Name BIAS RMSE CEFFlog Description Mean error Root of mean squared error Logarithmized Nash-Sutcliffe coefficient of efficiency IAg Willmott s index of agreement; 0 IAg 1 MAPE VarMSE Rlin Mean average percentual error Variance part of the mean squared error Coefficient of determination

15 Distribution of mean performance values BIAS RMSE CEFFlog IAg MAPE VARmse Rlin

16 Method N=4000 x(t)={x 1,x 2, x } Monte-Carlo simulations (7 free parameters): 4000 data items (vectors) à à elements Normalize each time step using x = ( x x ) σ x Each node now represents some model realizations with similar time series pattern Display means of the 7 parameter values on each node Display means of performance measures on each node x Δmi m i BMU SOM SOM-Training (15x22 = 330 nodes) Project the measured data onto the SOM (= find BMU) V

17 Identification of Best-Matching Unit (BMU) Input Q observed (t) BMU identify the map unit which is most similar to the measured discharge time series retrieve the model simulations attributed to this node

18 Position of observed discharge time series BIAS RMSE CEFFlog IAg = position of Q obseved on the SOM MAPE VARmse Rlin

19 Parameter ranges of BMU model realizations 1 most sensitive parameters 0.8 norm. param. value RetBasis RetInf RetOf StFFRet hl maxinf vl M. Herbst & M.C. Parameter Casper - iemss Barcelona

20 Model results 40 envelope of all Monte-Carlo simulations discharge [m³/s] /01/95 23/02/95 04/04/95 14/05/95 23/06/95 02/08/95 11/09/95 21/10/95 dd/mm/yy

21 Model results 40 BMU of observed discharge time series (7 model realizations) discharge [m³/s] /01/95 23/02/95 04/04/95 14/05/95 23/06/95 02/08/95 11/09/95 21/10/95 dd/mm/yy

22 Model results 40 observed discharge time series discharge [m³/s] /01/95 23/02/95 04/04/95 14/05/95 23/06/95 02/08/95 11/09/95 21/10/95 dd/mm/yy

23 Model results 40 result of manual expert model calibration discharge [m³/s] /01/95 23/02/95 04/04/95 14/05/95 23/06/95 02/08/95 11/09/95 21/10/95 dd/mm/yy

24 Model results result of RMSE optimization using Shuffled Complex Evolution (SCE-UA, Duan et al. 1993) 30 discharge [m³/s] /01/95 23/02/95 04/04/95 14/05/95 23/06/95 02/08/95 11/09/95 21/10/95 dd/mm/yy

25 Conclusions using a SOM, the model realizations can be ordered by similarity gives insights into the occurrence of insensitive or interacting parameters model optima with regard to different performance measures the SOM allows a kind of parameter estimation computational cost is still considerably high

26 Coming up next adding contextual meaning to the map data reduction improved performance BMU %BiasRR %BiasFDCm %BiasFHV %BiasFLV %BiasFMM

27 Coming up next adding contextual meaning to the map data reduction improved performance Model comparison using SOM model independence check model ensemble control Model LARSIM Model NASIM

28 The question behind the question Can we process the information contained in model time series more efficiently?? What is the information contained in the time series? How detailed do we have to describe the model behaviour (i.e. how many descriptors do we need?)

29 Thank you very much for your attention! correspondence to: All experiments have been conducted using the SOM Toolbox for MATLAB (Vesanto et al. 2000) by the SOM toolbox programming team.