Towards Model Evaluation using Self-Organizing Maps

Towards Model Evaluation using Self-Organizing Maps M. Herbst M.C. Casper Dept. of Physical Geography University of Trier, Germany

Model evaluation and model comparison Can we process the information contained in model time series more efficiently? Standard performance measures (RMSE, R², ) and model evaluation: assumptions about model errors violated loss of Information 30 discharge [m³/s] 25 20 15 Approaches to model evaluation using Self-Organizing Maps 10 5 0 14/01/95 23/02/95 04/04/95 14/05/95 23/06/95 02/08/95 11/09/95 21/10/95 dd/mm/yy

The Self-Organizing Map Some facts about Self-Organizing Maps (SOM): an unsupervised learning neural network model since 1982 developed by T. Kohonen (Helsinki University of Technology) biologically inspired by the organization of the cerebral cortex A Self-Organizing Map projects high-dimensional data on a low-dimensional map is topology-preserving: similar data is mapped to nearby locations common applications in pattern recognition, clustering etc.

Self-Organizing Map of model time series N=4000 x(t)={x 1,x 2, x 17472 } carry out a Monte Carlo Simulation with hydrological model arrange and cluster resulting discharge time series pattern by similarity similarity measure: Euclidean distance

The Self-Organizing Map architecture prototype vectors m i Input x(t)= {x 1, x 2, x 3 } Step 1: Initialization constitutes of neurons ( nodes ) located on a regular map grid data samples x(t) are considered as vectors with n (here 3) components each neuron has an associated prototype vector m ii

The SOM training (example) Euclidean distances di = x m i Input x(t)= {3, 4, 6} Step 2: randomly pick an input vector from the training set calculate Euclidean distances d ii find the neuron with the smallest d ii for the given x (the so called best-matching unit, BMU)

The SOM training (example) Updating of reference vectors: m i ( t + 1) = m ( t) + α( t) h ( t) [ x( t) m ( t) ] i ci i Input x(t)= {3, {5, 4, 9, 6} 1} (3;3.2; 5) Step 3: update reference vectors within the neighbourhood around the BMU ( Gaussian kernel ) pick another data item and repeat step 2 + 3 cycle through the whole input data (2.2;4. 8;5.2) (3;4.5; 6)

Adaptation to multidimensional distributions data item x(t) is projected onto a neuron on the map the neighbouring neurons around the BMU adjust their weight vectors m i by Δm i BMU the degree of adjustment decreases with distance from the BMU x Δm i m i SOM V model space modified after Ritter et. al. (1990)

Method N=4000 x(t)={x 1,x 2, x 17472 } Monte-Carlo simulations (7 free parameters): 4000 data items (vectors) à à 17.472 elements Normalize each time step using x = ( x x ) σ x Each node now represents some model realizations with similar time series pattern Display means of the 7 parameter values on each node Display means of performance measures on each node SOM-Training (15x22 = 330 nodes) Project the measured data onto the SOM (= find BMU)

Monte-Carlo Simulation: model parameters Name Function Range RetBasis storage coefficient baseflow [h] 0.5-3.5 RetInf storage coefficient interflow [h] 2.0-6.0 RetOf storage coefficient surface runoff [h] 2.0-6.0 StFFRet storage coefficient runoff in impervious surfaces (urban areas) [h] 2.0-6.0 hl horizontal hydraulic conductivity factor 2.0-8.0 maxinf maximum infiltration factor 0.025-1.025 vl vertical hydraulic conductivity factor 0.005-0.105

Method N=4000 x(t)={x 1,x 2, x 17472 } Monte-Carlo simulations (7 free parameters): 4000 data items (vectors) à à 17.472 elements Normalize each time step using x = ( x x ) σ x Each node now represents some model realizations with similar time series pattern Display means of the 7 parameter values on each node Display means of performance measures on each node x Δmi m i BMU SOM SOM-Training (15x22 = 330 nodes) Project the measured data onto the SOM (= find BMU) V

Parameter mean values on each node storage coefficient baseflow (RetBasis) storage coefficient interflow (RetInf) storage coeff. surface (RetOf) storage coefficient urban (StFFRet) partially sensitive parameters? sensitive parameters insensitive/interacting(?) parameters horizontal hydraulic conduct. factor (hl) max. infiltration factor (maxinf) vertical hydraulic conduct. factor (vl)

Statistical performance measures used Name BIAS RMSE CEFFlog Description Mean error Root of mean squared error Logarithmized Nash-Sutcliffe coefficient of efficiency IAg Willmott s index of agreement; 0 IAg 1 MAPE VarMSE Rlin Mean average percentual error Variance part of the mean squared error Coefficient of determination

Distribution of mean performance values BIAS RMSE CEFFlog IAg 0.15 0.1 0.05 1.7 1.6 1.5 1.4 1.3 0.5 0.4 0.3 0.2 0.9 0.89 0.88 0.87 0.86 0.85 MAPE VARmse Rlin 46 44 42 40 38 36 0.25 0.2 0.15 0.1 0.05 0.82 0.8 0.78 0.76

Identification of Best-Matching Unit (BMU) Input Q observed (t) BMU identify the map unit which is most similar to the measured discharge time series retrieve the model simulations attributed to this node

Position of observed discharge time series BIAS RMSE CEFFlog IAg 0.15 0.1 0.05 1.7 1.6 1.5 1.4 1.3 0.5 0.4 0.3 0.2 0.9 0.89 0.88 0.87 0.86 0.85 = position of Q obseved on the SOM MAPE VARmse Rlin 46 44 42 40 38 36 0.25 0.2 0.15 0.1 0.05 0.82 0.8 0.78 0.76

Parameter ranges of BMU model realizations 1 most sensitive parameters 0.8 norm. param. value 0.6 0.4 0.2 0 RetBasis RetInf RetOf StFFRet hl maxinf vl M. Herbst & M.C. Parameter Casper - iemss 2008 - Barcelona

Model results 40 envelope of all Monte-Carlo simulations 35 30 discharge [m³/s] 25 20 15 10 5 0 14/01/95 23/02/95 04/04/95 14/05/95 23/06/95 02/08/95 11/09/95 21/10/95 dd/mm/yy

Model results 40 BMU of observed discharge time series (7 model realizations) 35 30 discharge [m³/s] 25 20 15 10 5 0 14/01/95 23/02/95 04/04/95 14/05/95 23/06/95 02/08/95 11/09/95 21/10/95 dd/mm/yy

Model results 40 observed discharge time series 35 30 discharge [m³/s] 25 20 15 10 5 0 14/01/95 23/02/95 04/04/95 14/05/95 23/06/95 02/08/95 11/09/95 21/10/95 dd/mm/yy

Model results 40 result of manual expert model calibration 35 30 discharge [m³/s] 25 20 15 10 5 0 14/01/95 23/02/95 04/04/95 14/05/95 23/06/95 02/08/95 11/09/95 21/10/95 dd/mm/yy

Model results 40 35 result of RMSE optimization using Shuffled Complex Evolution (SCE-UA, Duan et al. 1993) 30 discharge [m³/s] 25 20 15 10 5 0 14/01/95 23/02/95 04/04/95 14/05/95 23/06/95 02/08/95 11/09/95 21/10/95 dd/mm/yy

Conclusions using a SOM, the model realizations can be ordered by similarity gives insights into the occurrence of insensitive or interacting parameters model optima with regard to different performance measures the SOM allows a kind of parameter estimation computational cost is still considerably high

Coming up next adding contextual meaning to the map data reduction improved performance BMU %BiasRR %BiasFDCm %BiasFHV %BiasFLV %BiasFMM

Coming up next adding contextual meaning to the map data reduction improved performance Model comparison using SOM model independence check model ensemble control Model LARSIM Model NASIM

The question behind the question Can we process the information contained in model time series more efficiently?? What is the information contained in the time series? How detailed do we have to describe the model behaviour (i.e. how many descriptors do we need?)

Thank you very much for your attention! correspondence to: herbstm@uni-trier.de All experiments have been conducted using the SOM Toolbox for MATLAB (Vesanto et al. 2000) 1997-2000 by the SOM toolbox programming team. http://www.cis.hut.fi/projects/somtoolbox/