Using Segmentation Constraints in an Implicit Segmentation Scheme for Online Word Recognition

Size: px

Start display at page:

Download "Using Segmentation Constraints in an Implicit Segmentation Scheme for Online Word Recognition"

May Hoover
8 years ago
Views:

1 1 Using Segmentation Constraints in an Implicit Segmentation Scheme for Online Word Recognition Émilie CAILLAULT LASL, ULCO/University of Littoral (Calais) Christian VIARD-GAUDIN IRCCyN, University of Nantes

Émilie CAILLAULT LASL, ULCO/University of Littoral

2 2 Aim Problem : Training of a neuro-markovian system in the context of cursive handwritten word recognition. Difficulty : Bootstrapping of the system allowing a training from scratch. Proposed solution : Implicit segmentation procedure Definition of a segmentation control function Definition of a quality segmentation measurement

Difficulty : Bootstrapping of the system allowing a training from scratch.

3 3 How train a neuro-markovian system? Character level training Teaching Signal: character Signal Différents hybrides MMC-MMC RN-MMC Classifier HMM MMC - RN SVM-MMC Kppv-MMC Word level training Teaching signal: word RN-MMC Parole/Hors-ligne Classifier HMM

Différents hybrides MMC-MMC RN-MMC Classifier HMM MMC - RN

4 Global system overview : MS-TDNN 1- MsTDNN-HMM 4 xr y dx dy cx cy P/D P O bs1 1 state / character: 66 states TD NN O bs1 O bs n O bservation w indows O t of the TD NN... TDN N inputs : tem poral sequence of features vectors for the w ord «quand» z... 3 states / c b a character: 198 states = 198 output neurons O bs1 Sam e weigths m atrices O bs n TD NN O bst Sam e convolutional w indow inside the observation window s of the TD NN O bs n O bservations vectors sequences processed by the H M M Scanning of the input word by a TDNN (temporal convolution). TDNN produces the sequence of observation probabilities. No explicit segmentation is carried out but just an overlapping window is regularly shifted. Advantages of the TDNN architecture: weight sharing reduction in computation times and in memory requirements. Reduction of the number of computation with an adapted topology of the TDNN due to the sliding observation window.

.. 3 states / c b a character: 198 states = 198 output neurons O bs1 Sam e weigths m atrices O bs n TD NN O bst Sam e convolutional w indow inside the observation window s of the TD NN O bs n O

5 1- MsTDNN-HMM Dynamic Programming Likelihood computation 5 Word Lexicon un deux trois quand q s q 1 q 2 q 3 q f q s q 1 q f Words HMM = concatenation of letters HMM 1-state 3-state letter letter model model q s 1 u 1 n 1 qf q s 1 q 1 u 1 a 1 n 1 d 1 q f DP DP Likelihood Scores For each entry in the lexicon, a pseudo-hmmword model is constructed dynamically by concatenating letter HMM Hence, the likelihood for each word in the lexicon is computed by multiplying the observation probabilities over the best path through the graph using the Viterbi algorithm. The word model with the highest likelihood is the top one recognition candidate. Character models a b c d.n o p q.u.z a ij = cst transition probability O1 O2 O3.. b j (O t ), observation probability Best Viterbi path Dynamic Programming to find the best path alignment Ot, Observation sequences q q q q u a a n n d d d

concatenating letter HMM Hence, the likelihood for each word in the lexicon is computed by multiplying the observation probabilities over the best path through the graph using the Viterbi algorithm.

6 Training Original data 1- MsTDNN-HMM Training : 6 TDNN Delta x - coord y - coord x - direction Y _ direction P ( Class 1 X ) P ( Class 2 X ) P ( Class 3 X ) P ( Class 4 X ) Global cost function Backpropagation of the gradient Normalized data 1.. Sequence of features vectors Normalisation Delta x - coord y - coord x - direction Y _ direction P ( Class 1 X ) P ( Class 2 X )... P ( Class 3 X ) P ( Class 4 X ) Obs 1 ObsT Sequence of vectors of observation probabilities, DP Features Extraction Character Likelihood Word Likelihood Computation Recognition Word Lexicon List of words ranked by their WL score N Word level training Classical backpropagation algorithm Cost function mixing MLE, MMI and character level discrimination [ICDAR 2005] «one» recognized as «over» True Hmm: (+) Best Hmm: ( ) ooonneee oovveeer Best character: ( ) aouvneel

.. P ( Class 3 X ) P ( Class 4 X ) Obs 1 ObsT Sequence of vectors of observation probabilities, DP Features Extraction Character Likelihood Word Likelihood Computation Recognition Word Lexicon List

7 Illustration of the segmentation problem 2- Implicit Segmentation Control 7 Random initialisation of the TDNN all outputs are equally probable Example: Possible paths for the English word of with 10 observations (TDNN windows) o f offfffffff ooooffffff ooooooofff ooffffffff ooooofffff ooooooooff ooofffffff ooooooffff ooooooooof

paths for the English word of with 10 observations (TDNN windows) o f offfffffff

8 How to obtain desired Likelihood distribution? 2- Implicit Segmentation Control 8 Segmentation paths for the word of offfffffff 1 ooffffffff ooofffffff ooooffffff ooooofffff ooooooffff ooooooofff ooooooooff ooooooooof 9 P(Γ) Existing solutions: Training with a character database - duration model for HMM sub-unit [Penacee] - manual segmentation constraints [Remus, Npen++] Proposed solution: - no modification of the HMM structure (transitions=1) - no training with a character database - integration of a duration model in the emission probabilities

ooooooofff ooooooooff ooooooooof 9 P(Γ) Existing solutions: Training with a character database - duration model for HMM sub-unit

9 Weighting function f T : 2- Implicit Segmentation Control 9 Outputstl (, ) = min( Outputs ( t, l)* f ( H );1) NN T Weighting of TDNN outputs % B H Observations range corresponding to one letter H=100 H>1000 E B = First observation position for the concerned letter E = Last observation position for the concerned letter H = Height of the function ƒ T S = E-B = letter size

range corresponding to one letter H=100 H>1000 E B = First observation position for the concerned letter

10 Weighting function applied : case of the word of 2- Implicit Segmentation Control weighting coefficient 50 o f observation position H=100

10 100 weighting coefficient 50 o f 1 0 1

11 2- Implicit Segmentation Control How to evaluate the effectiveness of this weighting function? ASR 11 ASR N N NLs( e) 1 nb _ obsi ( ) = ( ) NLci ( )* T / NLe ( ) e= 1 i= 1 ASR : Averaged homogenous Segmentation Rate e: index of the considered sample T: number of observations scanned by the TDNN N: (fixed or limited) number of training samples NL(e): number of letters in the word label NLs(e): number of sequences of different letters i: index of a sequence of letters from 1 to NLs NLc(i): number of the same consecutive letter and nb_obs(i): number of observations of the letter i in the optimal Viterbi path. Case: of NL(o) NL(f) ASR

observations scanned by the TDNN N: (fixed or limited) number of training samples NL(e): number of letters in the word label NLs(e): number of sequences of different letters i:

12 Experiments : 12 Original data Normalized data Normalisation Training in 2 stages : Global cost function Backpropagation of the gradient TDNN Delta x - coord y - coord x - direction Y _ direction P ( Class 1 X ) P ( Class 2 X ) P ( Class 3 X ) P ( Class 4 X ).. Sequence of features vectors P ( Class 1 X ) P ( Class 2 X )... P ( Class 3 X ) P ( Class 4 X ) Obs 1 ObsT Sequence of vectors of observation probabilities Weighting function ƒ T ƒ T P ( Class 1 X ) P ( Class 2 X ) P ( Class 3 X ) P ( Class 4 X ) 1 True HMM Word Likelihood Computation Features Extraction Delta x - coord y - coord x - direction Y _ direction Character Likelihood Label of the true word P ( Class 1 X ) P ( Class 2 X )... P ( Class 3 X ) P ( Class 4 X ) Obs 1 ObsT Constrained Sequence wtih duration model,, N 1-f T segmentation constraint function for 20 epochs 2- training with no segmentation constraint

.. P ( Class 3 X ) P ( Class 4 X ) Obs 1 ObsT Sequence of vectors of observation probabilities Weighting function ƒ T ƒ T P ( Class 1 X ) P ( Class 2 X ) P ( Class 3 X ) P ( Class 4 X ) 1 True HMM

13 Results on online word IRONOFF database IRONOFF word database English / French Lexicon : 197 words 2/3 Training : words 1/3 TEST : words 13 System 1State- TDNN 1S-TDNN Segmentation constraint 3State-TDNN 3S-TDNN Segmentation constraint Recognition rate (%) Relative improvement (%) Average Segmentation Rate (ASR) Relative improvement (%)

3State-TDNN 3S-TDNN Segmentation constraint Recognition rate (%) 87.09 91.31 90.51 94.

14 Conclusions 14 A simple and effective process to bootstrap a hybrid system without any labeled character database Important increasing of the recognition rate by an initialisation stage with a duration model apply directly on the TDNN outputs Weighting function ƒ T Better quality of segmentation better recognition rates ASR Averaged Segmentation Rate

initialisation stage with a duration model apply directly on the TDNN outputs Weighting

15 Future works : controlled segmentation stage 15 Measure of the ASR for each training iteration Adaptation of the parameter H in the weighting function to reduce the segmentation constraints according to the ASR value. H=ƒ(ASR). Initialisation with H = 100 high segmentation constraint, only balanced segmentation paths are possible Then relax constraint on H when ASR has reached a reasonable value.

16 16 Bonus

17 Introduction Problème Reconnaissance Car. Reconnaissance Mot Conclusion 17 Le moteur de reconnaissance : le TDNN MLP TDNN Global Vision Local vision to global vision Weight sharing

Numerical Field Extraction in Handwritten Incoming Mail Documents

Numerical Field Extraction in Handwritten Incoming Mail Documents Guillaume Koch, Laurent Heutte and Thierry Paquet PSI, FRE CNRS 2645, Université de Rouen, 76821 Mont-Saint-Aignan, France Laurent.Heutte@univ-rouen.fr