/ r i Cluster Processing

Transcription

1 l _ it I US A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2012/ A1 Vincent et al. (43) Pub. Date: Jan. 26, 2012 (54) SHAPE CLUSTERING IN POST OPTICAL Publication Classi?cation CHARACTER RECOGNITION PROCESSING (51) Int Cl (75) Inventors: Luc Raymond Vincent, W. Palo Smith, Alto, Mountain CA (US); ( ' ) View, CA (U S) (52) US. Cl /182; 382/225 (73) Assignee: GOOGLE INC., Mountain View, CA (Us) (57) ABSTRACT (21) APPl- NOJ 13/184,459 Techni q ues for sha P e clusterin g and a PP lications in P rocess (22) _ ing various documents, including an output of an optical Flled: Jul character recognition (OCR) process. The output of an OCR Related US. Application Data process is classi?ed into a plurality of clusters of clip images and a representative image for each cluster is generated to (62) Division of application No, 1 1/517,818,?led on Sep identify clusters Whose clip images Were incorrectly assigned 8, 2006_ character codes by the OCR process. Imaged DQCument Optical Character Recognition OCR Output Dowment Q (OCR) 302 (e.g., A Book) ' Engine 7 E V I ' : : : : Cluster Generation Gray Scale Data Or : E I : i Coior Data : Clip Images in Clusters : Q l W l : { : i : : / r i Cluster Processing 310 I &Q Post-OCR Processing : M Engine : v y : Modified Clusters Output 3g; : Processing i l Modified OCR Output Document E

2 Patent Application Publication Jan. 26, 2012 Sheet 1 0f 19 US 2012/ A1 146 FIG. 1

3 Patent Application Publication Jan. 26, 2012 Sheet 2 0f 19 US 2012/ A1 210 Use shape similarity based on a shape metric to classify clip / images in the OCR output into different clusters of clip images V 220 Generate a cluster image from clip images of each cluster / as a representative image of the cluster (e.g., an averaged image of clip images in each cluster) V Compare each cluster image with other cluster images 330 based on shape similarity to verify or to detect an error in one or more OCR character codes assigned to a cluster in the OCR output. V 240 Generate and assign one or more new OCR character / codes to a cluster whose OCR character codes assigned by the OCR engine are erroneous in the OCR output V Replace the erroneously assigned one or more OCR 50 character codes with the one or more new OCR character codes at each occurrence of the clip images of the cluster in the OCR output to produce a modified OCR output FIG. 2

4

5 Patent Application Publication Jan. 26, 2012 Sheet 4 0f 19 US 2012/ A1 410 Divide clip images into classes where clip images in one / class have one or more common OCR character codes and can have different shapes and sizes V In each class, divide clip images into buckets where clip images in one bucket within a class have identical and similar sizes and can be in different shapes V in each bucket, divide clip images into clusters where J clip images in one cluster have identical and similar shapes based on their shape distances from one another (e.g., characters identical and similar in font types and sizes) 440 K V Use clip images within each cluster to produce a cluster image as a representative for each cluster (e.g., an averaged image of the clip images in each cluster) 450 V Represent each cluster by the cluster image and the one or more common OCR character codes FIG. 4

6 Patent Application Publication Jan. 26, 2012 Sheet 5 0f 19 US 2012/ A1 510 K Compute centroid for clip image in each bounding box in a cluster (e.g., using gray scale or color data from original imaged document) i 520 Align all bounding boxes in each cluster by overlapping their centroids i 530 Average clip images in aligned bounding boxes to produce an averaged clip image as the cluster image K K FIG. 5

7 Patent Application Publication Jan. 26, 2012 Sheet 6 0f 19 US 2012/ A1 610 Compute shape distances between a particular cluster and all // other clusters, regardless of class " 620 Label each cluster whose shape distance from a nearest cluster in / all classes is greater than a threshold distance as an island cluster without changing a respective OCR character code for the cluster V For each non-island cluster, identify (1) the nearest cluster in the / same class and (2) the nearest cluster from a different class, both 4 of which have more samples than the non-island cluster under processing ls the nearest cluster an acceptable cluster? Yes ls the earest cluster in the same clas s the non-island cluster unde processing? Continue to search for a nearest acceptable cluster? 660 No No i 670 f v l 680 / Replace the OCR character code of the non-island cluster with the OCR character code associated with the nearest cluster as the new OCR character code assigned to the non-island cluster under processing FIG. 6 Maintain the respective OCR assigned character code for the non island cluster under processing

8 Patent Application Publication Jan. 26, 2012 Sheet 7 0f 19 US 2012/ Al 92>> 252C x00 $2; mg 282 $ :25 Q :0 m32am8< QE N 222:0 w52$8< Si

9 Patent Application Publication Jan. 26, 2012 Sheet 8 0f 19 US 2012/ A1 V Clusters <7> FIG. 8 1 r v Ciusters <3> FIG. 9

10 Patent Application Publication Jan. 26, 2012 Sheet 9 0f 19 US 2012/ A1 woo r U6 vmememmwém wmmmg $ A GEO 8:622 0cm?mxosm wmmmg 2250 :9: 55, wmhoom wocmuccoo \ r02 052% mwmooi >>ove>> A. 58: wbmeewp 1/ owm $520 $995 mm

11 Patent Application Publication Jan. 26, 2012 Sheet 10 0f 19 US 2012/ Al One Pixel l l 1100 l l l l l l l l Second l muster mage Sliding Direction Overlapping : Template Position : l: l l l First Overlapping muster Image 1100 Position Temp ate / 1001 Clip Image of A Suspect Word in A Bounding Box Spatial Position > FIG. 11A

12 Patent Application Publication Jan. 26, 2012 Sheet 11 0f 19 US 2012/ A \ Select one of available high-confidence clusters l as a cluster image template l \ Align the centroid of the cluster image template with a centroid of a beginning portion of the suspect Word image 1130 i \ Compute the shape distance between the cluster image template and the beginning portion of the suspect word image 1140 i Shift the cluster image template along a selected direction by one pixel to align the cluster image template with a subsequent portion of the suspect Word image 1150 i No Compute the shape distance between the cluster image template and the second portion of the suspect word image 1160 Has the end of the suspect word image been reached? 1170 Yes all available high confidence clusters been used as cluster image templates? Yes FIG. 1 1 B

13

14 Patent Application Publication Jan. 26, 2012 Sheet 13 0f 19 US 2012/ A1 0mm? M00 20> $288 RE. # m $2 < Now? Em? > 100 M00 P 02 N Z.02 I 5930 moo 59:0 W60.02 z w.02 P mooihon monimou moqhwoa mewwmooe A v 4... P Z wemcm.02 memcm / I \ i mcmwwmooi 0E m? com; w >26 Ema._o moo 26cm. own? Z?

15 Patent Application Publication Jan. 26, 2012 Sheet 14 0f 19 US 2012/ A1 20> E :0 moo 5i # uwmmg 28m < 5.3 N21 63 > moo 59w 2&5 26cm 59:0 moo N.2 N r _OZ 02 I a A M00 z.02 P.02 W698i mewwoooe wcmmcm Q s: A 3; i mewmwooe 83.GE Ema 28w >90 E

16 Patent Application Publication Jan. 26, 2012 Sheet 15 0f 19 US 2012/ A1 Sm? 69:0 moo 29:88 REE com? 22 EmEzooo ummmé Qoom $2 < 100 $56 P.02 > moqumom N.02 mr.ge Ill. Sad 28w > Ema

17 Patent Application Publication Jan. 26, 2012 Sheet 16 0f 19 US 2012/ A processing an image of a document to produce a collection / of non-overlapping sub-regions of the image, each sub-region being at a first resolution l 1620 generating multiple clusters of visually similar clip / sub-regions, each of the sub-regions in the collection being included in one of the clusters generating a representative cluster image for each of the / multiple clusters from the sub-regions in the respective cluster at a second resolution higher than the first resolution 1640 producing a replica image of the document by replacing / sub-regions tn the image with the cluster images for the clusters in which the respective sub-regions are included FIG. 16

18

19 Patent Application Publication Jan. 26, 2012 Sheet 18 0f 19 US 2012/ A1 Client 1802 OCR Servers (OCR Post-OCR) and 1810 Client 1801

20 Patent Application Publication Jan. 26, 2012 Sheet 19 0f 19 US 2012/ A1 226m wmmhoum ommv vtozawz cozmoeseeoo 92:25 9E 82 E mhmimw 100 $00-58 Em ~69 22 E26 I162

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41