Development of The Next Generation Document Reader -Eye Scanner- Toshiyuki Amano *a, Tsutomu Abe b, Tetsuo Iyoda b, Osamu Nishikawa b and Yukio Sato a a Dept. of Electrical and Computer Engineering, Nagoya Institute of Technology; b Intelligent Devices Laboratory, Fuji Xerox Co., Ltd. ABSTRACT Up to now a copying machine requires setting the document to the copy glass of the designated position. In addition to that, the copying machine of the present cannot copy the non-flat shape document (e.g. books, cylindrical objects, etc.) clearly. Besides, it cannot copy without distortion. So, we propose a next generation's document reader "Eye Scanner". The Eye Scanner is composed of the range finder, digital camera and pan-tilt stage system. Due to these devices the Eye Scanner is possible shape information and high-density texture image acquisition. Therefore the Eye Scanner is able to read the document by the free viewpoint and able to generate the image which is not distorted by the geometric conversion. Moreover, the Eye Scanner has the ability for high-resolution image generation by the digital camera, which placed on the optical coaxial position by the half-mirror. In this paper, we explain about the detail of the system. At the explanation of the methodology we explain the methodology about distortion correction by the free form transformation with shape information and explain about the technique of image merging. At the experiment, we show some result of distortion correction and show the result of image merging. Keywords: copying machine, image distortion, range finder, document reading 1. INTRODUCTION Today, the copying machine is a very useful business tool to make copy of the books and documents, and it is necessary in almost every office. However, it is forced to stretch a document evenly, and it has needs to arrange the document by the left side of the copy glass. Because the copying machine is a device copying flat shape object fundamentally and requires contact measurement. In addition to that, the copying machine causes distortion and blur problems at the printout by unevenness in case of the spread book. Kashimura and cho proposed shape recover method that based on the shape from shading, to solve these problems of distortion and blur[1],[2]. In these methods, it is possible to recover the book shape by the video camera and a few light source without any special device. However, these methods required assumptions of the book's material, color and light source position. For above reason, we developed the Eye Scanner that is range finder specialized for document scanning. The Eye Scanner is possible to revise posture and a distortion of the curved document acquired with free point of view to use of measured shape information and is able to get an image same as the state that arranged by copy glass. Generally it causes a problem of the drop of image resolution by the re-sampling at the geometrical image deformation by the 3D shape information but we solve this problem by the high-resolution image acquired by the digital camera of Eye Scanner. This paper is organized as follows: Firstly we introduce the structure of the Eye Scanner developed in Section 2. In the Section 3, we explain a geometrical deformation method of the image distortion, which captured the curved object surface. In the Section 4, we explain the relation between high-resolution texture image and shape information. In the Section 5, we explain a technique of divide capture that used motorized pan-tilt stage system. We show the results * Correspondence: E-mail: amano@elcom.nitech.ac.jp; phone +81 52 735-7133; fax +81 52 735-5424; Dept. of Electrical and Computer Engineering, Nagoya Institute of Technology, Gokiso-cyo, Showa-ku, Nagoya, Japan 466-8555
of some experiment in Section 6 and conclude it with Section 7. 2. STRUCTURE OF THE EYE-SCANNER This section explains a structure of Eye Scanner. The Eye Scanner is composed by range finder, high-resolution digital camera and motorized pan-tilt stage system (Fig. 1). As the range finder we used mechanism of Cubic-Scope[3] and based active stereo measurement. Range finder that composed in the Eye Scanner has the turned polygon mirror and NTSC video camera, and it has ability of measurement of 640 * 480 pixels range-image at 0.6 sec. Besides, the resolution of the range value is 8 bits (accuracy is about 0.5 mm if the measurement depth span is 120 mm.), and resolution of the texture image is 2048 * 1536 pixels at the maximum. These range image and texture image are captured on the same imaging plane by the optical half mirror(fig. 2). As the application, the Eye Scanner has ability of divide capture by the motorized pan-tilt stage. At the free point of view measurement of texture on the curved surface shape object, which used eye-scanner, we are with the aim of 1) High resolution capturing.(require over 300 dpi) 2) Texture stretching of the free form shape object. 3) Wide area capture by a technique of divide capture. 4) Non blur image capture by using dynamic focusing. CameraHead DigitalCamera Video Camera LaserScanner Halfmirror TiltStage PanStage (a) Exterior Figure 1: Eye Scanner SideView High resolution Digital Camera Optical Mirror Flash (b) Camera Head and motorized pan-tilt stage system FrontView Flash Infrared Laser Scanner Infrared Video CCD for Range Finder Optical Halfmirror Infrared Laser Polygon Mirror Optical Halfmirror (a) Side View Figure 2: Structure of the Camera Head (b) Front View
3. DISTORTION CORRECTION BY THE STRETCHING At the first, we need to express shape of the object surface as a mathematics model in order to stretch out texture on a free curved surface with shape information. This mathematical paper model is equal to the formulation of the free shape transformation without the expansion and contraction, but it is difficult to include all shape transformation and is not practical. Therefore, we assume the freedom of bending is 1, and realize the texture stretch. This approximation is well description at the books surface, if it is given only gravity without external force. 3.1. Posture Normalization of the Curved Surface To normalize posture, we adapt measured whole points on an object surface by a quadratic polynomial function shown below. Z World Coordinate X-Y-Z α y c =const. measured data g(x) X 1 2 N Xc=0 Y β o X(xc,yc) X Figure 3: Posture normalization Figure 4: Curved surface stretch by polynomial function 2 Z = a + a X + a Y + a X + a XY + a Y 00 01 10 02 11 20 2 (1). This expression is not enough to express the shape of a various object's surface. However, if we limit the object as smooth surface that is like book's page, it is well description for the posture normalization. From this shape approximation, we can get the roll angle around Z axis α shown in fig. 3 is Z = b + bxˆ + b Xˆ 2, Xˆ = X + cy 0 1 2 α = arctanc (2). After posture correction of the roll angle, we correct the tilt angle by the β shown below. 1 Z = d0 + dy 1, β = arctan d 1 (3).
3.2. Texture Stretching by the Numerical Integration We can express curved surface shape of paper with complete change numerical single-valued function by normalization of posture if suppose the degree of freedom of curve of paper to be 1. For texture stretching, we use shape approximation by a single-valued function after the posture normalization that described above. If whole points, which measured were normalized, these are able to deal as independent of Y axis at the texture stretch. Therefore we expressed the surface shape of paper by a polynomial expression n Z = k0 + kn X g( X) n (4) of 1 variable, and stretched each line with a method in the following (Fig. 4). 1. Calculate the length of curves from 0 of gx ( ) to X = X ( x y c, c) by numerical value integration. 2. Replace X value of range image in x c, y c ( X ( x y c, c) )with numerical value integration result. 3. Reduce gx ( ) from Z value of distance image in x c, y c ( Zx ( y c, c) ). 4. HIGH DETAILED IMAGE ACQUISITION BY THE DIGITAL CAMERA IMAGE A resolution of 300dpi is necessary usually in order to acquire in small characters of a document in detail (i.e. dictionary), but it is difficult with CCD camera of NTSC standard even if we use the method of divide capture. Therefore the Eye Scanner is mounting digital camera on the same optical plane as video CCD for range finder, by the optical half mirror shown by fig. 2. On such optical coaxial structure, the coordinates relation between digital camera ( X Y ) and range finder ( X Y ) is given by d d c c xd r11 r12 tx mx 0 0 xc y d r21 r22 ty 0 my 0 yc = 1 h h 1 0 0 1 1 1 2 (5). The parameters m x and m y are the image resolution ratio of digital camera and range finder, and the parameters r ij and t x, t y, h 1, h 2 are the mapping parameters for the image rotation, translation and skew. These parameters are predetermined by the calibration. 5. DIVIDE CAPTURE TECHNIQUE FROM FREE POINT OF VIEWS In eye scanner, we are aiming the actualization about technique of capture of the document that placed at various position from free point of view without distortion. As the method to capturing the document, which arranged at a free position, we can think to capture a wide field of vision by the range finder, which has fisheye lens or omni vision sensor. However it has problem of the resolution, so the Eye Scanner has the motorized pan-tilt stage system to use a technique of divide capture (Fig. 5). Moreover it has also the ability of high-resolution image acquisition by this divide capture technique not only for capturing from free point of view. In the divide capture, a coordinate system of a range finder needs to decide the coordinate system that the turn center point of pan-tilt stage system on the world coordinate is to become the origin of the range finder coordinate by the calibration. After the calibration, we get the transformation
Figure 5: The concept of the divide capture X cosθ sinθ 0 1 0 0 X0 Y sin sin Y = θ cosθ 0 0 cosφ φ 0 Z 0 0 1 0 sinφ cosφ Z 0 (6) of camera rotation by pan and tilt angle θ, φ. From this transformation, we can transform the range images, and we can get merged texture images by the image rendering. 6. EXPERIMENT RESULTS 6.1. Stretching of the Cylindrical Object Firstly we confirmed the ability of texture stretch of curved surface with the cylindrical object shown in fig. 6. The images shown in fig. 7 is range-image, this is acquired with Eye Scanner. In fig. 7, it is shown a value of range data by brightness of an image and a bright pixel means the pixel is near. The resolution of the texture image is 1024*768 pixels, and range images are 640*480 pixels. Firstly, we carried out posture normalization by the method described in section 4.1 (Fig. 8). Secondarily, we carried out the texture stretch by the method described in section 4.2 (Fig. 9). In stretch process, we decided the order of polynomial function shown at eq.(4) is 4. To visualize the result of image stretch, we used triangular patch and original rendering method. Our method is able to generate an image at arbitrary resolution from the physical size because the range data is measured by Eye Scanner. In this result, the resolution of rendering of stretch result did it with 144 dpi.
Figure 6: A texture image of trashcan Figure 7: Range images of trashcan Figure 8: Posture normalization result Figure 9: Texture stretch result of fig. 6 6.2. Stretching of the News Paper This section shows the distortion correction of the newspaper of fig. 10(a) as stretch of a free curved surface. This newspaper includes color printing and large letters. Therefore the reflection brightness of the projection pattern is changed at these parts of newspaper. We show the result of rendering based on range-image in the fig. 10(b). A reflected pattern has been absorbed at a photograph and a big letter, and shape data are lost. Therefore data lost is produced even in the stretched output. However, we can interpolate the lost part from shape of circumference if the shape is smooth such as paper. So we used the method of interpolation that punctuate some blocks and used the polynomial function approximation segmentally. In the fig. 10(c) we show the result of the interpolation of the fig. 10(b). The fig. 10(d) is a stretched result, after the interpolation. The output of fig. 11(b) is a result of having done stretch of an image of fig. 11(a) by a same method. These results generated an image as 300 dpi of image resolution. In an object shown with 11, the degree of freedom of curve is 1, but it has concave part and convex part in the same object. Blur of an image is found in stretched result a little, but, as for the stretch of shape, a good result was provided.
Figure 10(a): The newspaper which includes color printing Figure 10(b): Rendering result which has lost parts Figure 10(c): Interpolation result of the lost parts Figure 10(d): Stretch result of newspaper
Figure 11(a): The newspaper which has convex and concave Figure 11(b): Stretch result of fig. 11(a). 6.3. Divide Capture Images shown in fig. 12 are texture images that were captured at the pan-angles of -10[deg], 0[deg] and 10[deg]. At the divide capture technique merged these texture images after coordinate transformation by the shape data captured simultaneously. Besides, the angle of rotation of Eye Scanner is already known by the control signal of motorized pantilt stage. A result of fig. 13 was provided after the normalization of posture by the stretch of a curved surface. Generally, in case of panorama image generation not using range image, accurate calibration to match the lens center of a camera with the turn center is indispensable. Moreover, an image discontinuity may occur at the merged domain by the skewness, however we can merge images simply and precisely from shape information given by Eye Scanner. (a) 10[deg.] (b) 0[deg.] (c) +10[deg.] Figure 12: Texture images by the camera head rotation
Figure 13: Texture merge results by the divide capture technique 7.CONCLUSION We developed the next generation document reader "Eye Scanner", which measure texture and shape of curved surface shape object and generate high detailed image without a distortion by shape information. The EyeScanner can measure a document actively by to turn camera head with motorized pan tilt stage system. Moreover, as for eye scanner, high detailed texture measurement that is the divide capture and high accuracy with a digital camera is possible. We examine texture stretch technique for various curved surfaces in future and are to examine precision improvement of the divide capture technique. REFRENCES 1. M. Kashimura, T.Nakajima, N.Onda, H.Saito, S.Ozawa, "Practical Introduction of Image Processing Technology to Digital Archiving of Rare Books", International Conf. on Signal Processing Application Technology, 1999. 2. Seong Ik Cho; Saito, H.; Ozawa, S., Shape recovery of book surface using two shade images under perspective condition, Trans. of the Institute of Electrical Engineers of Japan, Part C, vol.117-c, no.10, pp. 1384-90, 1997. 3. Y.Sato, K.Hattori, M.Otsuki "Real-Time Handy Range finder Cubicscope", ICARCV'94 singapore November 9-11, pp.1148-1152, 1994.