This document is downloaded from DR-NTU, Nanyang Technological University Library, Singaore. Title Automatic Robot Taing: Auto-Path Planning and Maniulation Author(s) Citation Yuan, Qilong; Lembono, Teguh Santoso; Zou, Yuhua; Chen, I-Ming Yuan, Q., Lembono, T. S., Zou, Y., & Chen, I.-M. (2015). Automatic Robot Taing: Auto-Path Planning and Maniulation, 7th IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM). Date 2015 URL htt://hdl.handle.net/10220/38762 Rights 2015 IEEE. Personal use of this material is ermitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including rerinting/reublishing this material for advertising or romotional uroses, creating new collective works, for resale or redistribution to servers or lists, or reuse of any coyrighted comonent of this work in other works. The ublished version is available at: [htt://dx.doi.org/10.1109/iccis.2015.7274616].
Automatic Robot Taing: Auto-Path Planning and Maniulation Qilong Yuan, Teguh Santoso Lembono, Yuhua Zou, I-Ming Chen, Fellow, IEEE Abstract Many industrial alications, such as ainting, lasma sraying, require eole to work long hours to cover the uninvolved art with masking taes. In this aer, we introduce an automatic robot taing system which can reconstruct the 3D model of the taing art, do ath lanning and comlete the surface covering of the arts using masking taes automatically. The system uses a Kinect as the 3D scanner to acquire the 3D oint clouds of the taing art. Based on the digital 3D oints cloud model of the art, we introduces a ath lanning method to generate the moving ath of the systems. A rotary latform and a 6-DOF robot arm collaborate together to execute the robot taing rocess. For continuous taing, the system and method can handle Tubular shaes. Exeriment of taing a cylindroid shae is conducted to validate the system and method. For arts with more comlex geometry, further develoment on the automatic initial tae attaching and cutting mechanism is needed. With such system, the taing work can be much less troublesome. Also, the ath lanning method alies to alications such as drawing or writing on different surfaces etc. as well. I. INTRODUCTION Industrial robot is now widely used in many manufacturing alications. The robot machining, sraying[1, 2], wielding, robot finishing [3], ick and lace are well develoed technologies widely alied in the industrial robot workcells. In these robot automation tasks, the robot end-effectors are secially designed to meet the needs for the secific tasks. The robot-environment interaction during traditional robot maniulation can be oint contact, like wielding, surface contact, like finishing. In tasks like ainting and sraying, there is no direct contact between the robot and the object, but there are orientation requirement on the tools. The geometry of the art to be ainted need to be known in order to make a good ath lanning and auto-ainting [1, 2, 4, 5]. A more comlex robot automation task is auto-taing. Taing rocess is required in many industrial alications such as ackaging, roing armature aer while making motors, taing for iron ies etc. Taing regular shaes can be easily done through using some standard taing machine [6, 7]. However, if the geometry of the taing arts become comlex, the taing roblem becomes comlex. As a result, such taing works are mainly done manually. For examle, in lasma sraying, the uninvolved arts need to be covered by masking taes. Such oeration based on a skilled works will request half days work for a comlex mechanical art. It is exected that this job can be done by robotic system. But, to realize this, we firstly need to acquire the 3D model of the All authors are with School of Mechanical and Aerosace Engineering, Nanyang Technological University, Singaore 597627 (I-Ming Chen, hone: 67906203, e-mail: michen@ntu.edu.sg; Qilong Yuan. hone: 65-84545660, e-mail: yu0017ng@ntu.edu.sg). arts for taing. Also, we need to have way to make the ath lanning in order to make the automatic taing ossible. Taing requests the tae to be roerly attached to the surface of the arts and with some certain overlaing while covering the entire area of interests. This requires: (1). the recise acquisition of the arts to be taed. (2). Secial design of the taing robot end-effector. (3). strategies to make the roer taing ossible. Because, during the taing, not only the contacting oint of taing but also the surface normal vector are all imortant for the taing end-effector to follow in order to do roer taing. In the manual taing, the worker see the geometry of the arts and use his hand to control the orientation of the tae. At the same time, the worker uses his finger to ress and ush the tae to make sure that the tae is nicely attached to the surface of the art. In the overall rocess, the taing ath should be roerly lanned in order to let the tae cover the entire area of interests. To realize the automatic robot taing rocess, we need devices to see the objects. Thus, devices to get the model descrition of the art are needed in order to rovide comuter understandable data. We also need a robot to realize the taing motion, rovided a caable end-effector is designed to realize the human hand function. And we still need to have lanning strategies to let the robot to follow the taing ath to realize the comlete taing task. It is understandable that it is better to have feedback control of the end-effector to coe with the errors in the modelling and robot maniulation. But if it is too comlex and/or not economical, we can design some roer comliance mechanism to comensate the errors. In this aer, we introduce a system and corresonding methods for automatic robot taing. In the system, a 6 degree-of-freedom robot arm and a rotating latform is used to do the auto-taing. To get the geometry of the art, we aly the commercial 3D scanner sensors (Microsoft Kinect, or Artec Eva scanner) to get the oint cloud of the art based on the oen source KinFusion rogram [8, 9]. With the oint clouds, it is very handy to calculate the surface normal vector [10]. Based on the digital 3D model file of the art reconstructed from the 3D scanner, the taing aths for the robot system to follow and the angular trajectory of the rotation latform are lanned, following a ath lanning strategy that we roosed in this aer. The remaining art of the aer is organized in the following manner. Section II introduced the system setu for the robot taing. Section III described the ath lanning of the robot taing motion. Section IV rovides the exerimental results. Then, the aer is briefly concluded in Section V. 1
II. PLATFORM DESIGN In this automatic robot taing system, a 6 DOF robot maniulate and a one DOF rotating latform is alied. As shown in Figure 1, the taing art is fixed on the rotating latform. During the system initialization, the relative osition between the robot maniulator and the rotating latform is calibrated. B. Robot End-Effector Design The taing tasks require roer contact between surface of the taing object and the tae. In human manual taing, the finger of the worker resses the tae along the taing orientation in order to guarantee the taing quality. In the automatic taing, a secial mechanism is designed to relace the function of the human figure. In this work, we designed a taing end-effector as shown in Figure 2, the rear art holds the tae roller and guide the tae. The front art serves as a comliance finger to ress the tae to the surface and comensate some of the maniulation errors from the rotation latform and the robot arm. The stiffness of the sring is about 2000N/m. During the taing, the sring mechanism is ressed with 0.3cm allowing about 10N contact forces to ress the tae. This sring mechanism can comensate the error from the scanning results and the robot maniulation. Figure 1: Automatic taing system with rotating latform Then, a 3D scanner, Kinect sensor is used to get the 3D oint cloud model of the taing art. The oint cloud model from the scanning results can be rocessed to digital 3D model of the art with surface location and surface normal information. With the resulting 3D model of the art and the robot kinematics, the taing aths of the rotating latform and the robot maniulator are generated. A. Rotating latform The one DOF rotating latform is driven by a controllable steer motor so that it can be controlled to rotate a certain degree during a short eriod of time. In this way, the rotation latform can collaborate with the robot maniulate to conduct the taing tasks. The different rotating attern of the latform will affect the taing ath of the robot. Therefore, in this taing rocess, the rotating attern can be otentially used for otimization of the robot ath lanning in terms of working sace and maniulability etc. C. 3D scanning Figure 2: The taing end-effector In the 3D scanning model generation, the Kinect sensor is fixed aside from the rotating latform with the art fixing on it. The latform rotate for three round, during which the 3D oint cloud is generated based on the scanning algorithm, KinFusion for Kinect sensors. After the scanning of the model, some ost rocess of the model are still needed such as hole filling, merging multile scanning results, deleting internal oints etc. Then, the surface normal vector can be calculated based on the oint cloud model afterwards. Please take note that, the 3D model after the scanning is with resect to the initial frame of the scanner. It need to be transformed back to the initial reference frame of the rotating latform in order to make the ath lanning ossible. This can be done through utting some reference marker at the latform system and let the scanner recognize the relative osition between itself and the rotating latform. Let reresent the osture of the camera initial frame with resect to the latform reference frame. Then, all the oints, in the scanning model reresented in becomes: [,,1] = [,,1] ( 1 ) 2
Accordingly, the normal vectors, in the scanning model reresented in becomes: [,,0] = [,,0] ( 2 ) After transforming the model to the latform reference frame, the ath lanning of the taing maniulation is ossible. Figure 3: 3D scanning results III. PATH PLANNING AND OPTIMIZATION In the general case, a art with arbitrary geometry can be very comlex. Thus, the taing of such arts requires detail lanning on how to tae secific area and where to cut the taes in order to cover the entire surface nicely and,try to use as less tae. This request a cutter to automatically cutting the tae and attach the tae subsequently. Currently, without cutting the tae, we mainly discuss on continuous taing without cutting the tae. For a mechanical arts fixed on the rotary latform, surface geometry can be very different. The taing strategies for different shaes are also not unique. Here, we start with Tubular shaes and shaes similar to that as shown in Figure 4, because such shaes are very common and have very high chance for taing in ractice. After taing the vertical surface, the flat surface art can be taed based on the flat surface covering method. Figure 4: Shaes for taing in this study In the 3D model reconstruction based on the 3D scanner, the digital model including the oint cloud and the calculated surface normal are reresented in the scanners initial reference frame. After the taing art is fixed on the rotating latform, the art will rotate together with it. Therefore, this digital model will also change after a rotating motion. Therefore, how the rotating latform moves, will also affect the robot moving ath. In the taing rocess, the rotating latform holds the art for taing. While the latform rotate, the robot taer need to contact nicely with the surface of the art and with roer orientation in order to make a nice taing. Therefore, this is actually a collaboration between the robot taer and the rotating latform. When we observe a manual work taing a mechanical art, the worker sits in front of a desk and uts the art on a assive rotary latform on the desk. He is mainly manual rotating the latform while taing, trying to tae the art mainly in front of himself. By doing so, his oerating sace is within a very small area and thus saves the cost of energy. Similarly, in order to let the automatic robot taing system save sace and energy cost, we also try to let the robot arm move mainly in neighborhood of its front. Therefore, in the robot ath lanning, we can adjust the rotation attern and the robot taer to otimize the taing rocess, to minimize effort (energy) of robotic system for examle. A. Path Planning Algorithm In the taing rocess, we start from the bottom of the taing area. The ath lanning method is shown below. 1. Get the initial oint for taing from the digital 3D model with initial height (h=h0), and angle (θ=0) with resect to the reference frame of the latform. 2. Numerically calculate the erimeter (S) of the closed curve on section z=h0. The detailed method is as follows a. Int i=0; θ=0 b. When θ <360, θ= θ+dθ;i=i+1;search the surface oint v i in neighbourhood best satisfying (h0, θ).s=s+ v i-1v i c. i=0; θ=0 3. Determine the itch angle (Equivalent to screw itch if the art is cylinder) of the tae () based on the erimeter: Pitching angle ( in rad): = 1 / where r is the overla rate of the tae. Here r=20%. Then start the ath lanning. The main idea is to udate the itching angle and robot end-effector ose to make sure the tae is roerly taed to cover the surface with the given tye overlaing. 4. θ= θ+dθ;i=i+1. 5. Search the surface oint with normal vector, in neighbourhood best satisfying (h, θ). Then: h=h+ v i-1v i (This is the ress oint in the taing ath lanning). 6. If θ >=360,then i=0; θ= 0 and udate S and in the same way as in 2 and 3. 7. Decide on the rotation angle of the rotary latform and convert the 3D model data to the reference frame: = ( 3 ), =, ( 4 ) 8. Calculate the robot ose based on,,,. 9. If h<, go to 4. 3
Using the examle Part model for taing, we made a ath lanning. Figure 5 shows the taing ath on the art surface. The urle oints cloud reresents the model. The blue line shows the surface normal direction (useful for controlling the robot end-effector ressing direction). The green circle shows the taing roller while taing. The red color reresents the tae covered on the surface. Figure 5: The taing ath on the art surface. Figure 5 only rovides the relative ath on the end-effector. While the motion of the robot maniulator deends on the moving attern of rotating latform, as exlained in the 7 ste of the ath lanning. The rotating attern will effects the moving efficiency of the robot arm. For now, we mainly tested on two rotating atterns, one is continuously rotating the latform and the robot mainly move in and out and adjust its taing osture. A better solution is to rotate the latform every 90 degree and sto to let the robot arm finish taing the corresonding art and rotate another 90 degree. B. Determination of the end-effector ose X Y Z Y Pc Z X Figure 6: Taing End-Effector Posture Definition. Once the taing oint is known, its osition P c and its surface normal, with resect to the rotating latform is known. In this work, we let the end-effector ressing direction follow the surface normal direction. As indicated in Figure 7, the x-axis of the end-effector follows the normal vector direction on the surface. x = v ( 5 ) i, c f Y axis is defined to be in the tangintal direction of the surface and with th itch angle of, the itch angle of the tae. y = Rotx( )([0,0,1] x ) ( 6 ) z = x y ( 7 ) Then, the osture of the robot with resect to the rotary latform TP can be determined [11, 12] by the following equation, x, y, z Pc TP = 0 1. ( 8 ) This end-effector need to be resented in the robot base frame in order to use the robot inverse kinematics to solve out the robot joint angles. Let TBP be the relative ose between the robot base frame and the latform reference frame determined during the system initialization. In the robot base frame, the end-effector ose is, TB = TBPTP. ( 9 ) In the actual taing execution, initially, the robot end-effector arrive at the initial taing oint with a re-determined osture. For now, we still need eole to assist in the initial attaching of the tae. Figure 7: Tae attached at the initial oint when taing starts After the ath lanning, the rotating trajectory of the rotary latform and the corresonding robot maniulating ath are determined. Then, the system can aly the data for automatic robot taing. IV. EXPERIMENT The rocess of the robot taing is illustrated in Figure 8. The taing art is firstly fixed on the rotating latform. Then the system initialization is conducted to figure out the relative osition between the robot maniulate and the rotary latform. The fixing location of the arts is also calibrated. Then, the scanning of the 3D oint cloud is conducted. After a simle editing of the 3D digital model, the model is assed to the ath lanning software to generate the motion of the robot and rotary latform. Then, the system start to execute the automatic taing. 4
Figure 10: Comleted taing results The whole rocess takes within 2 minutes for current oerating seed. The system can further seed u for fast rocessing. When we try manually to make the taing, firstly, we cannot attach the tae with a correct itch angle. Meanwhile, it is very difficult to get the tae go to the correct direction with nice taing results. It takes time to get a worker trained to make faster and better taing or other skillful works. From these oint of views, robot automation in such work is helful. But of course, if the task is very comlex with various conditions, it requests more rograming works and secial design of the mechanical system. Figure 8: The working flow of the system (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Figure 9: Taing rocess of the automatic robot taing system An examle of the taing rocess is illustrated in Figure 9. Figure 10 shows the comleted taing model. V. CONCLUSION This aer introduced the setu and corresonding method of an automatic robot taing system. This system includes a 3D scanner, a rotating latform and a 6 DOF robot maniulator to realize the auto scanning and auto-taing task. The 3D scanner is used to reconstruct the 3D model of the arts to be taed. Based on the 3D oints cloud, and the robot system, a ath lanning method is introduced to generate the robot moving ath and the rotating latform motion. Currently, the system and method mainly handle the continuous taing of Tubular like shaes. Further develoment on the automatic initial tae attaching and cutting mechanism is needed to make the system fully automatic and available for more comlex geometry. Considering time consuming manual work of the taing rocess in workcells, the automatic robot taing system can be more efficient and save eole out of this tiring work. The ath lanning method based on digital 3D oint cloud data introduced in this work also alies to alications such as drawing or writing on different surfaces as well. Therefore, based on the automatic 3D scanning and some robotic system, robotic systems can be develoed to do these tasks on not only arts with known regular geometry but also arts with comlex shaes. ACKNOWLEDGMENT This work was suorted in art by the Agency for Science, Technology and Research, Singaore, under SERC Grant 12251 00005. REFERENCES [1] W. Chen and D. Zhao, "Path Planning for Sray Painting Robot of Workiece Surfaces," Mathematical Problems in Engineering, vol. 2013, 2013. [2] A. Gasaretto, R. Vidoni, D. Pillan, and E. Saccavini, "Automatic Path and Trajectory Planning for Robotic Sray Painting," in Robotics; 5
Proceedings of ROBOTIK 2012; 7th German Conference on, 2012,. 1-6. [3] C. Ng, K. Chan, W. Teo, and I. Chen, "A method for caturing the tacit knowledge in the surface finishing skill by demonstration for rogramming a robot," in Robotics and Automation (ICRA), 2014 IEEE International Conference on, 2014,. 1374-1379. [4] H. Chen, T. Fuhlbrigge, and X. Li, "Automated industrial robot ath lanning for sray ainting rocess: a review," in Automation Science and Engineering, 2008. CASE 2008. IEEE International Conference on, 2008,. 522-527. [5] S. Larsson and J. A. Kjellander, "Path lanning for laser scanning with an industrial robot," Robotics and autonomous systems, vol. 56,. 615-624, 2008. [6] W. J. Hottendorf, "BOX TAPING MACHINE," ed: Google Patents, 1972. [7] N. Horiguchi, Y. Kaneko, and S. Uda, "Feeder of wraing aer for coin wraing machine," ed: Google Patents, 1992. [8] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, et al., "KinectFusion: real-time 3D reconstruction and interaction using a moving deth camera," in Proceedings of the 24th annual ACM symosium on User interface software and technology, 2011,. 559-568. [9] R. A. Newcombe, A. J. Davison, S. Izadi, P. Kohli, O. Hilliges, J. Shotton, et al., "KinectFusion: Real-time dense surface maing and tracking," in Mixed and augmented reality (ISMAR), 2011 10th IEEE international symosium on, 2011,. 127-136. [10] N. J. Mitra, A. Nguyen, and L. Guibas, "Estimating surface normals in noisy oint cloud data," International Journal of Comutational Geometry & Alications, vol. 14,. 261-276, 2004. [11] J. McCarthy, Introduction to theoretical kinematics: MIT Press, Cambridge, MA, 1990. [12] R. Murray, Z. Li, S. Sastry, and S. Sastry, A mathematical introduction to robotic maniulation: CRC, 1994. 6